FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Issue a warning if a surrogate char is encountered · aongko/unidecode@5ff1c35 · GitHub

Repository navigation

Commit 5ff1c35

Browse files
committed
Issue a warning if a surrogate char is encountered
Also, improved the section in README regarding "narrow" Python builds.
1 parent 6fba4e6 commit 5ff1c35

4 files changed

Lines changed: 65 additions & 3 deletions

File tree

‎README‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,9 +60,21 @@ Requirements
6060

6161
Nothing except Python itself.
6262

63-
You will need a Python build with "wide" Unicode characters in order for
64-
unidecode to work correctly with characters outside of Basic Multilingual
65-
Plane. Surrogate pair encoding of "narrow" builds is not supported.
63+
You need a Python build with "wide" Unicode characters (also called "UCS-4
64+
build") in order for unidecode to work correctly with characters outside of
65+
Basic Multilingual Plane (BMP). Common characters outside BMP are bold, italic,
66+
script, etc. variants of the Latin alphabet intended for mathematical notation.
67+
Surrogate pair encoding of "narrow" builds is not supported in unidecode.
68+
69+
If your Python build supports "wide" Unicode the following expression will
70+
return True::
71+
72+
>>> import sys
73+
>>> sys.maxunicode > 0xffff
74+
True
75+
76+
See PEP 261 for details regarding support for "wide" Unicode characters in
77+
Python.
6678

6779

6880
Installation

‎tests/basic_2.py‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,10 +51,51 @@ def showwarning_new(message, category, *args):
5151

5252
def test_bmp(self):
5353
for n in xrange(0,0x10000):
54+
# skip over surrogate pairs, which throw a warning
55+
if 0xd800 <= n <= 0xdfff:
56+
continue
57+
5458
# Just check that it doesn't throw an exception
5559
t = unichr(n)
5660
unidecode(t)
5761

62+
@unittest.skipIf(sys.maxunicode < 0x10000, "narrow build")
63+
def test_surrogate_pairs(self):
64+
# same character, written as a non-BMP character and a
65+
# surrogate pair
66+
s = u'\U0001d4e3'
67+
68+
# Note: this needs to be constructed at run-time, otherwise
69+
# a "wide" Python seems to optimize it automatically into a
70+
# single character.
71+
s_sp_1 = u'\ud835'
72+
s_sp_2 = u'\udce3'
73+
s_sp = s_sp_1 + s_sp_2
74+
75+
self.assertEqual(s.encode('utf16'), s_sp.encode('utf16'))
76+
77+
log = []
78+
def showwarning_new(message, category, *args):
79+
if ("Surrogate character" in str(message)) and \
80+
(category is RuntimeWarning):
81+
log.append((message, category))
82+
else:
83+
showwarning_old(message, category, *args)
84+
85+
showwarning_old = warnings.showwarning
86+
warnings.showwarning = showwarning_new
87+
warnings.filterwarnings("always")
88+
89+
a = unidecode(s)
90+
a_sp = unidecode(s_sp)
91+
92+
self.assertEqual('T', a)
93+
94+
# Two warnings should have been logged
95+
self.assertEqual(2, len(log))
96+
97+
warnings.showwarning = showwarning_old
98+
5899
def test_circled_latin(self):
59100
# 1 sequence of a-z
60101
for n in xrange(0, 26):

‎tests/basic_3.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,10 @@ def test_ascii(self):
1111

1212
def test_bmp(self):
1313
for n in range(0,0x10000):
14+
# skip over surrogate pairs, which throw a warning
15+
if 0xd800 <= n <= 0xdfff:
16+
continue
17+
1418
# Just check that it doesn't throw an exception
1519
t = chr(n)
1620
unidecode(t)

‎unidecode/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,11 @@ def unidecode(string):
4444
if codepoint > 0xeffff:
4545
continue # Characters in Private Use Area and above are ignored
4646

47+
if 0xd800 <= codepoint <= 0xdfff:
48+
warnings.warn( "Surrogate character %r will be ignored. "
49+
"You might be using a narrow Python build." % (char,),
50+
RuntimeWarning, 2)
51+
4752
section = codepoint >> 8 # Chop off the last two hex digits
4853
position = codepoint % 256 # Last two hex digits
4954

0 commit comments

Comments
 (0)

Back | FazBrowse Home | New Git URL