Repository navigation
Bugs and inconsistencies in unicodedata #80667
Description
| BPO | 36486 |
|---|---|
| Nosy | @malemburg, @vstinner, @ezio-melotti, @dscorbett |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2019-03-30.14:41:13.171>
labels = ['3.8', 'type-bug', '3.7', 'expert-unicode', 'docs']
title = 'Bugs and inconsistencies in unicodedata'
updated_at = <Date 2019-04-05.18:28:55.383>
user = 'https://github.com/dscorbett'bugs.python.org fields:
activity = <Date 2019-04-05.18:28:55.383>
actor = 'terry.reedy'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Unicode']
creation = <Date 2019-03-30.14:41:13.171>
creator = 'dscorbett'
dependencies = []
files = []
hgrepos = []
issue_num = 36486
keywords = []
message_count = 1.0
messages = ['339203']
nosy_count = 5.0
nosy_names = ['lemburg', 'vstinner', 'ezio.melotti', 'docs@python', 'dscorbett']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36486'
versions = ['Python 3.7', 'Python 3.8']Linked PRs
- gh-80667: Fix Tangut ideographs names in unicodedata #101585
- gh-80667: fix case-sensitivity of some unicode literal escapes #107281
- [3.14] gh-80667: Fix case-sensitivity of some Unicode literal escapes (GH-107281) #144753
- [3.13] gh-80667: Fix case-sensitivity of some Unicode literal escapes (GH-107281) #144754
- gh-80667: Fix Tangut ideographs names in unicodedata #144789
- [3.14] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-144789) #144871
- [3.13] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-144789) (GH-144871) #144875
Activity
dscorbett commented on Mar 30, 2019
In unicodedata, the functions lookup and name have some bugs and inconsistencies.
lookup matches case-insensitively, except for the algorithmic names of Hangul syllables and CJK unified ideographs, which must be in all caps. The documentation does not explain how character names are fuzzily matched.
lookup accepts names like “CJK UNIFIED IDEOGRAPH-04E00”, where the code point has a leading zero.
lookup and name don’t implement rule NR2, defined in chapter 4 of Unicode, for Tangut ideographs’ names.
cfbolz commented on Oct 10, 2022
just hit the difference between CJK UNIFIED IDEOGAPH-*, HANGUL SYLLABLE * and other unicode names again:
>>> "\N{SNOWMAN}\N{SNOWmAn}\N{sNoWMAN}\N{snowman}" '☃☃☃☃' >>> # but: >>> "\N{CJK UNIFIED IDEOGRAPH-3400}" '㐀' >>> "\N{cjk UNIFIED IDEOGRAPH-3400}" File "<stdin>", line 1 "\N{cjk UNIFIED IDEOGRAPH-3400}" ^ SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-29: unknown Unicode character namegh-80667: Fix case-sensitivity of some Unicode literal escapes (GH-10…
pythongh-80667: Fix case-sensitivity of some Unicode literal escapes (p…
6 remaining items
gh-80667: Fix Tangut ideographs names in unicodedata (GH-144789)
[3.14] pythongh-80667: Fix Tangut ideographs names in unicodedata (py…
[3.14] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-…
[3.14] pythongh-80667: Fix lookup for Tangut ideographs in unicodedata (
[3.13] gh-80667: Fix lookup for Tangut ideographs in unicodedata (GH-…
serhiy-storchaka commented on Feb 16, 2026
Thank you for your report, @dscorbett, and for your PRs, @wismill and @SnoopJ. My apologies that it took so long to review them -- we have few Unicode experts -- but finally this was resolved.
3.13 and 3.14 will only accept the Tangut ideographs' names, but not produce them. 3.15 will also produce them.
Egyptian hieroglyphs, Khitan small script characters and Nushu characters were already supported, because their names were listed explicitly in UnicodeData.txt. But this gives us opprtunity for optimization, see #144882.
SnoopJ commented on Feb 16, 2026
Thanks @serhiy-storchaka, glad to be of some assistance! I'm glad to see your follow-up issue as well, I was wondering about further improvements when dealing with named ranges like that as I looked over your draft PR.
pythongh-80667: Fix case-sensitivity of some Unicode literal escapes (p…
pythongh-80667: Fix Tangut ideographs names in unicodedata (pythonGH-…
pythongh-80667: Fix case-sensitivity of some Unicode literal escapes (p…
pythongh-80667: Fix Tangut ideographs names in unicodedata (pythonGH-…