Repository navigation
gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 - #155293
Merged
encukou merged 7 commits intoAug 18, 2026
Merged
encukou merged 7 commits into
encukou merged 7 commits into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com>
Copy link
Copy Markdown
Contributor
Author
|
@StanFromIreland @malemburg Now that GHA service has recovered, requesting a review on this PR. |
Sorry, something went wrong.
encukou
reviewed
Aug 11, 2026
Comment thread
Lib/test/test_codecs.py
Outdated
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Copy link
Copy Markdown
Member
|
I filed a suggestion as a PR: sethmlarson#2. Otherwise, LGTM -- I didn't find another issue. @malemburg would be the best person to review this, of course. Marc-André, should we wait for you? |
Sorry, something went wrong.
malemburg
reviewed
Aug 13, 2026
malemburg
left a comment
Copy link
Copy Markdown
Member
There was a problem hiding this comment.
I don't have time for a review, but from casual inspection, this looks good to me. Thanks.
Sorry, something went wrong.
Copy link
Copy Markdown
Member
|
Thanks for the fix!
OK; let me know if you want to delegate. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
Thanks @sethmlarson for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
GH-156017 is a backport of this pull request to the 3.15 branch. |
Sorry, something went wrong.
encukou
added a commit
to encukou/cpython
that referenced
this pull request
Aug 18, 2026
…RFC 3454 (pythonGH-155293) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> (cherry picked from commit 7e109d0)
Copy link
Copy Markdown
|
GH-156020 is a backport of this pull request to the 3.14 branch. |
Sorry, something went wrong.
pablogsal
pushed a commit
that referenced
this pull request
Aug 18, 2026
… RFC 3454 (GH-155293) (#156017) gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>
6 of 8 tasks
encukou
added a commit
that referenced
this pull request
Sep 4, 2026
… RFC 3454 (GH-155293) (GH-156020) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14 and below, so unassigned characters don't have bidi values. Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>
Yhg1s
pushed a commit
that referenced
this pull request
Sep 8, 2026
… RFC 3454 (GH-155293) (GH-156020) (#156930) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14 and below, so unassigned characters don't have bidi values. (cherry picked from commit 1e54caa) Also, add Unicode_3_2_0_FunctionsTest as in the later versions, to make the new test work. Co-authored-by: Seth Larson seth@python.org Co-authored-by: Stan Ulbrych 89152624+stanfromireland@users.noreply.github.com Co-authored-by: Petr Viktorin encukou@gmail.com
encukou
added a commit
that referenced
this pull request
Sep 9, 2026
… RFC 3454 (GH-155293) (GH-156020) (GH-156922) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14 and below, so unassigned characters don't have bidi values. (cherry picked from commit 1e54caa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> --------- Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com>
pablogsal
pushed a commit
that referenced
this pull request
Oct 1, 2026
… RFC 3454 (GH-155293) (GH-156020) (#156932) * [3.10] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14 and below, so unassigned characters don't have bidi values. (cherry picked from commit 1e54caa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> * Add Unicode_3_2_0_FunctionsTest * Regenerate Unicode data --------- Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com>
pablogsal
pushed a commit
that referenced
this pull request
Oct 1, 2026
… RFC 3454 (GH-155293) (GH-156020) (#156931) * [3.11] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020) Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14 and below, so unassigned characters don't have bidi values. Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> (cherry picked from commit 1e54caa) * Add Unicode_3_2_0_FunctionsTest * Regenerate Unicode data --------- Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.
This PR fixes makeunicodedata.py and re-runs mkstringprep.py with the new Unicode database. In the future mkstringprep.py should be run as a part of makeunicodedata.py.
I'll take on backporting this PR manually, as we'll need to apply the patch and then re-execute makeunicodedata.py for each Python version as each contains a different Unicode version.