FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters by shihai1991 · Pull Request #22219 · python/cpython · GitHub

/ cpython Public

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters - #22219

Merged
vstinner merged 12 commits into
python:masterfrom
shihai1991:bpo_39337_new_2
Oct 14, 2020
ConversationCommits (12)ChecksFiles changed
Merged

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters#22219
vstinner merged 12 commits into
python:masterfrom
shihai1991:bpo_39337_new_2

Conversation

shihai1991 commented Sep 12, 2020 •
edited
Loading

Copy link
Copy Markdown
Member

encodings.normalize_encoding() now ignores non-ASCII characters.

https://bugs.python.org/issue39337

shihai1991 changed the title encodings.normalize_encoding() should ignore non-ASCII letters WIP: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020
update test
0fcafb8 
update test
shihai1991 commented Sep 12, 2020
View reviewed changes
Comment thread Lib/test/test_source_encoding.py Outdated
shihai1991 changed the title WIP: encodings.normalize_encoding() should ignore non-ASCII letters [WIP]bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020
shihai1991 changed the title [WIP]bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters Sep 12, 2020
Revert "update test"
dea24d8 
This reverts commit 0fcafb8.
shihai1991 mentioned this pull request Sep 13, 2020
Closed
shihai1991 mentioned this pull request Sep 29, 2020
Merged
taleinat requested a review from vstinner September 30, 2020 06:44
vstinner reviewed Oct 1, 2020
View reviewed changes
Comment thread Misc/NEWS.d/next/Library/2020-09-13-02-02-18.bpo-39337.L3NXTt.rst Outdated
Comment thread Lib/test/test_source_encoding.py Outdated
shihai1991 changed the title bpo-39337: encodings.normalize_encoding() should ignore non-ASCII letters bpo-39337: encodings.normalize_encoding() now ignores non-ASCII letters Oct 3, 2020
vstinner reviewed Oct 8, 2020
View reviewed changes
Comment thread Lib/test/test_source_encoding.py Outdated
apply victor's comment
3fa221f
Merge remote-tracking branch 'origin/master' into bpo_39337_new_2
a69eef8 
* origin/master: (147 commits)
  Fix the attribute names in the docstring of GenericAlias (pythonGH-22594)
  bpo-39337: Add a test case for normalizing of codec names (pythonGH-19069)
  bpo-41557: Update Windows installer to use SQLite 3.33.0 (pythonGH-21960)
  bpo-41976: Fix the fallback to gcc of ctypes.util.find_library when using gcc>9 (pythonGH-22598)
  bpo-41306: Allow scale value to not be rounded (pythonGH-21715)
  bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (pythonGH-22595)
  bpo-41376: Fix the documentation of `site.getusersitepackages()` (pythonGH-21602)
  Revert "bpo-26680: Incorporate is_integer in all built-in and standard library numeric types (pythonGH-6121)" (pythonGH-22584)
  bpo-41923: PEP 613: Add TypeAlias to typing module (python#22532)
  Fix comment about PyObject_IsTrue. (pythonGH-22343)
  bpo-38605: Make 'from __future__ import annotations' the default (pythonGH-20434)
  bpo-41905: Add abc.update_abstractmethods() (pythonGH-22485)
  bpo-41944: No longer call eval() on content received via HTTP in the UnicodeNames tests (pythonGH-22575)
  bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (pythonGH-22566)
  Post 3.10.0a1
  Python 3.10.0a1
  bpo-41584: clarify when the reflected method of a binary arithemtic operator is called (python#22505)
  bpo-41939: Fix test_site.test_license_exists_at_url() (python#22559)
  bpo-41774: Tweak new programming FAQ entry (pythonGH-22562)
  bpo-41936. Remove macros Py_ALLOW_RECURSION/Py_END_ALLOW_RECURSION (pythonGH-22552)
  ...
vstinner reviewed Oct 9, 2020
View reviewed changes
Comment thread Lib/test/test_codecs.py Outdated
Comment thread Lib/test/test_codecs.py Outdated
Comment thread Lib/test/test_codecs.py Outdated
vstinner reviewed Oct 11, 2020
View reviewed changes
Comment thread Lib/test/test_codecs.py Outdated
Comment thread Lib/test/test_codecs.py Outdated
Comment thread Lib/test/test_codecs.py Outdated
self.assertEqual(out, 'utf_8')
out = encodings.normalize_encoding('utf 8')
self.assertEqual(out, 'utf_8')
out = encodings.normalize_encoding('UTF 8')

vstinner Oct 11, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Would you mind to add a comment to explain that the function does not convert upper case letters to lower case letters, just to make the purpose of this test even more explicit?

shihai1991 Oct 12, 2020 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

To be honest, I don't know how to exact explain it~
I found a case in https://github.com/python/cpython/blob/master/Lib/locale.py#L358.
Looks like It's fine to update encodings.normalize_encoding() to conver to lower-case.

shihai1991 Oct 12, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

just describe the fact, Lol~

shihai1991 Oct 12, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Maybe we can enhance this encodings.normalize_encoding()? I am not sure~

Comment thread Lib/test/test_codecs.py Outdated
Comment thread Lib/encodings/__init__.py
chars.append('_')
chars.append(c)
if c.isascii():
chars.append(c)

vstinner Oct 11, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I wanted to ask you to add a ".. versionchanged:: 3.10" entry in the documentation, but then I noticed that the encodings module was never documented! Oh!

shihai1991 Oct 12, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

If end user will use this function or module, I can try to create the doc, but I need some time to do it :)

vstinner Oct 12, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

It can and must be addressed in a separated PR anymore. The lack of documentation should not hold this change.

shihai1991 Oct 12, 2020

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

ok, copy that.

vstinner reviewed Oct 12, 2020
View reviewed changes
Comment thread Lib/test/test_codecs.py Outdated
vstinner approved these changes Oct 12, 2020
View reviewed changes

vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

LGTM.

@taleinat @serhiy-storchaka or @malemburg: Do you want to double check this PR?

Comment thread Lib/test/test_codecs.py
vstinner reviewed Oct 13, 2020
View reviewed changes
Comment thread Doc/whatsnew/3.10.rst Outdated
shihai1991 changed the title bpo-39337: encodings.normalize_encoding() now ignores non-ASCII letters bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters Oct 13, 2020
vstinner merged commit c5b049b into python:master Oct 14, 2020

vstinner commented Oct 14, 2020

Copy link
Copy Markdown
Member

Merged, thanks.

shihai1991 commented Oct 14, 2020

Copy link
Copy Markdown
Member Author

thanks victor for your tons of comments ;)

xzy3 pushed a commit to xzy3/cpython that referenced this pull request Oct 18, 2020
bpo-39337: encodings.normalize_encoding() now ignores non-ASCII chara…
bb71c6d 
…cters (pythonGH-22219)
adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 13, 2021
bpo-39337: encodings.normalize_encoding() now ignores non-ASCII chara…
5959a19 
…cters (pythonGH-22219)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

vstinner vstinner approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

Back | FazBrowse Home | New Git URL