Repository navigation
gh-123682: Fix Unicode category check - #123683
SnoopJ wants to merge 2 commits into
Conversation
Sorry, something went wrong.
|
Ah, you have to update test_pyrepl: ERROR: test_control_character (test.test_pyrepl.test_pyrepl.TestPyReplOutput.test_control_character)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_pyrepl/test_pyrepl.py", line 657, in test_control_character
output = multiline_input(reader)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_pyrepl/support.py", line 18, in multiline_input
return reader.readline()
~~~~~~~~~~~~~~~^^
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/_pyrepl/reader.py", line 786, in readline
self.handle1()
~~~~~~~~~~~~^^
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/_pyrepl/reader.py", line 743, in handle1
event = self.console.get_event(block)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_pyrepl/support.py", line 123, in get_event
return next(self.events)
StopIteration
======================================================================
FAIL: test_control_characters (test.test_pyrepl.test_reader.TestReader.test_control_characters)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_pyrepl/test_reader.py", line 126, in test_control_characters
self.assert_screen_equals(reader, 'flag = "🏳️\\u200d🌈"')
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_pyrepl/test_reader.py", line 16, in assert_screen_equals
self.assertListEqual(actual, expected)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: Lists differ: ['flag = "🏳️🌈"'] != ['flag = "🏳️\\u200d🌈"']
First differing element 0:
'flag = "🏳️🌈"'
'flag = "🏳️\\u200d🌈"'
- ['flag = "🏳️🌈"']
+ ['flag = "🏳️\\u200d🌈"']
? +++++++
|
Sorry, something went wrong.
|
Oops! Hmm, I think that needs more attention than just in the tests. The second failure is because the ZWJ is in category Cf and is being erroneously treated as invalid. If I change the predicate to test only for Cc membership, that passes. The first failure looks more serious. The proximate cause is (I think) the missing newline (also in Cc) but an ad-hoc tweak to the predicate to try and let the newline pass through still fails. I think overall this means that a category check might not be the right way to express the intent of the original code. @pablogsal, do you mind chiming in here? My understanding of this check is that it's meant to filter out control characters like what's in C0 and C1, but it seems like some of those characters are not wanted in this predicate, so just testing ord(key) <= 0x1F wouldn't be appropriate either. The \x1d in the test should definitely pass through, for instance. |
Sorry, something went wrong.
tests fail, the current change looks wrong in fact
|
This PR is stale because it has been open for 30 days with no activity. |
Sorry, something went wrong.
|
This PR is stale because it has been open for 90 days with no activity. |
Sorry, something went wrong.
This changeset fixes the faulty predicate identified by #123682 when testing if an input character is in one of the Unicode Other categories
I assume this doesn't need a news fragment, but if I'm wrong, let me know and I'll add one.
(Sorry about the force-push chatter, I started this branch from an unrelated commit 😬)