gh-130804: Fix support of typing unicode chars in pyrepl by sergey-miryanov · Pull Request #130805 · python/cpython · GitHub

Fixed typing of unicode chars (I tested on Cyrillic chars on Windows)

Issue: [Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804

This does fix the issue on Windows. I've also tested on Linux and saw no breakage.

Co-authored-by: Petr Viktorin <encukou@gmail.com>

LGTM

Thanks a lot for the fix @sergey-miryanov

Thanks @sergey-miryanov for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

Sorry, @sergey-miryanov and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780 3.13

Tests have failed.
workflow

How was the PR landed then?

Test were ok in https://github.com/python/cpython/actions/runs/13775445246/job/38523588646

It's odd, it's the same set up yet somehow they are all failing now?

This PR also only modifies test_multiple_statements_fail_early so I don't see how is this related

Reverting doesn't fix anything:

~/github/python/main main ⇡
❯ git revert 7c98b0674daa3e4eb3e8f35afb61a0dba61d1780
[main 46a7449b8e9] Revert "gh-130804: Fix support of typing unicode chars in pyrepl (#130805)"
 3 files changed, 3 insertions(+), 17 deletions(-)
 delete mode 100644 Misc/NEWS.d/next/Core_and_Builtins/2025-03-10-21-46-37.gh-issue-130804.0PpcTx.rst

~/github/python/main main ⇡
❯ make -j -s
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)

~/github/python/main main ⇡
❯ ./python -m test test_repl -vvv
== CPython 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207]
== Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/pablogsal/github/python/main/build/test_python_worker_1544359æ
== CPU count: 36
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1616529274
0:00:00 load avg: 9.94 Run 1 test sequentially in a single process
0:00:00 load avg: 9.94 [1/1] test_repl
test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL
test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok
test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok
test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok
test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok
test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok
test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok
test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok
test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok
test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok
test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok
test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok
test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok

======================================================================
FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early
    self.assertNotIn("afterwards", output)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (heads/main:7c98b0674da, Mar 10 2025, 22:34:19) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n    return self.__get_result()\n           ~~~~~~~~~~~~~~~~~^^\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n    raise self._exception\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n    coro = func()\n  File "<stdin>", line 1, in <module>\n    1 / 0; print(\'afterwards\')\n    ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n                     "__main__", mod_spec)\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n    exec(code, run_globals)\n    ~~~~^^^^^^^^^^^^^^^^^^^\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n    sys.exit(return_code)\n    ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n'

----------------------------------------------------------------------
Ran 13 tests in 1.223s

FAILED (failures=1)
test test_repl failed
0:00:01 load avg: 9.94 [1/1/1] test_repl failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_repl

Total duration: 1.3 sec
Total tests: run=13 failures=1
Total test files: run=1/1 failed=1
Result: FAILURE

Its this PR: 6ab5c4a

Doesn't seem like it:

~/github/python/main main ⇡
❯ git revert 6ab5c4aa05bf35832a3ccd1e71b28b8475fa30f4
[main 1428e541980] Revert "gh-124927: Fix conversion issue between coordinates and position in REPL (#125001)"
 4 files changed, 1 insertion(+), 12 deletions(-)
 delete mode 100644 Misc/NEWS.d/next/Library/2024-10-05-13-25-07.gh-issue-124927.uzNA32.rst

~/github/python/main main ⇡
❯ make
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
./python -E ./Tools/build/generate-build-details.py `cat pybuilddir.txt`/build-details.json

~/github/python/main main ⇡
❯ ./python -m test test_repl -vvv
== CPython 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207]
== Linux-6.13.5-arch1-1-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/pablogsal/github/python/main/build/test_python_worker_1588881æ
== CPU count: 36
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 4038269957
0:00:00 load avg: 3.85 Run 1 test sequentially in a single process
0:00:00 load avg: 3.85 [1/1] test_repl
test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early) ... FAIL
test_toplevel_contextvars_async (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_async) ... ok
test_toplevel_contextvars_sync (test.test_repl.TestAsyncioREPL.test_toplevel_contextvars_sync) ... ok
test_asyncio_repl_is_ok (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_is_ok) ... ok
test_asyncio_repl_reaches_python_startup_script (test.test_repl.TestInteractiveInterpreter.test_asyncio_repl_reaches_python_startup_script) ... ok
test_close_stdin (test.test_repl.TestInteractiveInterpreter.test_close_stdin) ... ok
test_interactive_source_is_in_linecache (test.test_repl.TestInteractiveInterpreter.test_interactive_source_is_in_linecache) ... ok
test_interactive_traceback_reporting (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting) ... ok
test_interactive_traceback_reporting_multiple_input (test.test_repl.TestInteractiveInterpreter.test_interactive_traceback_reporting_multiple_input) ... ok
test_multiline_string_parsing (test.test_repl.TestInteractiveInterpreter.test_multiline_string_parsing) ... ok
test_no_memory (test.test_repl.TestInteractiveInterpreter.test_no_memory) ... ok
test_runsource_show_syntax_error_location (test.test_repl.TestInteractiveInterpreter.test_runsource_show_syntax_error_location) ... ok
test_interactive_syntax_error_correct_line (test.test_repl.TestInteractiveModeSyntaxErrors.test_interactive_syntax_error_correct_line) ... ok

======================================================================
FAIL: test_multiple_statements_fail_early (test.test_repl.TestAsyncioREPL.test_multiple_statements_fail_early)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/Lib/test/test_repl.py", line 304, in test_multiple_statements_fail_early
    self.assertNotIn("afterwards", output)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'afterwards' unexpectedly found in 'asyncio REPL 3.14.0a5+ (bisect/good-ecdf6b15b0c0570c3c3302ab95bdbfd3007ea941-2-g6ab5c4aa05b:6ab5c4aa05b,) [GCC 14.2.1 20250207] on linux\nUse "await" directly instead of "asyncio.run()".\nType "help", "copyright", "credits" or "license" for more information.\n>>> import asyncio\n>>> Traceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 448, in result\n    return self.__get_result()\n           ~~~~~~~~~~~~~~~~~^^\n  File "/home/pablogsal/github/python/main/Lib/concurrent/futures/_base.py", line 393, in __get_result\n    raise self._exception\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 42, in callback\n    coro = func()\n  File "<stdin>", line 1, in <module>\n    1 / 0; print(\'afterwards\')\n    ~~^~~\nZeroDivisionError: division by zero\n>>> \nexiting asyncio REPL...\nTraceback (most recent call last):\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 198, in _run_module_as_main\n    return _run_code(code, main_globals, None,\n                     "__main__", mod_spec)\n  File "/home/pablogsal/github/python/main/Lib/runpy.py", line 88, in _run_code\n    exec(code, run_globals)\n    ~~~~^^^^^^^^^^^^^^^^^^^\n  File "/home/pablogsal/github/python/main/Lib/asyncio/__main__.py", line 204, in <module>\n    sys.exit(return_code)\n    ~~~~~~~~^^^^^^^^^^^^^\nSystemExit: 0\n>>> \n'

----------------------------------------------------------------------
Ran 13 tests in 1.400s

FAILED (failures=1)
test test_repl failed
0:00:01 load avg: 3.85 [1/1/1] test_repl failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_repl

Total duration: 1.5 sec
Total tests: run=13 failures=1
Total test files: run=1/1 failed=1
Result: FAILURE

Weird? Only one left then would be yours a931a8b maybe it has something to do with the traceback modifications?

I'm pretty sure it's a931a8b

#131065

@sergey-miryanov Can you check #130805 (comment) ?

Seems the backport needs some adaptations to 3.13. Could you take a look?

Seems the backport needs some adaptations to 3.13. Could you take a look?

@pablogsal
There is a conflict because a65366e doesn't backported to 3.13 and this PR fixes regression from this commit. Should we backport a65366e first?

That's tricky. That may be considered a feature so I am not sure we can. @ambv what do you think?

python#130805)" This reverts commit 7c98b06.

…pythonGH-130805) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

GH-133462 is a backport of this pull request to the 3.13 branch.

…30805) (GH-133462) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

Fix support of typing unicode chars in pyrepl

3d674cf

sergey-miryanov requested review from ambv, lysnikolaou and pablogsal as code owners March 3, 2025 18:57

bedevere-app Bot added the awaiting review label Mar 3, 2025

bedevere-app Bot mentioned this pull request Mar 3, 2025

[Windows] New REPL doesn't allow to input non-ASCII Unicode characters #130804

Closed

sergey-miryanov added 3 commits March 4, 2025 00:45

Make mypy happy

c1f3aa8

Add assert to check if passed char is str

ef0d0ee

Merge branch 'main' into pythongh-130804 -pyrepl-unicode-chars

497a0bb

encukou reviewed Mar 10, 2025

View reviewed changes

Comment thread Lib/_pyrepl/base_eventqueue.py Outdated

sergey-miryanov and others added 2 commits March 11, 2025 01:06

Update Lib/_pyrepl/base_eventqueue.py

bfa2e4e

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Merge branch 'main' into pythongh-130804 -pyrepl-unicode-chars

c667049

pablogsal approved these changes Mar 10, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting review labels Mar 10, 2025

pablogsal added needs backport to 3.13 bugs and security fixes and removed awaiting merge labels Mar 10, 2025

Add NEWS entry

7cc16ea

pablogsal enabled auto-merge (squash) March 10, 2025 21:48

pablogsal disabled auto-merge March 10, 2025 21:48

pablogsal enabled auto-merge (squash) March 10, 2025 21:48

pablogsal merged commit 7c98b06 into python:main Mar 10, 2025

miss-islington-app Bot assigned pablogsal Mar 10, 2025

sergey-miryanov mentioned this pull request Mar 29, 2025

New REPL on Windows exits when accented character is pasted/typed #131878

Closed

chris-eibl added a commit to chris-eibl/cpython that referenced this pull request Apr 26, 2025

Revert "pythongh-130804 : Fix support of typing unicode chars in pyrepl (

a06c42a

python#130805)" This reverts commit 7c98b06.

chris-eibl mentioned this pull request Apr 26, 2025

gh-131878: Fix input of unicode characters with two or more code points in the REPL on Windows in vt mode #133030

Closed

This was referenced May 5, 2025

gh-131878: Fix input of unicode characters with two or more code points in new pyrepl on Windows #131901

Merged

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe #132440

Merged

ambv pushed a commit to ambv/cpython that referenced this pull request May 5, 2025

[3.13] pythongh-130804 : Fix support of typing unicode chars in pyrepl (…

e980f8b

…pythonGH-130805) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label May 5, 2025

ambv added a commit that referenced this pull request May 5, 2025

[3.13] gh-130804 : Fix support of typing unicode chars in pyrepl (GH-1…

76f52c4

…30805) (GH-133462) (cherry picked from commit 7c98b06) Co-authored-by: Sergey Miryanov <sergey.miryanov@gmail.com>

gh-130804: Fix support of typing unicode chars in pyrepl by sergey-miryanov · Pull Request #130805 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-130804: Fix support of typing unicode chars in pyrepl - #130805

gh-130804: Fix support of typing unicode chars in pyrepl#130805pablogsal merged 7 commits intopython:mainpython/cpython:mainfrom sergey-miryanov:gh-130804-pyrepl-unicode-charssergey-miryanov/cpython:gh-130804-pyrepl-unicode-charsCopy head branch name to clipboard

Conversation

sergey-miryanov commented Mar 3, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

miss-islington-app Bot commented Mar 10, 2025

Uh oh!

miss-islington-app Bot commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

StanFromIreland commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

pablogsal commented Mar 10, 2025

Uh oh!

sergey-miryanov commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Mar 11, 2025

Uh oh!

bedevere-app Bot commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-130804: Fix support of typing unicode chars in pyrepl#130805
pablogsal merged 7 commits into
python:mainfrom
sergey-miryanov:gh-130804-pyrepl-unicode-chars

sergey-miryanov commented Mar 3, 2025 •
edited by bedevere-app Bot
Loading

StanFromIreland commented Mar 10, 2025 •
edited
Loading

sergey-miryanov commented Mar 11, 2025 •
edited
Loading