Repository navigation
Unicode characters ≥ 0x10000 cannot be inputted/behaves unusually at the REPL terminal. #136595
Description
Bug report
Bug description:
When the machine locale is set to UTF-8, when inputting a Unicode character ≥ 0x10000:
In CPython 3.13.5:
https://github.com/user-attachments/assets/7777b063-76fe-4929-b854-cae7d61807d2
In Cpython 3.14.0b4:
>>> Traceback (most recent call last):
File "*\Python\Python314\Lib\_pyrepl\readline.py", line 394, in multiline_input
return reader.readline()
~~~~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 748, in readline
self.handle1()
~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 731, in handle1
self.do_cmd(cmd)
~~~~~~~~~~~^^^^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 661, in do_cmd
self.refresh()
~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 638, in refresh
self.screen = self.calc_screen()
~~~~~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\completing_reader.py", line 261, in calc_screen
screen = super().calc_screen()
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 315, in calc_screen
colors = list(gen_colors(self.get_unicode()))
File "*\Python\Python314\Lib\_pyrepl\utils.py", line 108, in gen_colors
for color in gen_colors_from_token_stream(gen, line_lengths):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "*\Python\Python314\Lib\_pyrepl\utils.py", line 168, in gen_colors_from_token_stream
for prev_token, token, next_token in token_window:
^^^^^^^^^^^^
File "*\Python\Python314\Lib\_pyrepl\utils.py", line 363, in prev_next_window
window = deque((None, next(iterator)), maxlen=3)
~~~~^^^^^^^^^^
File "*\Python\Python314\Lib\tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
for info in it:
^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
>>> Traceback (most recent call last):
File "*\Python\Python314\Lib\_pyrepl\readline.py", line 394, in multiline_input
return reader.readline()
~~~~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 748, in readline
self.handle1()
~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 731, in handle1
self.do_cmd(cmd)
~~~~~~~~~~~^^^^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 661, in do_cmd
self.refresh()
~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 638, in refresh
self.screen = self.calc_screen()
~~~~~~~~~~~~~~~~^^
File "*\Python\Python314\Lib\_pyrepl\completing_reader.py", line 261, in calc_screen
screen = super().calc_screen()
File "*\Python\Python314\Lib\_pyrepl\reader.py", line 315, in calc_screen
colors = list(gen_colors(self.get_unicode()))
File "*\Python\Python314\Lib\_pyrepl\utils.py", line 108, in gen_colors
for color in gen_colors_from_token_stream(gen, line_lengths):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "*\Python\Python314\Lib\_pyrepl\utils.py", line 168, in gen_colors_from_token_stream
for prev_token, token, next_token in token_window:
^^^^^^^^^^^^
File "*\Python\Python314\Lib\_pyrepl\utils.py", line 363, in prev_next_window
window = deque((None, next(iterator)), maxlen=3)
~~~~^^^^^^^^^^
File "*\Python\Python314\Lib\tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
for info in it:
^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
Two surrogates were "inputted" and so two UnicodeEncodeErrors.
CPython versions tested on:
3.13, 3.14
Operating systems tested on:
Windows
Activity
pythongh-136595: Normalize surrogate pairs in REPL input to fix Unico…
WFLing-seaer commented on Nov 9, 2025
I've met this problem too when trying to paste 𣳆㶔𤃫𣸎 to πython running on WinTerminal.
btw in CLI which python command starts pasting the word won't cause crash though it can't be fully pasted.
怎么哪里都有屏被轰坏.jpg
WFLing-seaer commented on Nov 9, 2025
this indicates there must be something wwrong with πython+WinTerminal. maybe compatibility problems.