GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe by chris-eibl · Pull Request #132440 · python/cpython · GitHub

E.g. on my keyboard with German layout, { is usually entered via pressing the AltGr key and 7, i.e. AltGr+7.

Likewise, }, [, ], \ and some more can only be entered via AltGr.

But since #128388 / #128389 these are swallowed by the REPL on Windows and can no longer be entered.

This happens in legacy Windows terminals, where the virtual terminal mode is turned off (e.g. cmd.exe).

In virtual terminal mode there are other issues, see #131878.

Issue: New REPL on Windows swallows characters entered via AltGr #132439

A missed opportunity when main was merged during development of #124119.

Which @sergey-miryanov takes care of here as part of #131901.

So maybe I should remove this one, but I'd really like to keep the other two raw_key changes (not only because I'd have to adapt almost all tests :)

IMHO you should merge my branch here :)

I can undo the change - then you won't have conflicts. I don't think we should (partially) merge between our two PRs?

Yeah, I think you are right - we shouldn't merge our PRs. You can keep your changes - I'm OK if here will be conflict.

Looking at the diff , previously

return Event(
    evt="key", data=code, raw=rec.Event.KeyEvent.uChar.UnicodeChar
)

was used, which in the new code should have been raw_key?

Actually, I was wondering why this change did not break anything, but AFAICT the raw member is not used in the whole code base except

cpython/Lib/_pyrepl/unix_console.py

Line 513 in 5d8e432

def getpending(self):

where the two getpending implementations dutifully respect e.raw += e.raw :)

When searching in the code base for getpending I found

cpython/Lib/_pyrepl/windows_console.py

Lines 518 to 521 in 5d8e432

    
               def getpending(self) -> Event: 
        
                   """Return the characters that have been typed but not yet 
        
                   processed.""" 
        
                   return Event("key", "", b"")

which clearly seems to be a bug to me? Because even though WindowsConsole._read_input() only reads one Windows INPUT_RECORD per call, get_event could have put something into self.event_queue.

I've addressed this in https://github.com/python/cpython/pull/132889/files#r2059235071.

Adding @vstinner, @eendebakpt and @paulie4 , since they were involved in #128389 and already talked about AltGr.

PS: switching to an english keyboard layout, I can use the AltGr key like right-Alt, since it is not used in this keyboard layout :)

The failing of Ubuntu (free-threading) is definitely unrelated, since this is a Windows specific change.

Do you think it'd be possible to add some tests for this change to ensure we don't accidentally regress again?

Yeah, I've already thought about it, but am still unsure how to best do it.

ATM, I am thinking of mocking

cpython/Lib/_pyrepl/windows_console.py

Line 411 in b87189d

def _read_input(self, block: bool = True) -> INPUT_RECORD | None:

to be able to test

cpython/Lib/_pyrepl/windows_console.py

Lines 426 to 432 in b87189d

    
               def get_event(self, block: bool = True) -> Event | None: 
        
                   """Return an Event instance.  Returns None if |block| is false 
        
                   and there is no event pending, otherwise waits for the 
        
                   completion of an event.""" 
        
                   while self.event_queue.empty(): 
        
                       rec = self._read_input(block)

AFAICT, all the existing tests just mock get_event, but for this, IMHO get_event itself must be tested.
Mocking _read_input seems to be the best way forward.

WDYT?

Agreed that we shouldn't be mocking get_event since that's where we're making changes. _read_input seems like a good option, we can capture some real inputs and then replay them with it.

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

because it will be part of 131901, anywayy

because it is unused

Excellent work again! Very well thought through.

Thanks @chris-eibl for the PR, and @ambv for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

Sorry, @chris-eibl and @ambv, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 07f416a3f063db6b91b8b99ff61a51b64b0503f1 3.13

Will repeat here too - problem with backport because VT support was not backported #130805 (comment) (for completeness)

Yeah, I am backporting the VT support too because it's the only way we can have a sensibly similar codebase for bug fixes.

…ltGr on cmd.exe (pythonGH-132440) (cherry picked from commit 07f416a) Co-authored-by: Chris Eibl <138194463+chris-eibl@users.noreply.github.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

GH-133460 is a backport of this pull request to the 3.13 branch.

…n cmd.exe (GH-132440) (GH-133460) (cherry picked from commit 07f416a) Co-authored-by: Chris Eibl <138194463+chris-eibl@users.noreply.github.com> Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

… cmd.exe (pythonGH-132440) Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

chris-eibl added 2 commits April 12, 2025 16:01

do not swallow characters that have been entered via AltGr

59d71b0

blurb it

cca6155

chris-eibl requested review from ambv, lysnikolaou and pablogsal as code owners April 12, 2025 14:31

bedevere-app Bot added the awaiting review label Apr 12, 2025

bedevere-app Bot mentioned this pull request Apr 12, 2025

New REPL on Windows swallows characters entered via AltGr #132439

Closed

chris-eibl commented Apr 12, 2025

View reviewed changes

Comment thread Lib/_pyrepl/windows_console.py

chris-eibl added OS-windows 3.13 bugs and security fixes 3.14 bugs and security fixes topic-repl Related to the interactive shell labels Apr 12, 2025

unrelated stuff worth to do (?)

1e61540

chris-eibl commented Apr 12, 2025

View reviewed changes

Comment thread Lib/_pyrepl/windows_console.py Outdated

chris-eibl requested a review from vstinner April 12, 2025 14:49

picnixz added needs backport to 3.13 bugs and security fixes and removed 3.13 bugs and security fixes 3.14 bugs and security fixes labels Apr 12, 2025

chris-eibl added 3 commits April 19, 2025 08:30

Merge remote-tracking branch 'origin/main' into fix_altgr

c60a727

fix missing con.restore() in test_multiline_ctrl_z

ed035e5

add WindowsConsoleGetEventTests

69e8779

chris-eibl commented Apr 20, 2025

View reviewed changes

Comment thread Lib/test/test_pyrepl/test_windows_console.py

This was referenced Apr 24, 2025

Pasting in new REPL is slow #130328

Closed

GH-130328: pasting in new REPL is slow on Windows #132884

Merged

chris-eibl commented Apr 25, 2025

View reviewed changes

Comment thread Lib/test/test_pyrepl/test_windows_console.py Outdated

chris-eibl and others added 6 commits April 25, 2025 21:40

Apply suggestions from code review

e67452b

Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

Update Lib/test/test_pyrepl/test_windows_console.py

396f48e

Merge branch 'main' into fix_altgr

40b339a

revert change in virtual terminal path

1fbe240

because it will be part of 131901, anywayy

Merge remote-tracking branch 'origin/main' into fix_altgr

2d59563

remove raw part of Event in get_event() in case of windows_console

adb89ba

because it is unused

chris-eibl commented May 5, 2025

View reviewed changes

Comment thread Lib/test/test_pyrepl/test_windows_console.py

ambv changed the title GH-132439: REPL on Windows swallows characters entered via AltGr GH-132439: Fix REPL swallowing characters entered with AltGr May 5, 2025

ambv approved these changes May 5, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels May 5, 2025

ambv merged commit 07f416a into python:main May 5, 2025

bedevere-app Bot removed the awaiting merge label May 5, 2025

miss-islington-app Bot assigned ambv May 5, 2025

ambv changed the title GH-132439: Fix REPL swallowing characters entered with AltGr GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe May 5, 2025

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label May 5, 2025

chris-eibl deleted the fix_altgr branch May 5, 2025 18:14

eendebakpt mentioned this pull request Jun 3, 2025

Dead keys on Windows with German keyboard layout #135105

Closed

Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025

pythonGH-132439 : Fix REPL swallowing characters entered with AltGr on…

bace14d

… cmd.exe (pythonGH-132440) Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>

chris-eibl mentioned this pull request Mar 13, 2026

Event.raw being added to itself instead of to another event's .raw in UnixConsole.getpending #145886

Closed

                           elif self.__vt_support:
                               # If virtual terminal is enabled, scanning VT sequences
-                              self.event_queue.push(rec.Event.KeyEvent.uChar.UnicodeChar)
+                              self.event_queue.push(raw_key)

+                                      self.event_queue.insert(Event(evt="key", data=key, raw=raw_key))
                                       return Event(evt="key", data="\033")  # keymap.py uses this for meta
-                                  return Event(evt="key", data=key, raw=key)
+                                  return Event(evt="key", data=key, raw=raw_key)

	def getpending(self) -> Event:
	"""Return the characters that have been typed but not yet
	processed."""
	return Event("key", "", b"")

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe by chris-eibl · Pull Request #132440 · python/cpython · GitHub

Uh oh!

Uh oh!

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe - #132440

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe#132440ambv merged 13 commits intopython:mainpython/cpython:mainfrom chris-eibl:fix_altgrchris-eibl/cpython:fix_altgrCopy head branch name to clipboard

Conversation

chris-eibl commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

chris-eibl Apr 12, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eibl Apr 20, 2025

Choose a reason for hiding this comment

Uh oh!

sergey-miryanov Apr 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chris-eibl Apr 20, 2025

Choose a reason for hiding this comment

Uh oh!

sergey-miryanov Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eibl Apr 12, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eibl Apr 20, 2025

Choose a reason for hiding this comment

Uh oh!

chris-eibl Apr 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chris-eibl commented Apr 12, 2025

Uh oh!

chris-eibl commented Apr 12, 2025

Uh oh!

chris-eibl commented Apr 12, 2025

Uh oh!

tomasr8 commented Apr 17, 2025

Uh oh!

chris-eibl commented Apr 17, 2025

Uh oh!

tomasr8 commented Apr 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ambv left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington-app Bot commented May 5, 2025

Uh oh!

miss-islington-app Bot commented May 5, 2025

Uh oh!

sergey-miryanov commented May 5, 2025

Uh oh!

ambv commented May 5, 2025

Uh oh!

bedevere-app Bot commented May 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

GH-132439: Fix REPL swallowing characters entered with AltGr on cmd.exe#132440
ambv merged 13 commits into
python:mainfrom
chris-eibl:fix_altgr

chris-eibl commented Apr 12, 2025 •
edited
Loading

sergey-miryanov Apr 20, 2025 •
edited
Loading

chris-eibl Apr 20, 2025 •
edited
Loading