[Bug]: matplotlib figures inhibit `input` longer than 100 chars in IPython · Issue #25756 · matplotlib/matplotlib · GitHub

Bug summary

After matplotlib activates a plot in IPython, input behavior is changed. In particular, input longer than 100 characters is no longer directly accepted. Since IPython uses input in its REPL loop when --simple-prompt is set, this defeats various modes (such as Emacs) which use this.

Code for reproduction

% python3 -m IPython --simple-prompt
Python 3.11.3 (main, Apr  7 2023, 19:29:16) [Clang 14.0.0 (clang-1400.0.29.202)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: input('Test1: '); print("success!")
Test1: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3
success!

In [2]: import matplotlib.pyplot as plt; plt.plot([1])
Installed osx event loop hook.
Out[2]: [<matplotlib.lines.Line2D at 0x10a704a10>]

In [3]: input('Test2: '); print("success!")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3
# Return does not send the input

Actual outcome

To send the input, an additional newline is required, which is then interpreted as an additional (blank) command. Commands shorter than 100 characters are still directly accepted.

Expected outcome

Python's native input should work the same in IPython before and after activating a matplotlib figure, for arbitrary command lengths (up to the terminals line length limit).

Additional information

Issue posted with IPython as well: ipython/ipython#14030. I have tested disabling enable_gui, but this is not the culprit. To see the bug, you must actually create a figure (but do not need to show it). This does not occur in regular python sessions, only IPython (and possibly only on MacOS?).

Operating system

MacOS 12.6

Matplotlib Version

3.7.1 (same behavior on various versions back to 3.4)

Matplotlib Backend

MacOSX (though same behavior with other backends, e.g. qt6)

Python version

3.11.3 (same behavior back to 3.9)

Jupyter version

n/a

Installation

pip

I have tested disabling enable_gui, but this is not the culprit.

What exactly does this mean? In pyplot we will detect if IPython is imported, and if so run IPython specific code

matplotlib/lib/matplotlib/pyplot.py

Lines 173 to 207 in d77801e

    
           def install_repl_displayhook(): 
        
               """ 
        
               Connect to the display hook of the current shell. 
        
               The display hook gets called when the read-evaluate-print-loop (REPL) of 
        
               the shell has finished the execution of a command. We use this callback 
        
               to be able to automatically update a figure in interactive mode. 
        
               This works both with IPython and with vanilla python shells. 
        
               """ 
        
               global _REPL_DISPLAYHOOK 
        
               if _REPL_DISPLAYHOOK is _ReplDisplayHook.IPYTHON: 
        
                   return 
        
               # See if we have IPython hooks around, if so use them. 
        
               # Use ``sys.modules.get(name)`` rather than ``name in sys.modules`` as 
        
               # entries can also have been explicitly set to None. 
        
               mod_ipython = sys.modules.get("IPython") 
        
               if not mod_ipython: 
        
                   _REPL_DISPLAYHOOK = _ReplDisplayHook.PLAIN 
        
                   return 
        
               ip = mod_ipython.get_ipython() 
        
               if not ip: 
        
                   _REPL_DISPLAYHOOK = _ReplDisplayHook.PLAIN 
        
                   return 
        
               ip.events.register("post_execute", _draw_all_if_interactive) 
        
               _REPL_DISPLAYHOOK = _ReplDisplayHook.IPYTHON 
        
               from IPython.core.pylabtools import backend2gui  # type: ignore 
        
               # trigger IPython's eventloop integration, if available 
        
               ipython_gui_name = backend2gui.get(get_backend()) 
        
               if ipython_gui_name: 
        
                   ip.enable_gui(ipython_gui_name)

which is called at tho bottom of plt.switch_backend.

If you set the backend to Agg (or any of the non-interactive backends) does the problem go away? My guess is that something about the way the GUI eventloop integrates....

iirc IPython patches input.

I can reproduce this on linux with qt6.

I have tested disabling enable_gui, but this is not the culprit.

What exactly does this mean?

I mean I considered that the installation of the interactive event loop by IPython was responsible. So I altered TerminalInteractiveShell::enable_gui to return immediately, without any change in behavior (other than inhibiting the "Installed osx event loop hook." message).

I've also tried:

auditing input: no audit events arrive after the first Return on >100 char input.
tracing all line by line events: no trace events of any kind arrive after the first Return on >100 char input.

So it seems some quite primitive change in input data handling gets activated somehow by matplotlib.

If you set the backend to Agg (or any of the non-interactive backends) does the problem go away?

Yes!

My guess is that something about the way the GUI eventloop integrates....

This was my theory too, but I haven't found the culprit.

ok, good news and bad news: It is indeed the inputhook (good news, we know where the problem is), but it is at the level of the GUI toolkits interacting with PyOSInputHook and readline (bad news, it involves fgets and upstream).

First test is to use PyQt6 which install an inputhook on import but also provides a method to remove / restore it:

✔ 12:13:28 $ python3 -m IPython --simple-prompt
Python 3.10.10 (main, Mar  5 2023, 22:26:53) [GCC 12.2.1 20230201]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import PyQt6

In [2]: import PyQt6.QtCore

In [3]: PyQt6.QtCore.pyqtRemoveInputHook
Out[3]: <function PyQt6.QtCore.pyqtRemoveInputHook>

In [4]: input('Test2: '); print("succes")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravarava
succes

In [5]: import matplotlib.pyplot as plt; plt.plot(range(5))
Installed qt6 event loop hook.
Out[5]: [<matplotlib.lines.Line2D at 0x7f2951060670>]

In [6]: input('Test2: '); print("succes")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3

succes

In [7]:
In [7]: PyQt6.QtCore.pyqtRemoveInputHook()

In [8]: input('Test2: '); print("succes")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3
succes

In [9]: PyQt6.QtCore.pyqtRestoreInputHook()

In [10]: input('Test2: '); print("succes")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3

succes

In [11]:
In [11]:

To confirm, if we force to use PySide6 (which does not even ship with a readline associated inputhook)

✔ 12:21:02 $ python3 -m IPython --simple-prompt
Python 3.10.10 (main, Mar  5 2023, 22:26:53) [GCC 12.2.1 20230201]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import PySide6.QtCore

In [2]: import matplotlib.pyplot as plt; plt.plot(range(5))
Installed qt6 event loop hook.
Out[2]: [<matplotlib.lines.Line2D at 0x7fdf460f19c0>]

In [3]: type(fig.canvas).mro()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 type(fig.canvas).mro()

NameError: name 'fig' is not defined

In [4]: fig = plt.gcf()

In [5]: type(fig.canvas).mro()
Out[5]:
[matplotlib.backends.backend_qtagg.FigureCanvasQTAgg,
 matplotlib.backends.backend_agg.FigureCanvasAgg,
 matplotlib.backends.backend_qt.FigureCanvasQT,
 matplotlib.backend_bases.FigureCanvasBase,
 PySide6.QtWidgets.QWidget,
 PySide6.QtCore.QObject,
 PySide6.QtGui.QPaintDevice,
 Shiboken.Object,
 object]

In [6]: input('Test2: '); print("succes")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3
succes

In [7]:

we do not have the problem.

I also tested with tk (assuming that if the buggy bit was the input hook the tk one would have a better chance of being right).

I have a theory:

Start from https://github.com/python/cpython/blob/385d8d2b6438c7825006df6106dab41a95332a6a/Parser/myreadline.c#L47-L58

    while (1) {
        if (PyOS_InputHook != NULL) {
            (void)(PyOS_InputHook)();
        }


        errno = 0;
        clearerr(fp);
        char *p = fgets(buf, len, fp);
        if (p != NULL) {
            return 0; /* No error */
        }
        int err = errno;

so while waiting for the human run the input hook, the input hooks are expected to exit when they detect a keystroke (or just run for a very short time). If you then look at https://github.com/python/cpython/blob/385d8d2b6438c7825006df6106dab41a95332a6a/Parser/myreadline.c#L304-L322

    do {
        size_t incr = (n > 0) ? n + 2 : 100;
        if (incr > INT_MAX) {
            PyMem_RawFree(p);
            PyEval_RestoreThread(tstate);
            PyErr_SetString(PyExc_OverflowError, "input line too long");
            PyEval_SaveThread();
            return NULL;
        }
        pr = (char *)PyMem_RawRealloc(p, n + incr);
        if (pr == NULL) {
            PyMem_RawFree(p);
            PyEval_RestoreThread(tstate);
            PyErr_NoMemory();
            PyEval_SaveThread();
            return NULL;
        }
        p = pr;
        int err = my_fgets(tstate, p + n, (int)incr, sys_stdin);

that is where the 100 comes from.

My guess as that the story is something like:

you paste in / type a long line
you hit enter, the inputhook exits and python reads the first 100 characters
the inputhook restarts, and just runs
you hit enter again, it sees the new line(s) at the end and returns

Problems with my theory:

I thought inputhooks exited on every keystroke, but apparently only on return (?!)

I suspect the right fix is for in my_fgets for upstream to keep trying to drain the buffer before starting the input hook.

Wow, good sleuthing. And here I was ripgrepping for 100 all around the IPython and matplotlib codebases. Who is installing the input hook? Is there an opportunity to debug on that side?

The PyQt6.QtCore.pyqtRemoveInputHook() trick works here too (but of course disables figure updates).

I tried installing PySide6, but it did not fix things, maybe because it defaulted to the Mac version of the event loop (update: same story when forcing with %gui qt6).

% python3 -m IPython --simple-prompt 
Python 3.11.3 (main, Apr  7 2023, 19:29:16) [Clang 14.0.0 (clang-1400.0.29.202)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import PySide6.QtCore

In [2]: import matplotlib.pyplot as plt; plt.plot(range(5))
Installed osx event loop hook.
Out[2]: [<matplotlib.lines.Line2D at 0x11cb7f790>]

In [3]: input('Test2: '); print("success")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=12345
# double Ret needed
success

In [4]: 
In [4]:

type(fig.canvas).mro() is the most reliable way to say which backend you are using, you can also start with the `MPLBACKEND=qtagg' env set. If you do not set the backend we guess and on mac we try OSX first.

I think this is the input hook for the OSX backend:

matplotlib/src/_macosx.m

Lines 89 to 179 in 52a2a7c

    
           static int wait_for_stdin(void) 
        
           { 
        
               int interrupted = 0; 
        
               const UInt8 buffer[] = "/dev/fd/0"; 
        
               const CFIndex n = (CFIndex)strlen((char*)buffer); 
        
               CFRunLoopRef runloop = CFRunLoopGetCurrent(); 
        
               CFURLRef url = CFURLCreateFromFileSystemRepresentation(kCFAllocatorDefault, 
        
                                                                      buffer, 
        
                                                                      n, 
        
                                                                      false); 
        
               CFReadStreamRef stream = CFReadStreamCreateWithFile(kCFAllocatorDefault, 
        
                                                                   url); 
        
               CFRelease(url); 
        
               CFReadStreamOpen(stream); 
        
           #ifdef PYOSINPUTHOOK_REPETITIVE 
        
               if (!CFReadStreamHasBytesAvailable(stream)) 
        
               /* This is possible because of how PyOS_InputHook is called from Python */ 
        
           #endif 
        
               { 
        
                   int error; 
        
                   int channel[2]; 
        
                   CFSocketRef sigint_socket = NULL; 
        
                   PyOS_sighandler_t py_sigint_handler = NULL; 
        
                   CFStreamClientContext clientContext = {0, NULL, NULL, NULL, NULL}; 
        
                   clientContext.info = runloop; 
        
                   CFReadStreamSetClient(stream, 
        
                                         kCFStreamEventHasBytesAvailable, 
        
                                         _stdin_callback, 
        
                                         &clientContext); 
        
                   CFReadStreamScheduleWithRunLoop(stream, runloop, kCFRunLoopDefaultMode); 
        
                   error = socketpair(AF_UNIX, SOCK_STREAM, 0, channel); 
        
                   if (!error) { 
        
                       CFSocketContext context; 
        
                       context.version = 0; 
        
                       context.info = &interrupted; 
        
                       context.retain = NULL; 
        
                       context.release = NULL; 
        
                       context.copyDescription = NULL; 
        
                       fcntl(channel[0], F_SETFL, O_WRONLY | O_NONBLOCK); 
        
                       sigint_socket = CFSocketCreateWithNative( 
        
                           kCFAllocatorDefault, 
        
                           channel[1], 
        
                           kCFSocketReadCallBack, 
        
                           _sigint_callback, 
        
                           &context); 
        
                       if (sigint_socket) { 
        
                           CFRunLoopSourceRef source = CFSocketCreateRunLoopSource( 
        
                               kCFAllocatorDefault, sigint_socket, 0); 
        
                           CFRelease(sigint_socket); 
        
                           if (source) { 
        
                               CFRunLoopAddSource(runloop, source, kCFRunLoopDefaultMode); 
        
                               CFRelease(source); 
        
                               sigint_fd = channel[0]; 
        
                               py_sigint_handler = PyOS_setsig(SIGINT, _sigint_handler); 
        
                           } 
        
                       } 
        
                   } 
        
                   NSEvent* event; 
        
                   while (true) { 
        
                       while (true) { 
        
                           event = [NSApp nextEventMatchingMask: NSEventMaskAny 
        
                                                      untilDate: [NSDate distantPast] 
        
                                                         inMode: NSDefaultRunLoopMode 
        
                                                        dequeue: YES]; 
        
                           if (!event) { break; } 
        
                           [NSApp sendEvent: event]; 
        
                       } 
        
                       CFRunLoopRun(); 
        
                       if (interrupted || CFReadStreamHasBytesAvailable(stream)) { break; } 
        
                   } 
        
                   if (py_sigint_handler) { PyOS_setsig(SIGINT, py_sigint_handler); } 
        
                   CFReadStreamUnscheduleFromRunLoop( 
        
                       stream, runloop, kCFRunLoopCommonModes); 
        
                   if (sigint_socket) { CFSocketInvalidate(sigint_socket); } 
        
                   if (!error) { 
        
                       close(channel[0]); 
        
                       close(channel[1]); 
        
                   } 
        
               } 
        
               CFReadStreamClose(stream); 
        
               CFRelease(stream); 
        
               if (interrupted) { 
        
                   errno = EINTR; 
        
                   raise(SIGINT); 
        
                   return -1; 
        
               } 
        
               return 1; 
        
           }

and where it is installed:

matplotlib/src/_macosx.m

Line 288 in 52a2a7c

PyOS_InputHook = wait_for_stdin;

(but I confess I do not understand how that line installs it) so we are doing that.

It looks like tk does it when you create the tk application (https://github.com/python/cpython/blob/518050ced18422fd00fadc1a81b0d942b98e2e5b/Modules/_tkinter.c#L689)

For Qt, it does it after some sort of initialization, but I do not understand enough of the sip build chain to understand:

%PostInitialisationCode
// Process events from the input hook.
PyOS_InputHook = qtcore_input_hook;
%End

but given that the function requires using the QApplication (which must be a process singleton) I suspect it is connected just after the application is created.

I do not know off the top of my head what gtk and wx do.

Right you are (below). PySide doesn't seem to process interactive events for me, so I guess it functions like Agg (no input hook?). That seems to indicate that all input hooks are showing this problem (though did you say wx didn't? I can't test that one). Maybe a better argument for an upstream fix?

% MPLBACKEND=qtagg python3 -m IPython --simple-prompt
Python 3.11.3 (main, Apr  7 2023, 19:29:16) [Clang 14.0.0 (clang-1400.0.29.202)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import PySide6.QtCore

In [2]: import matplotlib.pyplot as plt; plt.plot(range(5))
Installed qt6 event loop hook.
Out[2]: [<matplotlib.lines.Line2D at 0x120381f10>]

In [3]: type(plt.gcf().canvas).mro()
Out[3]: 
[matplotlib.backends.backend_qtagg.FigureCanvasQTAgg,
 matplotlib.backends.backend_agg.FigureCanvasAgg,
 matplotlib.backends.backend_qt.FigureCanvasQT,
 matplotlib.backend_bases.FigureCanvasBase,
 PySide6.QtWidgets.QWidget,
 PySide6.QtCore.QObject,
 PySide6.QtGui.QPaintDevice,
 Shiboken.Object,
 object]

In [4]: input('Test2: '); print("success!")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3
success!

In [5]:

I'm also confused as to why this works without issue:

% python3
Python 3.11.3 (main, Apr  7 2023, 19:29:16) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.pyplot as plt
>>> plt.ion()
<contextlib.ExitStack object at 0x101ff8050>
>>> plt.plot(range(5))
[<matplotlib.lines.Line2D object at 0x105082710>]
>>> input('Test2: '); print("success!")
Test2: avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3
'avaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravaravar=3'
success!
>>>

Where is the figure event loop running if not via an input hook? Corollary: if there's no input hook, why are figure and prompt both responsive?

Wow, good sleuthing.

I wish I could say this was my first round of debugging this sort of issue 🤣 .

Sorry, I meant to say "I do not know what gtk and wx do about PyOS_InputHook".

Maybe a better argument for an upstream fix?

Yes, I am convinced that this is an upstream CPython bug and should be fixed there.

PySide doesn't seem to process interactive events for me, so I guess it functions like Agg (no input hook?).

It does not process events while waiting for the human, if you do plt.show() it will run the event loop in a blocking way and be live (but it will block your prompt for any other work). See https://matplotlib.org/stable/users/explain/interactive_guide.html#eventloop-integration-mechanism for more details.

I'm also confused as to why this works without issue:

For me with PyQt6 I also see the 100 character limit issue unless I do PyQt6.QtCore.pyqtRemoveInputHook() (and if I do so the figure is not interactive whille input is blocking the prompt) in prompt toolkit IPython. So, I am also confused why it works!

It must just be a matter of whether the input hook gets installed, which I admit I'm quite hazy on (between IPython automatic and command line, matplotlib's auto-backend-selecting, %gui, %matplotlib, plt.ion, etc.). It would seem an odd coincidence that all of the input hooks are misbehaving in the same way (tk too?). Which I guess is further support of your assessment that this is an upstream issue with the way these hooks are called.

One definitive test might be to change the 100 to say 150, and see if the limit responds in kind for various backends. Are you in a position to compile a CPython with such an edit? Thanks for your work on this.

ok, things I now understand:

for PyOS_InputHook the hook waits for stdin to be ready to read not for a key to be pressed (like the the prompt toolkit input hooks)
fgets blocks until the number of characters asked for, a new line of EOF (https://www.ibm.com/docs/en/i/7.4?topic=functions-fgets-read-string)
putting the PyOS_InputHook at the bottom of that loop fixes the problem....but breaks the input hook while waiting.

This patch seems to fix the problem (but segfaults on Python exit which may or may not be related)

diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index 513856d8a4..aa5a5e4273 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -167,6 +167,9 @@ the same library that the Python runtime is using.
    event loops, as done in the :file:`Modules/_tkinter.c` in the
    Python source code.

+   This function should block until stdin is ready to be read and
+   ``fgets`` will return immediately.
+

 .. c:var:: char* (*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, const char *)

diff --git a/Parser/myreadline.c b/Parser/myreadline.c
index 3f0e29f051..417908ada1 100644
--- a/Parser/myreadline.c
+++ b/Parser/myreadline.c
@@ -30,7 +30,7 @@ int (*PyOS_InputHook)(void) = NULL;
    except if _PyOS_InterruptOccurred() returns true. */

 static int
-my_fgets(PyThreadState* tstate, char *buf, int len, FILE *fp)
+my_fgets(PyThreadState* tstate, char *buf, int len, FILE *fp, int n)
 {
 #ifdef MS_WINDOWS
     HANDLE handle;
@@ -45,7 +45,7 @@ my_fgets(PyThreadState* tstate, char *buf, int len, FILE *fp)
 #endif

     while (1) {
-        if (PyOS_InputHook != NULL) {
+        if (PyOS_InputHook != NULL && n == 0) {
             (void)(PyOS_InputHook)();
         }

@@ -319,7 +319,7 @@ PyOS_StdioReadline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt)
             return NULL;
         }
         p = pr;
-        int err = my_fgets(tstate, p + n, (int)incr, sys_stdin);
+        int err = my_fgets(tstate, p + n, (int)incr, sys_stdin, n);
         if (err == 1) {
             // Interrupt
             PyMem_RawFree(p);

Building CPython is far easier than building packages in the scipy space. https://github.com/tacaswell/build_the_world is set of scripts I use to try and build everything from CPython up, I do not think I have ever had building CPython fail ....

If fgets blocks waiting for \n or len chars, doesn't that mean the input hook runs only once (per increment), then fgets waits for (up to) 100 characters to arrive? That doesn't seem consistent with figure event behavior during command creation. For example the x,y readout on plots certainly works in the midst of typing a long command. I.e., I'd expect the while(1) loop to be spinning fast. Maybe it relies on fgets returning error/EOF over and over, until data are actually present on stdin? Or maybe data only get presented on the stdin stream line by line. Hmm... confusing.

doesn't that mean the input hook runs only once (per increment), then fgets waits for (up to) 100 characters to arrive?

Yes, but the order is:

PyOS_InputHook runs (forever) which spins the UI event loop
stdin declares itself ready to read and via the UI event loop stops the UI event loop
fgets reads (up to) 100 characters (or n+2 in subsequent calls)

I think it most operations that while(1) loop only actually runs once and returns very early.

hiting ctrl-D is what sends EOF, if there is nothing there fgets just waits.

Digging into this a bit more, it seems to be the case that when starting Python with a prompt the stdin is line buffered so things only show up as readable in new-line terimnated chunks. I think this means my proposed fix will behave very badly if you turn that off.

xref https://discuss.python.org/t/unbuffered-stdin-bug-in-docs-or-implementation/20361/24

Also https://www.gnu.org/software/libc/manual/html_node/I_002fO-on-Streams.html for understading what the c code is doing with the stream.

Oh I think I see. So the input hook is responsible for using select or similar on stdin, and during its event loop it only returns when there is something for fgets to read, which will either include a newline or "enough" characters as requested. If there are more than enough character data, the input hook is called again, but this time it doesn't break until yet more data are available (i.e. another newline).

Most terminals are line-buffered, but you could try !stty raw to see how bad it goes. I've occasionally used a pipe instead of TTY to communicate with iPython; the gui event loop certainly won't run in that case (though I wish it would; it's much faster than TTY's on MacOS).

Just from the (impressed!) peanut gallery:

(but I confess I do not understand how that line installs it)

https://docs.python.org/3/c-api/veryhigh.html#c.PyOS_InputHook seems to say that the API to set the inputhook is literally just to overwrite the global variable.

My brain is apparently sufficiently warped by Python I did not grok that was a global 🤦🏻 .

I'm trying to get a minimal reproduction example before going upstream with this bug + patch. The following is about as minimal of a Tk app as I think is rich enough to show everything working

from tkinter import *
from tkinter import ttk


def run():
    root = Tk()
    frm = ttk.Frame(root, padding=10)
    frm.grid()
    lbl = ttk.Label(frm, text="Hello World!")
    lbl.grid(column=0, row=0)

    j = 0

    def set_label():
        nonlocal j
        j += 1
        lbl["text"] = f"push count = {j}"

    ttk.Button(frm, text="Push Me!", command=set_label).grid(column=1, row=0)

    return root, frm

however if you just fire up the plain Python prompt it will use the readline version of the terminal + input which does not have the bug. Still working on how to reproduce without using IPython....

Hmm. What happens if you unload the readline module and use input?

once you can type Python at the interactive Python shell it is too late (the readline modules registers itself as the provider of interactive stdin functions on module import and you can only do that registration via the c-api). I did not track through where Python tries to import readline on startup if it detects it is in interactive mode at a tty, but I suspect it is "early".

I got a minimal reproducer working (ok, not strictly minimal but it is self-describing):

import string
import sys
from tkinter import Tk
from tkinter import ttk


def run():
    """
    This sets up a minimal tk application that has enough functionality
    to verify it is "live" and the inputhook is running while waiting for
    user input.
    """
    root = Tk()
    frm = ttk.Frame(root, padding=10)
    frm.grid()
    lbl = ttk.Label(frm, text="push count = 0")
    lbl.grid(column=0, row=0)

    j = 0

    def set_label():
        nonlocal j
        j += 1
        lbl["text"] = f"push count = {j}"

    ttk.Button(frm, text="Push me!", command=set_label).grid(column=1, row=0)

    return root, frm


run()

print("This is a demo of a bug in the non-readline based stdio code\n\n")
print(f"The readline module is not loaded {'readline' in sys.modules}")

test_string = (string.ascii_lowercase + string.ascii_uppercase) * 2

print(
    f"""
We are using the test string :

\t{test_string}

as it is easy to eyeball the length (it is 104 characters long in 26
character blocks).

You should see a tk window with a button that says "Push me!" and a counter.
Pushing the button should increment the counter.

Follow the instructions to demonstrate the bug.


"""
)

print(
    f"""
To see a case where it works paste

{test_string[:10]}

into the prompt below.  Before hitting return, try pushing
the button on the UI to verify that the inputhook is running.
"""
)

a = input("paste here >> ")
print(f"You pasted {a}")


print(
    f"""
To see it fail past the full string

{test_string}

into the prompt below (you will have to hit enter twice)
"""
)

a = input("paste here >> ")
print(f"You pasted {a}")

print("There is still a newline in the buffer, this input will be 'skipped'\n")
a = input("you can not input here >> ")
print(f"we got an empty string!: {a=!r} (also note no new line in stdout)")


print(
    f"""

you can now play with it or ctrl-d to exit.

The longest string that works is:

{test_string[:98]}

"""
)

while True:
    print("\n")
    a = input("test input >> ")
    print(f"what you entered: {a=}")

I'm going to go ahead and open a bug (and PR) upstream with this.

python/cpython#103931 is the upstream PR.

I am going to close this as it requires and upstream fix and I do not think there is anything else we can do at the Matplotlib (or IPython) level to address this.

I am discouraged by finding 15 year old PyOS_InputHook bugs that never got traction. In the meantime, can you think of any practical work-arounds, aside from unsetting/resetting the input hook around long commands?

Update: as a "hammer a nail using an accordion" approach, you can import readline and then input will "use it to provide elaborate line editing and history features", which seems to circumvent the problem, though it also produces some spurious prompts.

[Bug]: matplotlib figures inhibit `input` longer than 100 chars in IPython · Issue #25756 · matplotlib/matplotlib · GitHub

Repository navigation

[Bug]: matplotlib figures inhibit input longer than 100 chars in IPython #25756

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def install_repl_displayhook():
	"""
	Connect to the display hook of the current shell.

	The display hook gets called when the read-evaluate-print-loop (REPL) of
	the shell has finished the execution of a command. We use this callback
	to be able to automatically update a figure in interactive mode.

	This works both with IPython and with vanilla python shells.
	"""
	global _REPL_DISPLAYHOOK

	if _REPL_DISPLAYHOOK is _ReplDisplayHook.IPYTHON:
	return

	# See if we have IPython hooks around, if so use them.
	# Use ``sys.modules.get(name)`` rather than ``name in sys.modules`` as
	# entries can also have been explicitly set to None.
	mod_ipython = sys.modules.get("IPython")
	if not mod_ipython:
	_REPL_DISPLAYHOOK = _ReplDisplayHook.PLAIN
	return
	ip = mod_ipython.get_ipython()
	if not ip:
	_REPL_DISPLAYHOOK = _ReplDisplayHook.PLAIN
	return

	ip.events.register("post_execute", _draw_all_if_interactive)
	_REPL_DISPLAYHOOK = _ReplDisplayHook.IPYTHON

	from IPython.core.pylabtools import backend2gui # type: ignore
	# trigger IPython's eventloop integration, if available
	ipython_gui_name = backend2gui.get(get_backend())
	if ipython_gui_name:
	ip.enable_gui(ipython_gui_name)

	static int wait_for_stdin(void)
	{
	int interrupted = 0;
	const UInt8 buffer[] = "/dev/fd/0";
	const CFIndex n = (CFIndex)strlen((char*)buffer);
	CFRunLoopRef runloop = CFRunLoopGetCurrent();
	CFURLRef url = CFURLCreateFromFileSystemRepresentation(kCFAllocatorDefault,
	buffer,
	n,
	false);
	CFReadStreamRef stream = CFReadStreamCreateWithFile(kCFAllocatorDefault,
	url);
	CFRelease(url);

	CFReadStreamOpen(stream);
	#ifdef PYOSINPUTHOOK_REPETITIVE
	if (!CFReadStreamHasBytesAvailable(stream))
	/* This is possible because of how PyOS_InputHook is called from Python */
	#endif
	{
	int error;
	int channel[2];
	CFSocketRef sigint_socket = NULL;
	PyOS_sighandler_t py_sigint_handler = NULL;
	CFStreamClientContext clientContext = {0, NULL, NULL, NULL, NULL};
	clientContext.info = runloop;
	CFReadStreamSetClient(stream,
	kCFStreamEventHasBytesAvailable,
	_stdin_callback,
	&clientContext);
	CFReadStreamScheduleWithRunLoop(stream, runloop, kCFRunLoopDefaultMode);
	error = socketpair(AF_UNIX, SOCK_STREAM, 0, channel);
	if (!error) {
	CFSocketContext context;
	context.version = 0;
	context.info = &interrupted;
	context.retain = NULL;
	context.release = NULL;
	context.copyDescription = NULL;
	fcntl(channel[0], F_SETFL, O_WRONLY \| O_NONBLOCK);
	sigint_socket = CFSocketCreateWithNative(
	kCFAllocatorDefault,
	channel[1],
	kCFSocketReadCallBack,
	_sigint_callback,
	&context);
	if (sigint_socket) {
	CFRunLoopSourceRef source = CFSocketCreateRunLoopSource(
	kCFAllocatorDefault, sigint_socket, 0);
	CFRelease(sigint_socket);
	if (source) {
	CFRunLoopAddSource(runloop, source, kCFRunLoopDefaultMode);
	CFRelease(source);
	sigint_fd = channel[0];
	py_sigint_handler = PyOS_setsig(SIGINT, _sigint_handler);
	}
	}
	}

	NSEvent* event;
	while (true) {
	while (true) {
	event = [NSApp nextEventMatchingMask: NSEventMaskAny
	untilDate: [NSDate distantPast]
	inMode: NSDefaultRunLoopMode
	dequeue: YES];
	if (!event) { break; }
	[NSApp sendEvent: event];
	}
	CFRunLoopRun();
	if (interrupted \|\| CFReadStreamHasBytesAvailable(stream)) { break; }
	}

	if (py_sigint_handler) { PyOS_setsig(SIGINT, py_sigint_handler); }
	CFReadStreamUnscheduleFromRunLoop(
	stream, runloop, kCFRunLoopCommonModes);
	if (sigint_socket) { CFSocketInvalidate(sigint_socket); }
	if (!error) {
	close(channel[0]);
	close(channel[1]);
	}
	}
	CFReadStreamClose(stream);
	CFRelease(stream);
	if (interrupted) {
	errno = EINTR;
	raise(SIGINT);
	return -1;
	}
	return 1;
	}

[Bug]: matplotlib figures inhibit `input` longer than 100 chars in IPython · Issue #25756 · matplotlib/matplotlib · GitHub

Repository navigation

[Bug]: matplotlib figures inhibit input longer than 100 chars in IPython #25756

Description

Bug summary

Code for reproduction

Actual outcome

Expected outcome

Additional information

Operating system

Matplotlib Version

Matplotlib Backend

Python version

Jupyter version

Installation

Activity

tacaswell commented on Apr 24, 2023

tacaswell commented on Apr 24, 2023

jdtsmith commented on Apr 24, 2023

jdtsmith commented on Apr 24, 2023

tacaswell commented on Apr 24, 2023

jdtsmith commented on Apr 24, 2023

tacaswell commented on Apr 24, 2023

jdtsmith commented on Apr 24, 2023

jdtsmith commented on Apr 24, 2023

tacaswell commented on Apr 24, 2023

tacaswell commented on Apr 24, 2023

jdtsmith commented on Apr 25, 2023

tacaswell commented on Apr 25, 2023

jdtsmith commented on Apr 25, 2023

tacaswell commented on Apr 25, 2023

jdtsmith commented on Apr 25, 2023

anntzer commented on Apr 25, 2023

tacaswell commented on Apr 25, 2023

tacaswell commented on Apr 27, 2023

jdtsmith commented on Apr 27, 2023

tacaswell commented on Apr 27, 2023

tacaswell commented on Apr 27, 2023

tacaswell commented on Apr 27, 2023

jdtsmith commented on May 7, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions