Terminal interface using prompt_toolkit? · Issue #8489 · ipython/ipython · GitHub

prompt_toolkit is a pure-Python terminal interface library aimed at creating interactive consoles. ptpython is a Python console built with it, including ptipython which uses IPython internally. Unlike most terminal interface things, it also works on Windows.

We are considering switching our primary terminal interface from readline to prompt_toolkit, because:

readline is now rather limiting: we can't do syntax highlighting or proper multiline editing; showing tab completions permanently adds them to the terminal scrollback buffer. We can provide a much better user experience with something like prompt_toolkit.
Depending on readline is a pain beyond Linux: we use pyreadline on Windows, but that's essentially unmaintained and is about to break with Python 3.5. Macs ship with libedit, which is theoretically equivalent but has bugs, so we rely on the third party gnureadline package.

We'll probably need to maintain a simpler fallback mode for situations where prompt_toolkit may not work (@fperez thought ssh-ing to old servers was a likely example).

@jonathanslenders : how much work do you think it would be to convert our existing readline + input() interface to prompt_toolkit? I know you've already made ptipython, and I've started having a look at that, but I'm not clear how much of that code we'd need and how much duplicates things we already do.

Hi @takluyver,

I'm very happy to hear about this.
ptipython, while nice, is not the future. (I don't like the current integration with IPython that I did myself.)

What you need for the repl is not much, probably somewhere about 500 or 1000 lines of code to have a nice front-end for IPython that does all the highlighting and autocompletion.

Basically, we need to have one place in IPython where we read the input. That's a blocking call that runs the input interface, and when the user accepts the input, it returns the Python code as a string.

One issue though is that IPython runs its own event loop. (It has several, I believe.) This is mainly important for rendering the mathplotlib graphs. The challenge is to be able to run prompt_toolkit on these event loops. I think it should be possible. Much of the architecture of prompt_toolkit has improved a lot in the last months: now it's easy to run such an interface on any event loop.

I am very willing to help with the integration. The last few weeks I have been working on a few major refactorings in prompt_toolkit. (many performance improvements + much better I/O abstraction.) I will finish this first. It has to be really stable, and then I can help with IPython.

One question. What do you mean by ssh-ing to old servers? SSH does not care about which terminal the client uses. It's only an environment variable passed through. But I heard that the mosh shell on the other hand, doesn't like prompt_toolkit. (I think it makes some invalid assumptions about the program running in the terminal.)

Jonathan

Awesome, thanks! We also have some other stuff to do before we dive into this (4.0 will be mainly a big reorganisation, with no major new features), so there's no rush to do this immediately.

That sounds like it will fit into our architecture well: we already have one relatively isolated piece that reads the input - that's the 60 lines or so starting here.

Good point about running with other event loops. Our current strategy is to hook PyOS_InputHook up to the different event loops so that it gets called regularly to process new input. It's definitely one of the more delicate pieces of the codebase. I expect we can work something out.

I think we were just generally guessing that there would be some situations in which fancy terminal interfaces wouldn't work properly, so we'd need a plainer fallback. Maybe this isn't necessary at all - have you come across anything like that?

Please pull me in for mpl related issues.

We just added a bit tighter integration with the IPython event loops (matplotlib/matplotlib#4091).

@tacaswell , I think the question is more around IPython's integration with the different event loops for mpl backends, that make it possible to keep display and interact with figures without blocking IPython - that PR looks like something a bit different.

It looks like the way we currently do it is specific to Python's readline bindings, so we'll probably have to do that integration another way with prompt_toolkit - though it sounds like that might actually be easier than the current situation.

Hi all.
Today I pushed a new release of both prompt_toolkit and ptpython. Windows support has been improved significantly. It feels almost like Unix, especially with a console like ConEmu.

The simplest example of a blocking call that reads Python input from the user and returns a string can be found in this example: https://github.com/jonathanslenders/ptpython/blob/master/examples/python-input.py

For the intergration, we would need to write a custom eventloop, or a simple adaptor that allows one eventloop to run "on top" of another. Keep in mind that Windows and Unix systems use a different event loop. I did not had the time yet to invest in the integration, and the coming weeks I'm on holiday. But hopefully we can do it this summer, that would be nice.

One thing that you can already think of is how much of ptpython you'd like to see for an IPython front-end. The colorschemes? The current configuration menu? Jedi completions? Docstrings? Function signatures? A custom status bar, showing IPython specific stuff? If we share functionality with ptpython, do we also share code? ptpython is about 2300 lines right now.

Jonathan

Thanks, I'll dig into that example a bit and think about how we can integrate with it. For a first pass, I imagine we wouldn't use that much of the ptpython functionality: we have our own equivalents of some of it, and we're focussing on the notebook at present. But once it's in place and working, I'm sure there will be pressure to do more interesting things with it.

Enjoy your holiday - we're still concentrating on getting 4.0 out, so there's no rush. I look forward to working with you on this.

Thanks!

A small update, I still found some time. Here is an example of prompt-toolkit integration in the GTK event loop: prompt-toolkit/python-prompt-toolkit#134
I guess I'll do some more improvements, clean up and add Windows support. But that's all very easy.

We don't even have to patch PyOS_InputHook like IPython is currectly doing.

Jonathan

Update.
I tried another approach after realising that creating custom EventLoop classes as adaptors for each GUI tool kit would be difficult. It could result in a big change in the IPython code and maybe for some event loops it would not even be feasible to insert the objects to wait for. (Especially windll.kernel32.CreateEventA, which doesn't work together with a select-based event loop.)

So I added input hooks to prompt-toolkit, exactly like Python has PyOS_InputHook. This should make the change in IPython rather small, and clean up some ctypes stuff.
Have a look at this example: https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/examples/inputhook.py
and this file: https://github.com/jonathanslenders/python-prompt-toolkit/blob/master/prompt_toolkit/eventloop/inputhook.py
Everything is in this commit:
prompt-toolkit/python-prompt-toolkit@f0586b1

Required changes on the IPython side:

The IPython.lib.inputhook.stdin_ready* should be replaced by something that uses InputhookContext.input_is_ready or InputHookContext.fileno(). It's platform independent. (The prompt-toolkit eventloops handle the platform specific differences.)
InputHookManager.clear_input_hook and InputHookManager.set_input_hook should be updated to set a callable which can be passed to get_input. That callable receives such an InputHookContext which can be tested for readyness.

So, the examples that I tested seem to work very well. Tested on both Windows and Linux.

That sounds very promising, thanks @jonathanslenders!

Thanks, that looks really nice. Quick question: it looks like inputhook is passed when the event loop is created. Is it easy to support adding an inputhook later? Or to switch to a new event loop between inputs? Our current system allows you to do e.g. %gui qt in a running IPython session and add the inputhook. Losing that would probably be acceptable given other benefits, but I think it should be simple.

That should be easy.

Just noticed that integration with tk needed a bit more work. By default, tk already sets PyOS_InputHook itself, so there was not much to do before. With prompt_toolkit however, that would not prevent blocking the UI. So I had to make an ipython.lib.inputhooktk.py file. Something like this: (I need to do some more testing later this week, or after my holiday, but I think it's rather easy what still has to be done.)

import _tkinter

def inputhook_tk(app, inputhook_context):
    # Add a handler that sets the stop flag when `prompt-toolkit` has input to
    # process.
    stop = [False]
    def done(*a):
        stop[0] = True

    app.createfilehandler(inputhook_context.fileno(), _tkinter.READABLE, done)

    # Run the TK event loop as long as we don't receive input.
    while app.tk.dooneevent(_tkinter.ALL_EVENTS):
        if stop[0]:
            break

    app.deletefilehandler(inputhook_context.fileno())

By the way, inputhook_context there is a parameter coming from prompt-toolkit. I guess it'll make more sense when I have a full patch ready.

That makes sense. I think PyOS_InputHook was probably created for Tkinter, so it's reasonable that we'd have to replace it now that we're moving away from that. And it's really nice that we can do things like this without touching ctypes. Or not directly, at least - I guess you're still dealing with that level within prompt_toolkit.

prompt-toolkit uses only ctypes for Windows. We don't need any 'unsafe' code for posix platforms. It's simply waiting with select, reading from stdin, and writing to stdout. Nothing fancy. For Windows, we need ctypes in order to access the Win32 API, e.g. ctypes.windll.kernel32.ReadConsoleInputW.

I made the pull request above. It's far from finished. But that way you can already have a look. As far as I could test, the integration with all GUI toolkits is fine. Most work to be done is clean up a lot of readline stuff. (which is everywhere...). Especially the history and completer need some attention.

You're a star, thanks! We're in feature freeze for a new release at the moment, but I'll definitely have a look at it.

Hi all,

A short update. Apparently, this summer I had less time than expected, and there was still enough work for prompt-toolkit itself to become even better. It got many improvements, though probably nothing worth mentioning for IPython right now. A few improvements, coming to prompt-toolkit that I'm working on are better navigation for completion menus, scheduling of sleep events in the event loops, and some other small polishing.

So, about IPython. I had a working proof of concept, but there is really a lot to do. Actually everything depending on coloransi.py should be refactored to have a consistent look. But I propose to leave that for later and focus first only on TerminalInteractiveShell and the event loop integration. (Anyway, I guess I won't be able to do everything myself, but I should be able to make it work and do the integration with the event loops and the prompt-toolkit UI.)

As you said in #8597 : In the kernel, we could also call the UI eventloop from there instead of the other way around. I guess it should be equally fast, it doesn't really matter. (Basically with the input hooks we hand over control as well to the GUI loop.) I think I didn't look yet at the code for the kernels, so I could be missing some details. I would do what's the easiest, most flexible and maintainble.

So, from my side, what I'm going to do is create a new pull request, rebased on master with the minimal changes I need for the terminal UI as I have right now, and then we'll see from there.
(Python 3.5 is arriving, so better not to wait too long anymore.)

Jonathan

I might be misunderstanding something, but the kernel eventloops shouldn't need any integration with PT, since they are always in a non-terminal process talking over zmq. The frontend is in a different process, which doesn't run any code in the kernel case.

...and thanks for the update! I know @takluyver is very interested in this, and likely to help out if he has time. He is on holiday until the end of September, though. I don't think the 3.5 pressure is as great as it once was because pyreadline is now able to run on Python 3.5. So we have time to do it carefully.

This is now done ! Yeah one step closer to 5.0 !

ref: #9399

Terminal interface using prompt_toolkit? · Issue #8489 · ipython/ipython · GitHub

Repository navigation

Terminal interface using prompt_toolkit? #8489

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Terminal interface using prompt_toolkit? · Issue #8489 · ipython/ipython · GitHub

Repository navigation

Terminal interface using prompt_toolkit? #8489

Description

Activity

jonathanslenders commented on May 29, 2015

takluyver commented on May 30, 2015

tacaswell commented on May 30, 2015

takluyver commented on May 30, 2015

jonathanslenders commented on Jun 25, 2015

takluyver commented on Jun 25, 2015

jonathanslenders commented on Jun 28, 2015

jonathanslenders commented on Jun 29, 2015

minrk commented on Jun 29, 2015

takluyver commented on Jun 29, 2015

jonathanslenders commented on Jun 29, 2015

takluyver commented on Jun 29, 2015

jonathanslenders commented on Jul 1, 2015

takluyver commented on Jul 1, 2015

jonathanslenders commented on Sep 8, 2015

minrk commented on Sep 9, 2015

minrk commented on Sep 9, 2015

Carreau commented on May 2, 2016

ryneeverett commented on May 2, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions