Bring back readline capability? · Issue #10364 · ipython/ipython · GitHub

Even if it's not what is used by default, I've experienced and heard about enough other people experiencing pains with the prompt_toolkit-based input breaking people's workflows, that I think we should seriously consider bringing that code back in.

I think it was a mistake to remove it altogether, and change the default at the same time.

recent discussion on the Sage mailing list about this
Add multiline coding in ipython 5 with --simple-prompt #9816 has several users reporting breakage
there are several users brought up prompt toolkit having regressions compared to readline in prompt_toolkit ignores .inputrc #9388
and similarly search backwards by default #10075 contains sentiment from a user saying this: "Multiline editing is a very nice feature. As a longtime vim and bash user, though, the switch to IPython 5.x and prompt_toolkit has been jarring."
Use vi mode for prompt_toolkit layout #10085 lists a limitation and inconsistency of prompt_toolkit which will not be addressed upstream.
Graceful forced exit? #9944 is another issue that came up with the prompt-toolkit transition, which used to work in old readline based code, (though I understand Thomas recently proposed a fix for this in Exit terminal interface gracefully on signals #10185, there is discussion there about the limitations of that, too).
%magic% doesn't work in Emacs 25.1-2 ipython (run-python) buffer on Windows 7. #10211 is an emacs user affected a change that as far as I can tell is also attributable to the transition to prompt toolkit.
ipython 5 tab completion not compatible with jpype #10315 is another unexpected breakage in IPython 5, because prompt_toolkit uses a separate thread for completions (whereas completions had to be synchronous in readline, so jpype completions worked fine in IPython <= 4).
Multi-line input only processes the first line #9948 - user unable to paste multi-line using tmux.
IPython terminal prompt missing after upgrading from 2.3.1 to 5.1.0 #10071 - randomly missing prompt after upgrade to IPython 5 inside docker (because terminal size was reported as 0x0)

I'm willing to do the work to get this back in time for 6.0, so I'll tag #10329 here.

I think this is a particularly important issue because long time users and advocates of IPython are finding themselves frustrated by or even unwilling to use IPython 5 as it is right now.

I'm happy to add that back as an option, though most of the code has been removed and that will likely be a lot of efforts to reintegrate readline.

If we can figure out how to make that optional (so that it can evolve faster than than core to re-fix potential bug that were introduced by the revival, that would be great.

We might want to delay #10356 if we bring RL back.

See #9260 as well.

And #9399 is the bulk of the removal.

I am strongly -1 on this. We were able to drop a lot of code and complexity when we ditched readline, (and we're working on dropping still more), while dramatically improving the user experience for most users. If we bring back a readline option, we have all the complexity of both interfaces to maintain ~forever. We always knew a change this big would break some workflows, but I'm confident that it's a net improvement, and I do not want us to be in the business of maintaining two alternative terminal interfaces.

I would rather we:

Continue to work on anything we can improve both in IPython and in prompt_toolkit to remove obstacles to switching. Jonathan has generally been impressively responsive and helpful whenever we've pinged him about issues relating to PTK, so I think we have a good chance of getting any changes we need accepted. This includes things like better documenting how to set custom shortcuts, and maybe experimenting with reading .inputrc (though I'd rather that was an off-by-default option, for reasons I've explained elsewhere). If we give people an easy option to go back to readline, these improvements are not going to happen.
If there are still people who really insist on using readline, spin out a separate package like rlipython which provides the terminal interface, but make it very clear that it's community maintained, not something we are officially supporting.

Given that there were always a number of problems with pyreadline on Windows (not least of it, the fact that it globally affected the standard Python REPL as well as IPython) I'd prefer that not requiring (py)readline remain the default, at least on Windows. If a user prefers to install pyreadline manually and enable it, that's fine, but IPython should work without pyreadline present, and not include it as a dependency of a normal install.

If there are still people who really insist on using readline, spin out a separate package like rlipython which provides the terminal interface, but make it very clear that it's community maintained, not something we are officially supporting.

I think that is reasonable, though I believe have some integration (like a flag and/or an env variable) with IPython that restore previous behavior is reasonable. The goal would be to have already well spread scripts and recipes that do rely on IPython readline still working with a simple switch.

My guess is that a simple config in IPythonApp, that make the TerminalInteractiveShell class use a traitlet, plus a --readline flag that have a preset value to a known external package would be enough.

The ipython as a separate package will also allow to not block IPython 6, and get a fast iteration cycle if necessary.

If we give people an easy option to go back to readline, these improvements are not going to happen.

I doubt that is completely true. The multiline, highlighting, and now completion are still a large advantage of the PTK interface. And user that are pinning to 4.x because they can't even use bothe RL/PTK in parallel have no incentive send patch to IPython / PTK to smooth the behavior.

I wouldn't integrate it with a switch, because that makes it look like a supported option, and people will file bugs about it here. rlipython is less typing than ipython --readline anyway, and if people are keen to use it as ipython, they can alias it.

Just a note on the current experience of ctrl-r:

In Bash, when you hit ctrl-r then hit escape, it will exit you out.

In this one it does nothing when it should close the search area.

Just a note on the current experience of ctrl-r:
In Bash, when you hit ctrl-r then hit escape, it will exit you out.
In this one it does nothing when it should close the search area.

This one is easy to fix, but not totally:

 ~/dev/ipython master [1]"!!" $ git diff
diff --git a/IPython/terminal/shortcuts.py b/IPython/terminal/shortcuts.py
index 22ad111..89713cd 100644
--- a/IPython/terminal/shortcuts.py
+++ b/IPython/terminal/shortcuts.py
@@ -54,6 +54,10 @@ def register_ipython_shortcuts(registry, shell):
     registry.add_binding(Keys.ControlC, filter=HasFocus(DEFAULT_BUFFER)
                         )(reset_buffer)

+
+    registry.add_binding(Keys.Escape, filter=HasFocus(SEARCH_BUFFER)
+                        )(reset_search_buffer)
+
     registry.add_binding(Keys.ControlC, filter=HasFocus(SEARCH_BUFFER)
                         )(reset_search_buffer)

That will work, but the I-search-backward: will still be visible until you press a new key. This new key will behave has if the backward search was dismissed, but it feels extremely weird.

I actually think we should host this on the main IPython org, and put it on pypi. We can also send a slightly more welcoming message to the community, along the lines of:

this is something that we offer in support of historical compatibility and certain specific use cases where our main interface (prompt-toolkit) isn't optimal. But we do not envision any significant development beyond fixing critical bugs. We only have the resources to offer this as a best-effort solution. If you are interested in doing any significant development on this tool, please let us know via an issue, and we can explore transferring maintainership of the package to you."

If @ivanov has a few cycles to put in this direction that would be great, but if not I'm willing to do it over the next couple of weeks. I think it's an important part of engaging with all corners of our user community.

Thanks @Carreau for pushing the work this far!

Thanks everyone, especially @Carreau for getting rlipython up. I've started working on it and whipping it into shape. In case anyone missed it, it now lives under ipython/rlipython. I'll try to get a release out in the next two weeks as I start working with it, but for now I can say it works! There's one known issue right now - In/Out prompts not being colored, which I am tracking in ipython/rlipython#3, but other than that, it seems great.

@ivanov, ping me on that if you'd like a hand... I'd offered above but I was more than a bit naïve about my time availability these last few weeks. Nonetheless I want to help this reach a solid solution for all cmd line users, so I'm still willing to pitch in if you need a hand (and would enjoy doing so :)

Closed by #10373 and backported as #10463

Hi, with this issue closed, is there any way to disable prompt_toolkit completely?

Hi, with this issue closed, is there any way to disable prompt_toolkit completely?

yes, if you set the TerminalIPythonApp.interactive_shell_class to rlipython in your config file, then prompt_toolkit should not even be imported. The readme of rlipython give more info on how to do this.

For anyone still subscribed on this issue: I just released rlipython version 0.1.0, you can now pip install rlipython and get the old readline working by default in IPython after you import rlipython; rlipython.install().

Late to the party, but let me reiterate that RT at present is a no-no is to projects that have modules not designed for multithreaded initialisation, see Sage's ticket 22766. (And this is the primary reason we need to do something about it in Sage).

As well, a comment on a general problem with the present RT: it triggers multi-threaded importing of Python modules while doing tab-completion. We very much doubt that it is safe.

Last but not the least - given that IPython tab completion causes segfaults, how about a means for non-interactive testing of it?

@jonathanslenders have you run across any issues with completion running in a separate thread? Is there an option to make it run synchronously?

Hi @takluyver,

Excuse me for the late reply. I got married last month, and so I've been offline for a while. ;)

I'm working on prompt_toolkit 2.0 which already supports both synchronous (in the main thread) and asynchronous (in other thread) completion. I've been working on 2.0 for almost a year and hope to release that in a couple of months.

For synchronous completion, you have to keep in mind that if the completer is not super fast, that this will introduce a delay after every key stroke (if you have complete-while-typing on). I believe that Jedi for instance is not super fast.

I know that certain libraries have some issues with asynchronous completion. I think we should encourage the authors of these libraries to make sure that importing them in the first place can be done from any thread. In the future, I'd make it an option to run the completions sync or async, but still prefer to have async completion by default. It makes the user experience so much nicer.

On Sat, Oct 21, 2017 at 2:47 PM, Jonathan Slenders ***@***.*** > wrote: I know that certain libraries have some issues with asynchronous completion. I think we should encourage the authors of these libraries to make sure that importing them in the first place can be done from any thread.

IMHO it's wishful thinking that this is always easy to achieve. There are Python libraries that are basically wrapping very complex 3rd party code, and changes needed for thread-safe initialisation have to be done upstream. A concrete example we are banging our heads is Maxima run in an (embedded into Python) Lisp system ECL. The latter, as all Lisps, needs a garbade collector, and it is using [Boehm GC](https://github.com/ivmai/bdwgc). On some platforms the latter must be initialised in the main thread---or it used to be the case, e.g. for FreeBSD, until I pointed it out, and it was [quickly fixed](bdwgc/bdwgc#180). As you can imagine, in other cases one could be totally out of luck with such a fix, as it needed expert knowledge of a quite complicated system. And there are several issues to be resolved, one of which is that in a single-threaded scenario a library can do its own signal handling, whereas in a multi-threaded scenario the signal handling has to be centralised. The other, even harder to resolve, is that that 3rd party code might not be thread-safe. (going back to the example above, we see all of this...) It appears that the best option would be for PT to do all the imports in the main thread---I have no clue whether it's easy or even feasible to implement, though. Finally, let me point out that in effect PT does a multithreaded importing. As importing is a startup bottleneck for largish systems, it's very tempting to allow multithreaded importing. I would really like to hear what CPython people would say about it; I won't be surprised if the answer were that it must not be done. PS. I don't mean to ruin your honeymoon, congratulations :-)

Congratulations! Of course there's no need to apologise - we get your awesome library for free, so any support has to be as and when it's convenient for you. Best wishes to you and your partner.

I think we'll likely switch to synchronous completions when they're available. We don't use complete-as-you-type, because we enable the history search feature which is mutually exclusive with that.

Bring back readline capability? · Issue #10364 · ipython/ipython · GitHub

Repository navigation

Bring back readline capability? #10364

13 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bring back readline capability? · Issue #10364 · ipython/ipython · GitHub

Repository navigation

Bring back readline capability? #10364

Description

Activity

ivanov commented on Mar 1, 2017

ivanov commented on Mar 1, 2017

Carreau commented on Mar 1, 2017

Carreau commented on Mar 1, 2017

Carreau commented on Mar 1, 2017

Carreau commented on Mar 1, 2017

Carreau commented on Mar 1, 2017

takluyver commented on Mar 2, 2017

pfmoore commented on Mar 2, 2017

Carreau commented on Mar 2, 2017

takluyver commented on Mar 2, 2017

rgbkrk commented on Mar 2, 2017

Carreau commented on Mar 2, 2017

13 remaining items

fperez commented on Mar 22, 2017

ivanov commented on Apr 10, 2017

fperez commented on Apr 21, 2017

Carreau commented on May 10, 2017

heroxbd commented on May 11, 2017

Carreau commented on May 11, 2017

ivanov commented on May 17, 2017

dimpase commented on Oct 8, 2017

takluyver commented on Oct 9, 2017

jonathanslenders commented on Oct 21, 2017

dimpase commented on Oct 21, 2017

takluyver commented on Oct 23, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions