Repository navigation
IPython (terminal) should be smarter about how it computes invisible characters #8724
Description
The IPython terminal prompt calculates invisible characters by computing it with and without colors and taking the difference (see
ipython/IPython/core/prompts.py
Line 353 in f2d072e
But there are issues with this. For one thing, custom prompts could have side effects, and this causes it to be rendered multiple times (although I doubt this is a real issue in the wild). More importantly, there can be other invisible characters not accounted for by the built-in color codes, like custom ANSI escape sequences. If you do this, the out and in2 prompts are indented too far, because it counts characters that should be invisible.
The correct way to test this is to look for \[ and \] (i.e., \001 and \002) blocks, which is what readline does. readline may have a function to check the visible length (I haven't checked), although it shouldn't be hard to implement manually.
The code also implicitly assumes that the number of invisible characters is static, which also might not be true for a custom prompt with LazyEvaluate formatting.
I'll likely be sending a PR to fix this.
Activity
minrk commented on Aug 13, 2015
Great! Sounds like you've thought it through. Looking forward to it.
asmeurer commented on Aug 13, 2015
Reading the readline C code, I believe the readline function would be expand_prompt, which isn't exposed in the Python API (even in the rl library).
asmeurer commented on Aug 13, 2015
The basic fix (counting all invisible characters, not just known color codes) is easy.
The more advanced fix (taking into consideration that a dynamic prompt might have a different number invisible characters each time it is displayed) is awkward with the existing API, which implicitly assumes that the number invisible characters is the same for any given template. PromptManager.invisible_characters is a dictionary mapping prompt type (in, out, etc.) to number of invisible characters. Really, it should be a function which is called on a prompt string. You can only count the number of invisible characters on a prompt string once it is rendered, and you should only render it once.
My personal use-cases don't have this situation, but it's not hard to imagine some (e.g., say you put a colored git branch in your prompt, but the branch is not there if your current directory is not in a git repository). The current code would do the wrong thing if you don't use the known IPython color codes, and even after the basic fix checking for \001, and \002 it would do the wrong thing if you change directories inside the session, or some other change where the number of color codes changes.
takluyver commented on Aug 13, 2015
We're planning to move away from readline in favour of prompt_toolkit, in any case, so this may become moot.
asmeurer commented on Aug 13, 2015
OK. I'll just deal with the very simple fix that fixes my problem then.
Is there code for prompt_toolkit that I can look at? Will any of the various prompt customization mechanisms be changing?
takluyver commented on Aug 13, 2015
https://github.com/jonathanslenders/python-prompt-toolkit/
The author has also started some integration work with us in #8597, but I think we might need to bring it in via a different route. Too early to say how it will affect prompt customisation.
jonathanslenders commented on Aug 16, 2015
Thanks for linking the issue.
In the end, we should not allow ANSI escape sequences anymore at this point, that is a wrong abstraction, dependent on the underlying terminal. We want to be platform independent. But for now, you can of course just try to fix it.
Further, using len to calculate the width of a string on the terminal is most of the time wrong. Normally, we should use wcwidth instead, but in this particular case, I think it's still fine, because we are talking about a width difference.
About replacing readline with prompt-toolkit, I hope to be able to resume work soon. I'm back from vacation, but in a week I'll leave again, and there are a few other priorities.
Jonathan
asmeurer commented on Aug 16, 2015
For my use-case, I have custom escape sequences that I am trying to integrate with IPython. However, with prompt-toolkit, the correct way may be to integrate that directly with prompt-toolkit rather than with IPython. Would love to hear your thoughts on that.
jonathanslenders commented on Aug 16, 2015
That's interesting. I guess it's early for me to say anything about it. But for some of those sequences it makes sense to build them into prompt-toolkit. E.g. before_prompt and after_prompt. Maybe before_output as well, but after_output it out of our control. (I have no idea whether it's an all or nothing thing.) Further, I guess there is no way they will be able to understand multiline commands.
Question: Iterm2 is really the only terminal supporting this?
asmeurer commented on Aug 16, 2015
No, it doesn't support multiline commands. after_output generally comes right before before_prompt.
The codes are originally from Final Term (which is now defunct), so I suppose it's possible that other terminals use it. iTerm2 is the only terminal I know of, but I'm not a terminal expert, and I really only care about iTerm2.
Be a little smarter about invisible characters in terminal prompts
asmeurer commented on Aug 16, 2015
PR: #8738
Carreau commented on Jan 19, 2016
Can this be closed since #8738 was merged ?
asmeurer commented on Jan 19, 2016
My fix was relatively basic. There are still some cases where this could break for very dynamic prompts (see the commit message of asmeurer@8e996f8).
Carreau commented on Jan 19, 2016
Ok, i'll bump to 5.0 then. Thanks !
Carreau commented on Jun 2, 2016
Now you should be able to emit ZeroWidthEscape Tokens in the prompts functions.
Marking that as documentation.
Document colours and escapes in prompts