Improve PyREPL auto-indentation · Issue #133710 · python/cpython · GitHub

Bug report

On second thought this issue should be an enhancement instead of a bug report. Sorry for the wrong template.

Bug description:

There are some cases where _pyrepl auto-indentation works not well.

Cases

A line ending with : in a multi-line string is wrongly indented.

Observed

>>> s = '''
... Note:
... ␣␣␣␣|

Expected

>>> s = '''
... Note:
... |

# inside strings is seen as a comment, the following : is ignored.

Observed

>>> if ' ' == '#':
... |

Expected

>>> if ' ' == '#':
... ␣␣␣␣|

When the entire cursor line is a comment and is already indented, pressing Enter gives a further indent.

Observed

>>> def f():
...     # foo⤶
... ␣␣␣␣␣␣␣␣|

Expected

>>> def f():
...     # foo⤶
... ␣␣␣␣|

Possible solution

Currently _should_auto_indent() parses the buffer from right to left and stops at the first newline it encounters. Only the last line that is not a comment line of the buffer is parsed.

But by parsing from right to left we can't tell if a # starts a comment or is part of a string. For example if we, from right to left, encounter a " first and then a #, we don't know if the # is a comment. To know that, we need to know if the " is a string boundary, but the # might comment out the ", so we can't be sure. There is a information dependency cycle.

To fix this I made a change to parse the buffer from left to right, keeping track of whether current char is inside a string or a comment. This approach solves the above three cases. However the whole buffer is parsed on every call of _should_auto_indent(), with very long buffer, there might be noticeable delay when pressing Enter.

I think this is a big change. It affects how _should_auto_indent() works as a whole. I am hesitated to create a PR and just put it here first to hopefully get feedback.

CPython versions tested on:

3.15

Operating systems tested on:

Linux

Linked PRs

gh-133710: Enhance pyrepl auto-indent #140710

However the whole buffer is parsed on every call of _should_auto_indent(), with very long buffer, there might be noticeable delay when pressing Enter.

I think that this may unfortunately be a deal breaker. Thoughts @ambv ?

Thanks for the response! I will see if I can profile it over the weekend. If it ends up causing significant lag, what do you think about left-to-right full parsing only when the buffer is under certain threshold length, and falling back to the original right-to-left partial parsing for longer buffer?

I would prefer to keep the implementation simple and not split behavior based on input size. What problem are we solving here exactly? Just that auto-indent works inside multiline strings?

@ambv Apologies for the split behavior suggestion, I agree it would complicate the implementation unnecessarily.

@pablogsal I've profiled the performance impact of the change. Here's a comparison between the original and enhanced versions when processing a multiline string with up to 10k characters. While the relative slowdown is significant, the absolute delay remains low (~0.02s for 10k characters on an i5-7200U 2.5GHz CPU with an HDD). The delay is not noticeable, I think, and in interactive REPL scenarios the inputs would be shorter.

The code:

import timeit
import random
import string

def _should_auto_indent__main_branch(buffer: list[str], pos: int) -> bool:
    # check if last character before "pos" is a colon, ignoring
    # whitespaces and comments.
    last_char = None
    while pos > 0:
        pos -= 1
        if last_char is None:
            if buffer[pos] not in " \t\n#":  # ignore whitespaces and comments
                last_char = buffer[pos]
        else:
            # even if we found a non-whitespace character before
            # original pos, we keep going back until newline is reached
            # to make sure we ignore comments
            if buffer[pos] == "\n":
                break
            if buffer[pos] == "#":
                last_char = None
    return last_char == ":"

def _should_auto_indent__enhance_branch(buffer: list[str], pos: int) -> bool:
    # check if last character before "pos" is a colon, ignoring
    # whitespaces, comments, and contents within strings.
    last_char = None
    # A stack to keep track of string delimiters. Push a quote when entering a
    # string, pop it when the string ends. If the stack is empty, we're not
    # inside a string. When see a '#', it's a comment start if we're not inside
    # a string; otherwise, it's just a '#' character within a string.
    str_delims: list[str] = []
    in_comment = False
    char_line_indent_start: int | None = None
    char_line_indent = 0
    lastchar_line_indent = 0
    cursor_line_indent = 0

    i = -1
    while i < pos - 1:
        i += 1
        char = buffer[i]

        # update last_char
        if char == "#":
            if str_delims:
                last_char = char # '#' inside a string is just a character
            else:
                in_comment = True
        elif char == "\n":
            # newline ends a comment
            in_comment = False
            if i < pos - 1 and buffer[i + 1] in " \t":
                char_line_indent_start = i + 1
            else:
                char_line_indent_start = None # clear last line's line_indent_start
                char_line_indent = 0
        elif char not in " \t":
            if char_line_indent_start is not None:
                char_line_indent = i - char_line_indent_start
            if not in_comment and not str_delims:
                # update last_char with non-whitespace chars outside comments and strings
                last_char = char
                lastchar_line_indent = char_line_indent

        # update stack
        if char in "\"'" and (i == 0 or buffer[i - 1] != "\\"):
            if str_delims and str_delims[-1] == char:
                str_delims.pop()
            else:
                str_delims.append(char)
    cursor_line_indent = char_line_indent
    return last_char == ":" and cursor_line_indent <= lastchar_line_indent

# 1000x100 multilines string
multiline_str = "\n".join("".join(random.choices(string.ascii_letters, k=100)) for _ in range(1000))

if __name__ == "__main__":
    for i in range(2, 12, 2):
        buffer = list(f'"""\n{multiline_str[:i*1000]}\n"""')
        pos = len(buffer)

        t1 = timeit.timeit("_should_auto_indent__main_branch(buffer, pos)", number=100, globals=globals()) / 100
        t2 = timeit.timeit("_should_auto_indent__enhance_branch(buffer, pos)", number=100, globals=globals()) / 100

        print("Multiline string length:", i*1000)
        print("before:", t1)
        print("after:", t2)
        print("diff:", f"{t2-t1=}, {t2/t1=}", "\n")

The result:

Multiline string length: 2000
before: 7.77800043579191e-06
after: 0.0038612419995479284
diff: t2-t1=0.0038534639991121365, t2/t1=496.43118837840484

Multiline string length: 4000
before: 7.662000134587288e-06
after: 0.007558695999905467
diff: t2-t1=0.00755103399977088, t2/t1=986.5173410510015

Multiline string length: 6000
before: 7.987999706529081e-06
after: 0.011519652000279166
diff: t2-t1=0.011511664000572637, t2/t1=1442.1197325362255

Multiline string length: 8000
before: 9.829999762587248e-06
after: 0.01693329099973198
diff: t2-t1=0.016923460999969393, t2/t1=1722.6135715872238

Multiline string length: 10000
before: 7.742000161670148e-06
after: 0.019101623999886214
diff: t2-t1=0.019093881999724543, t2/t1=2467.2724878586805

Improve PyREPL auto-indentation · Issue #133710 · python/cpython · GitHub

Repository navigation

Improve PyREPL auto-indentation #133710

Bug report

Bug description:

Cases

Possible solution

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Improve PyREPL auto-indentation · Issue #133710 · python/cpython · GitHub

Repository navigation

Improve PyREPL auto-indentation #133710

Description

Bug report

Bug description:

Cases

Possible solution

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

pablogsal commented on Oct 30, 2025

tanloong commented on Oct 30, 2025

ambv commented on Oct 31, 2025

tanloong commented on Nov 9, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions