gh-119118: Fix performance regression in tokenize module by lysnikolaou · Pull Request #119615 · python/cpython · GitHub

Cache line object to avoid creating a Unicode object for all of the tokens in the same line.
Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference.

Issue: tokenize.generate_tokens() performance regression in 3.12 #119118

- Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference.

Hummm, seems also that this solution fails with test_tokenize with -uall:

======================================================================
ERROR: test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) (file='/Users/pgalindo3/github/python/main/Lib/test/test_difflib.py')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pgalindo3/github/python/main/Lib/test/test_tokenize.py", line 1959, in test_random_files
    self.check_roundtrip(f)
    ~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/pgalindo3/github/python/main/Lib/test/test_tokenize.py", line 1827, in check_roundtrip
    tokens2_from5 = [tok[:2] for tok in tokenize.tokenize(readline5)]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 484, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 578, in _generate_tokens_from_c_tokenizer
    raise e from None
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 574, in _generate_tokens_from_c_tokenizer
    for info in it:
        yield TokenInfo._make(info)
  File "<string>", line 467
    dateb = '3 fév'
                   ^
IndentationError: unindent does not match any outer indentation level

======================================================================
ERROR: test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) (file='/Users/pgalindo3/github/python/main/Lib/test/test_html.py')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pgalindo3/github/python/main/Lib/test/test_tokenize.py", line 1959, in test_random_files
    self.check_roundtrip(f)
    ~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/pgalindo3/github/python/main/Lib/test/test_tokenize.py", line 1827, in check_roundtrip
    tokens2_from5 = [tok[:2] for tok in tokenize.tokenize(readline5)]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 484, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 578, in _generate_tokens_from_c_tokenizer
    raise e from None
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 574, in _generate_tokens_from_c_tokenizer
    for info in it:
        yield TokenInfo._make(info)
  File "<string>", line 85
    check('&notin;', '∉')
                         ^
IndentationError: unindent does not match any outer indentation level

======================================================================
ERROR: test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) (file='/Users/pgalindo3/github/python/main/Lib/test/test_str.py')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/pgalindo3/github/python/main/Lib/test/test_tokenize.py", line 1959, in test_random_files
    self.check_roundtrip(f)
    ~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/pgalindo3/github/python/main/Lib/test/test_tokenize.py", line 1827, in check_roundtrip
    tokens2_from5 = [tok[:2] for tok in tokenize.tokenize(readline5)]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 484, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 578, in _generate_tokens_from_c_tokenizer
    raise e from None
  File "/Users/pgalindo3/github/python/main/Lib/tokenize.py", line 574, in _generate_tokens_from_c_tokenizer
    for info in it:
        yield TokenInfo._make(info)
  File "<string>", line 1008
    try:
        ^
IndentationError: unindent does not match any outer indentation level

----------------------------------------------------------------------

Another idea: we do have the token already as unicode (str) so if the line has not changed we can keep adding the size of the token itself to our state and take into account the number of whitespace characters between the tokens (but this is always ASCII so we can basically just calculate as the diff of two pointers).

I'm discussing another fix with @lysnikolaou offline

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

The docs failure seems unrelated. @hugovk do we know what may be happening here?

The latest results are:

cpython on  performance-tokenize [$] via C v15.0.0-clang via 🐍 v3.11.3 
❯ python tmp/t.py
cpython darwin 3.11.3 (main, May  8 2023, 13:16:43) [Clang 14.0.3 (clang-1403.0.22.14.1)]
Time taken: 0.5428769588470459

cpython on  performance-tokenize [$] via C v15.0.0-clang via 🐍 v3.11.3 
❯ ./python.exe tmp/t.py
cpython darwin 3.14.0a0 (heads/performance-tokenize-dirty:ab3437096a7, May 28 2024, 19:26:19) [Clang 15.0.0 (clang-1500.3.9.4)]
Time taken: 0.4570140838623047

The test failures appear to be unrelated. We can probably merge this.

The docs failure seems unrelated. @hugovk do we know what may be happening here?

Yep, need to merge in latest main, the new option has been added there (re: #119221).

Thanks @lysnikolaou for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

…nGH-119615) * pythongh-119118: Fix performance regression in tokenize module - Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference. (cherry picked from commit d87b015) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

GH-119682 is a backport of this pull request to the 3.13 branch.

…nGH-119615) * pythongh-119118: Fix performance regression in tokenize module - Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference. (cherry picked from commit d87b015) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

GH-119683 is a backport of this pull request to the 3.12 branch.

…19615) (#119682) - Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference. (cherry picked from commit d87b015) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

…19615) (#119683) - Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference. (cherry picked from commit d87b015) Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

…n#119615) * pythongh-119118: Fix performance regression in tokenize module - Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference. Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

pythongh-119118 : Fix performance regression in tokenize module

66f9385

- Cache line object to avoid creating a Unicode object for all of the tokens in the same line. - Speed up byte offset to column offset conversion by using the smallest buffer possible to measure the difference.

bedevere-app Bot added the awaiting core review label May 27, 2024

bedevere-app Bot mentioned this pull request May 27, 2024

tokenize.generate_tokens() performance regression in 3.12 #119118

Closed

devdanzin mentioned this pull request May 28, 2024

tokenize in 3.12 makes copies of each line, 3.11 does not #119654

Closed

lysnikolaou requested a review from pablogsal May 28, 2024 12:12

📜🤖 Added by blurb_it.

143a420

pablogsal reviewed May 28, 2024

View reviewed changes

Comment thread Python/Python-tokenize.c Outdated

Update Python/Python-tokenize.c

8318ab8

pablogsal reviewed May 28, 2024

View reviewed changes

Comment thread Python/Python-tokenize.c Outdated

pablogsal reviewed May 28, 2024

View reviewed changes

Comment thread Python/Python-tokenize.c Outdated

WIP: strcmp lines (need to fix) and use line buffer for end col offset

566adc3

lysnikolaou and others added 2 commits May 28, 2024 16:19

Get rid of strcmp

858eff1

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>

Make NEWS linter happy

ab34370

pablogsal approved these changes May 28, 2024

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels May 28, 2024

pablogsal added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels May 28, 2024

pablogsal approved these changes May 28, 2024

View reviewed changes

Merge branch 'main' into performance-tokenize

b97c51e

pablogsal enabled auto-merge (squash) May 28, 2024 18:19

pablogsal merged commit d87b015 into python:main May 28, 2024

bedevere-app Bot removed the awaiting merge label May 28, 2024

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label May 28, 2024

bedevere-app Bot removed the needs backport to 3.12 only security fixes label May 28, 2024

gh-119118: Fix performance regression in tokenize module by lysnikolaou · Pull Request #119615 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-119118: Fix performance regression in tokenize module - #119615

gh-119118: Fix performance regression in tokenize module#119615
pablogsal merged 7 commits into
python:mainfrom
lysnikolaou:performance-tokenize

lysnikolaou commented May 27, 2024 •
edited by bedevere-app Bot
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pablogsal commented May 28, 2024

Uh oh!

pablogsal commented May 28, 2024 •
edited
Loading

Uh oh!

Uh oh!

pablogsal commented May 28, 2024

Uh oh!

pablogsal commented May 28, 2024

Uh oh!

lysnikolaou commented May 28, 2024

Uh oh!

hugovk commented May 28, 2024

Uh oh!

miss-islington-app Bot commented May 28, 2024

Uh oh!

bedevere-app Bot commented May 28, 2024

Uh oh!

bedevere-app Bot commented May 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-119118: Fix performance regression in tokenize module by lysnikolaou · Pull Request #119615 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-119118: Fix performance regression in tokenize module - #119615

gh-119118: Fix performance regression in tokenize module#119615pablogsal merged 7 commits intopython:mainpython/cpython:mainfrom lysnikolaou:performance-tokenizelysnikolaou/cpython:performance-tokenizeCopy head branch name to clipboard

Conversation

lysnikolaou commented May 27, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pablogsal commented May 28, 2024

Uh oh!

pablogsal commented May 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented May 28, 2024

Uh oh!

pablogsal commented May 28, 2024

Uh oh!

lysnikolaou commented May 28, 2024

Uh oh!

hugovk commented May 28, 2024

Uh oh!

miss-islington-app Bot commented May 28, 2024

Uh oh!

bedevere-app Bot commented May 28, 2024

Uh oh!

bedevere-app Bot commented May 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-119118: Fix performance regression in tokenize module#119615
pablogsal merged 7 commits into
python:mainfrom
lysnikolaou:performance-tokenize

lysnikolaou commented May 27, 2024 •
edited by bedevere-app Bot
Loading

pablogsal commented May 28, 2024 •
edited
Loading