gh-151464: exclude '<>' token from tokenize output by skirpichev · Pull Request #154854 · python/cpython · GitHub

Was:

$ echo '1 <> 2' | python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,4:            OP             '<>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''

Now (regardless on __future__.barry_as_FLUFL import):

$ echo '1 <> 2' | ./python -m tokenize
1,0-1,1:            NUMBER         '1'
1,2-1,3:            OP             '<'
1,3-1,4:            OP             '>'
1,5-1,6:            NUMBER         '2'
1,6-1,7:            NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''

in accordance with the Grammar:
https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters

Also adds a custom error message for <> ("not equal" in Pascal and Python 2).

Issue: Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464

Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2).

Thanks for this work :)

I've just gone through these changes and changes looks good as per my understanding ofcourse i'm not expert 😀 , and it was really interesting to going through these changes, at first I found its bit tricky to understand some portion.

I'm excited to see the review process in this from experts and try to understand how it goes, also if you feel this comment as noise please feel free to mark as off-topic.

Left some comments

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Left some comments

I have made the requested changes; please review again

Thanks for making the requested changes!

@pablogsal: please review the changes made to this pull request.

Documentation build overview

📚 cpython-previews | 🛠️ Build #34646001 | 📁 Comparing e1fc4cc against main (219768f)

🔍 Preview build

398 files changed · + 9 added · ± 388 modified · - 1 deleted

+ Added

± Modified

- Deleted

library/dialog.html

@pablogsal, does it make sense for you at all? I doubt I can reduce patch further.

# Conflicts: # Grammar/python.gram # Parser/parser.c # Parser/pegen.c

Thanks @skirpichev for the PR, and @pablogsal for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

Sorry, @skirpichev and @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 198bc7683641f6b44d23fe5dc228f732a7cf3df8 3.15

Sorry, @skirpichev and @pablogsal, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 198bc7683641f6b44d23fe5dc228f732a7cf3df8 3.14

Sorry, @skirpichev and @pablogsal, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 198bc7683641f6b44d23fe5dc228f732a7cf3df8 3.13

GH-158189 is a backport of this pull request to the 3.15 branch.

GH-158190 is a backport of this pull request to the 3.14 branch.

GH-158191 is a backport of this pull request to the 3.13 branch.

…onGH-154854) * pythongh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76)

…#158191) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

…#158189) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

…onGH-154854) * pythongh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76)

…#158190) * gh-151464: exclude '<>' token from tokenize output Was: ``` $ echo '1 <> 2' | python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,4: OP '<>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` Now (regardless on ``__future__.barry_as_FLUFL`` import): ``` $ echo '1 <> 2' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,2-1,3: OP '<' 1,3-1,4: OP '>' 1,5-1,6: NUMBER '2' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ``` in accordance with the Grammar: https://docs.python.org/3.14/reference/lexical_analysis.html#operators-and-delimiters Also adds a custom error message for ``<>`` ("not equal" in Pascal and Python 2). * +1 * address review: lowercase and move invalid rule * address review: news * address review: move test_guido_as_bdfl_ineq_tokens() * address review: revert _PyTokenizer_From* changes * + revert unrelated change * address review: remove whatsnew entry (cherry picked from commit 198bc76) Co-authored-by: Sergey B Kirpichev <skirpichev@gmail.com>

Great work on the tokenizer fix, @skirpichev! Thanks a lot! ❤️

bedevere-app Bot mentioned this pull request Jul 29, 2026

Tokenizer produces token '<>', missing in the language grammar (regardless on __future__.barry_as_FLUFL settings) #151464

Closed

+1

c53a17f

skirpichev marked this pull request as ready for review July 29, 2026 08:49

skirpichev requested review from lysnikolaou and pablogsal as code owners July 29, 2026 08:49

bedevere-app Bot added the awaiting review label Jul 29, 2026

Aniketsy reviewed Jul 31, 2026

View reviewed changes

pablogsal requested changes Aug 10, 2026

View reviewed changes

Comment thread Misc/NEWS.d/next/Core_and_Builtins/2026-07-29-08-43-14.gh-issue-151464.o_mtmz.rst Outdated

Comment thread Lib/test/test_flufl.py Outdated

Comment thread Parser/lexer/state.h Outdated

Comment thread Parser/tokenizer/tokenizer.h Outdated

Comment thread Grammar/python.gram Outdated

bedevere-app Bot added awaiting changes and removed awaiting review labels Aug 10, 2026

skirpichev added 5 commits August 11, 2026 02:57

Merge branch 'master' into save-python/151464

e525b8f

address review: lowercase and move invalid rule

f4dd644

address review: news

ba5843d

address review: move test_guido_as_bdfl_ineq_tokens()

9ea138f

address review: revert _PyTokenizer_From* changes

3f9c57e

skirpichev requested a review from AA-Turner as a code owner August 11, 2026 01:20

bedevere-app Bot added awaiting change review and removed awaiting changes labels Aug 11, 2026

bedevere-app Bot requested a review from pablogsal August 11, 2026 01:22

skirpichev added 3 commits August 21, 2026 06:26

Merge branch 'main' into save-python/151464

a966092

Merge branch 'master' into save-python/151464

d81b19c

+ revert unrelated change

3ff8480

Merge branch 'master' into save-python/151464

d134772

# Conflicts: # Grammar/python.gram # Parser/parser.c # Parser/pegen.c

skirpichev closed this Sep 16, 2026

bedevere-app Bot added awaiting merge and removed awaiting change review labels Sep 25, 2026

pablogsal added needs backport to 3.13 only security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 bugs and security fixes labels Sep 25, 2026

pablogsal merged commit 198bc76 into python:main Sep 25, 2026
64 of 65 checks passed

bedevere-app Bot removed the awaiting merge label Sep 25, 2026

miss-islington-app Bot assigned pablogsal Sep 25, 2026

pablogsal mentioned this pull request Sep 25, 2026

[3.15] gh-151464: exclude '<>' token from tokenize output (GH-154854) #158189

Merged

bedevere-app Bot removed the needs backport to 3.15 bugs and security fixes label Sep 25, 2026

pablogsal mentioned this pull request Sep 25, 2026

[3.14] gh-151464: exclude '<>' token from tokenize output (GH-154854) #158190

Merged

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Sep 25, 2026

pablogsal mentioned this pull request Sep 25, 2026

[3.13] gh-151464: exclude '<>' token from tokenize output (GH-154854) #158191

Merged

bedevere-app Bot removed the needs backport to 3.13 only security fixes label Sep 25, 2026

skirpichev deleted the save-python/151464 branch September 26, 2026 00:14

gh-151464: exclude '<>' token from tokenize output by skirpichev · Pull Request #154854 · python/cpython · GitHub

Repository navigation

gh-151464: exclude '<>' token from tokenize output - #154854

gh-151464: exclude '<>' token from tokenize output#154854pablogsal merged 14 commits intopython:mainpython/cpython:mainfrom skirpichev:save-python/151464skirpichev/cpython:save-python/151464Copy head branch name to clipboard

Conversation

skirpichev commented Jul 29, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aniketsy left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app Bot commented Aug 10, 2026

Uh oh!

skirpichev commented Aug 11, 2026

Uh oh!

bedevere-app Bot commented Aug 11, 2026

Uh oh!

read-the-docs-community Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

skirpichev commented Sep 8, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

miss-islington-app Bot commented Sep 25, 2026

Uh oh!

bedevere-app Bot commented Sep 25, 2026

Uh oh!

bedevere-app Bot commented Sep 25, 2026

Uh oh!

bedevere-app Bot commented Sep 25, 2026

Uh oh!

pablogsal commented Sep 25, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-151464: exclude '<>' token from tokenize output#154854
pablogsal merged 14 commits into
python:mainfrom
skirpichev:save-python/151464

skirpichev commented Jul 29, 2026 •
edited by bedevere-app Bot
Loading

pablogsal left a comment •
edited
Loading

read-the-docs-community Bot commented Aug 11, 2026 •
edited
Loading