bpo-27494: Fix a 2to3 parser bug (trailing comma) by jstasiak · Pull Request #60 · python/cpython · GitHub

While this is a valid Python 2 and Python 3 syntax lib2to3 would choke
on it:

    set(x for x in [],)

This patch changes the grammar definition used by lib2to3 so that the
actual Python syntax is supported now and backwards compatibility is
preserved.

https://bugs.python.org/issue27494

Codecov Report

Merging #60 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
+ Coverage   82.37%   82.37%   +<.01%     
==========================================
  Files        1427     1427              
  Lines      350948   350951       +3     
==========================================
+ Hits       289091   289099       +8     
+ Misses      61857    61852       -5

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 649a7ca...d7f230d. Read the comment docs.

I'm not a core dev, but I took some time to review this anyway. (I recently familiarized myself with lib2to3 in order to make a different PR -- reviews welcome.)

Background:

This PR changes the lib2to3 grammar's handling of for syntax, to distinguish between the following two cases:

list comprehensions and generator expressions, e.g.
- [x for x in foo]
- (x for x in foo)
set and dictionary comprehensions, e.g.
- {x for x in foo}
- {x : y for x,y in zip(foo,bar)}

Previously, the grammar used the same node type for those two cases. Unfortunately, that node specification is overly complicated due to backwards compatibility considerations. As a consequence, the grammar failed to handle some valid programs.

Review:

LGTM. The problematic case is indeed fixed, and now covered by the test suite. The three "fixers" that are affected by this Grammar change have been updated. (I see no others that need editing.)

I agree with the need for a verbose comment here, since it's not immediately obvious why old_comp_iter, etc. are necessary. IMHO, you should remove overly cautious words like "seems" and "I believe". Also, I think it might be worth adding a reference to this PR and/or bpo-27494 in the comment itself.

Thanks for the feedback, it makes sense; let me improve this a little.

Yes, just to be precise:

This PR changes the lib2to3 grammar's handling of for syntax, to distinguish between the following two cases:

list comprehensions and generator expressions, e.g. (...)

set and dictionary comprehensions, e.g (...)

The second group would be set/dictionary comprehensions and call arguments (like print(x for x in [],))

cc @benjaminp @serhiy-storchaka @vadmium @gpshead @1st1 (I looked up who changed lib/lib2to3 recently, apologies for nagging)

I'm happy to make any changes necessary to get this merged, I just need to know what's required.

@ambv is this something you could review? I'm asking you because I see you're an author of another 2to3 patch (#1724)

This commit adds a test case to trigger an assertion failure during shutdown and fixes the bug. The assertion was added by my first attempt to fix issue python#60. Thanks to Masamitsu Murase for reporting the bug and for providing the fix. (grafted from 655f2a2292a237636849b4a82fb0a2dde1ee9847 and 109d5d067b14)

Closed by accident, apologies.

…ession While this is a valid Python 2 and Python 3 syntax lib2to3 would choke on it: set(x for x in [],) This patch changes the grammar definition used by lib2to3 so that the actual Python syntax is supported now and backwards compatibility is preserved.

OK, I'm lost regarding Travis build failures. It seems that the configuration that Travis uses to build the docs (https://travis-ci.org/python/cpython/jobs/279869243/config) is different than in builds of some more recent PR-s (https://travis-ci.org/python/cpython/jobs/279831444/config for example).

I suspect for whatever reason it's not picking up updates to .travis.yml (made since I created my branch even though I pushed them to the branch now) and that makes it fail.

One thing I suspect may work around this is closing this PR and opening a new one...

I gave up on trying to make this pull request work with Travis and created another one: #3771. The new one has been actually built successfully now!

&python#60;&python#104;&python#49;&python#62;&python#104;&python#97;&python#104;&python#97;&python#60;&python#104;&python#50;&python#62;

Fixes #60

* Update .coveragerc * Keep whitespace consistent. Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>

* Fix: Correct target_bb_id in jump array * Doc: Updated doc for add_metadata_to_jump_2d_array

the-knights-who-say-ni added the CLA signed label Feb 13, 2017

jstasiak force-pushed the fix-lib2to3 branch from 94c12e3 to d7f230d Compare February 13, 2017 13:28

stuarteberg approved these changes Feb 14, 2017

View reviewed changes

jstasiak force-pushed the fix-lib2to3 branch from d7f230d to e302044 Compare February 15, 2017 21:54

ambv self-requested a review July 17, 2017 22:36

jstasiak force-pushed the fix-lib2to3 branch from e302044 to e5a673d Compare September 25, 2017 23:45

jstasiak closed this Sep 26, 2017

jstasiak force-pushed the fix-lib2to3 branch from e5a673d to 4a2d00c Compare September 26, 2017 00:05

jstasiak reopened this Sep 26, 2017

jstasiak force-pushed the fix-lib2to3 branch 3 times, most recently from 1818dc6 to 4ae43d5 Compare September 26, 2017 09:00

jstasiak force-pushed the fix-lib2to3 branch from 4ae43d5 to 71b60a5 Compare September 26, 2017 19:20

jstasiak closed this Sep 26, 2017

jstasiak mentioned this pull request Sep 26, 2017

bpo-27494: Fix 2to3 handling of trailing comma after a generator expression #3771

Merged

Mr3x33 added a commit to Mr3x33/cpython that referenced this pull request Sep 17, 2021

aaaa

c755279

&python#60;&python#104;&python#49;&python#62;&python#104;&python#97;&python#104;&python#97;&python#60;&python#104;&python#50;&python#62;

colesbury referenced this pull request in colesbury/nogil Oct 6, 2021

test_code.py: fix flaky test due to data race

50ce32e

Fixes #60

gvanrossum mentioned this pull request Apr 10, 2022

Test issue #47020

Closed

jaraco pushed a commit that referenced this pull request Dec 2, 2022

Update the Procfile (GH-60 )

e766eaf

jaraco added a commit to jaraco/cpython that referenced this pull request Feb 17, 2023

exclude build env from cov reporting (python#60 )

e719f86

* Update .coveragerc * Keep whitespace consistent. Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>

isidentical pushed a commit to isidentical/cpython that referenced this pull request Mar 28, 2023

Merge pull request python#60 from mgmacias95/fix_nested_bug

87f9bfd

oraluben pushed a commit to oraluben/cpython that referenced this pull request Jul 12, 2023

Fix: Correct target_bb_id in jump array (python#60 )

22580dc

* Fix: Correct target_bb_id in jump array * Doc: Updated doc for add_metadata_to_jump_2d_array

gvanrossum mentioned this pull request Aug 22, 2023

heap-use-after-free in _PyFunction_LookupByVersion #108253

Closed

LinanV mentioned this pull request Oct 23, 2023

Objects/typeobject.c: No such file or directory. #111203

Closed

ngoldbaum mentioned this pull request Sep 23, 2024

Crash running PyO3 tests with --test-threads=1000 #124375

Closed

This was referenced Feb 11, 2025

segmentfault when pip installing setuptools #129992

Closed

segmentfault when pip installing setuptools #129993

Closed

AA-Turner pushed a commit to AA-Turner/cpython that referenced this pull request Apr 10, 2025

Remove unnecessary _Py_TPFLAGS_MATCH_SELF from Template type (python#60 )

422a515

DominiquePACCO mentioned this pull request Jun 3, 2025

PdfWriter add_page adds (re)writes old page instead of new #135062

Closed

sergey-miryanov mentioned this pull request Dec 18, 2025

Observed memory leak in ssl library: Python 3.14 GC issue #142516

Closed

ngoldbaum mentioned this pull request Mar 6, 2026

Remove deprecated and undocumented function ctypes.SetPointerType #133866

Closed

bpo-27494: Fix a 2to3 parser bug (trailing comma) by jstasiak · Pull Request #60 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-27494: Fix a 2to3 parser bug (trailing comma) - #60

bpo-27494: Fix a 2to3 parser bug (trailing comma)#60
jstasiak wants to merge 1 commit into
python:masterfrom
jstasiak:fix-lib2to3

jstasiak commented Feb 13, 2017 •
edited by bedevere-bot
Loading

Uh oh!

Uh oh!

codecov Bot commented Feb 13, 2017

Uh oh!

stuarteberg left a comment •
edited
Loading

Uh oh!

Uh oh!

stuarteberg Feb 14, 2017

Uh oh!

jstasiak Feb 15, 2017

Uh oh!

jstasiak commented Feb 15, 2017

Uh oh!

jstasiak commented Mar 23, 2017

Uh oh!

jstasiak commented Jul 17, 2017 •
edited
Loading

Uh oh!

Uh oh!

jstasiak commented Sep 26, 2017

Uh oh!

jstasiak commented Sep 26, 2017

Uh oh!

jstasiak commented Sep 26, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bpo-27494: Fix a 2to3 parser bug (trailing comma) by jstasiak · Pull Request #60 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-27494: Fix a 2to3 parser bug (trailing comma) - #60

bpo-27494: Fix a 2to3 parser bug (trailing comma)#60jstasiak wants to merge 1 commit intopython:masterpython/cpython:masterfrom jstasiak:fix-lib2to3jstasiak/cpython:fix-lib2to3Copy head branch name to clipboard

Conversation

jstasiak commented Feb 13, 2017 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Feb 13, 2017

Codecov Report

Uh oh!

stuarteberg left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuarteberg Feb 14, 2017

Choose a reason for hiding this comment

Uh oh!

jstasiak Feb 15, 2017

Choose a reason for hiding this comment

Uh oh!

jstasiak commented Feb 15, 2017

Uh oh!

jstasiak commented Mar 23, 2017

Uh oh!

jstasiak commented Jul 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jstasiak commented Sep 26, 2017

Uh oh!

jstasiak commented Sep 26, 2017

Uh oh!

jstasiak commented Sep 26, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bpo-27494: Fix a 2to3 parser bug (trailing comma)#60
jstasiak wants to merge 1 commit into
python:masterfrom
jstasiak:fix-lib2to3

jstasiak commented Feb 13, 2017 •
edited by bedevere-bot
Loading

stuarteberg left a comment •
edited
Loading

jstasiak commented Jul 17, 2017 •
edited
Loading