gh-124697: Represent inlined comprehensions as subscopes in the symbol table by iritkatriel · Pull Request #156819 · python/cpython · GitHub

Resolves #124697

Inlined comprehensions are now represented in the symbol table as block of a new type InlinedComprehensionBlock, which is a subscope of the enclosing scope (not a separate compilation unit).

This moves the complexity of compiling inlined comprehensions from codegen to the symbol table
construction.

It removes the smell of the compiler modifying the symbol table in codegen.

Issue: refactor the implementation of inlined comprehensions #124697

… symbol table

Co-authored-by: Cursor <cursoragent@cursor.com>

…l tables Co-authored-by: Cursor <cursoragent@cursor.com>

Documentation build overview

📚 cpython-previews | 🛠️ Build #34705179 | 📁 Comparing b61c46f against main (d557d64)

🔍 Preview build

104 files changed · + 8 added · ± 95 modified · - 1 deleted

+ Added

± Modified

- Deleted

library/dialog.html

This looks great! Thank you for working on this ❤️

Very nice! This also fixes #121377 and #156664, but doesn't add tests for their reproducers. Can you add those tests?

I have a PR for the latter almost ready at #156691, but I'm happy to retire it in favor of this PR. @carljm @iritkatriel what do you think about backporting either my change or others? This PR feels a bit much to backport, but we may want to backport fixes for some of the bugs. 156691 is also a somewhat invasive change though.

Codex found that this now fails:

import sys

def outer(x):
    def inner():
        return [(lambda: x, dict(**sys._getframe().f_locals))
                for x in x]
    return inner()

outer([1])

With TypeError: dict() got multiple values for keyword argument 'x'.. #156691 has a larger change to framelocalsproxy to deal with this sort of thing; you may want to incorporate its approach.

Another relevant bug is #156091 but that one still crashes under this PR.

I'd be cautious about backporting an invasive fix for #156664. It's clearly a rarely-encountered edge case, given that nobody other than PyPy's test suite discovered the bug since 3.12, and the risk of introducing new bugs in an invasive fix seems high (particularly without this refactor in place.) I think "better the bugs you know than the ones you don't" applies in this case.

Codex did find a few issues.

Impressive!

@iritkatriel I see all the inline comments are marked resolved, so I take it this is ready for another review?

@iritkatriel I see all the inline comments are marked resolved, so I take it this is ready for another review?

I think so. Thanks.

This is excellent, thank you!!

I wonder if we need to do any compilation performance testing of this PR on release builds? We are now doing this set allocation and per-name deduplication for every frame, including ordinary functions with no comprehensions, cells, or free variables. (Same for values(), items(), and len().)

If this cost is totally invisible in real workloads, maybe this is fine and we just keep the code simpler. But it also seems quite feasible to limit the duplicate-tracking cost to code objects that could actually be affected by it.

Good point, though I think this code impacts introspection tools rather than the compiler. Let me see if it's easy to optimize.

Doable, but not trivial. I'd like to defer the optimisation to a separate PR.

This is super helpful, thank you

🤖 New build scheduled with the buildbot fleet by @iritkatriel for commit be7f556 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F156819%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

🤖 New build scheduled with the buildbot fleet by @iritkatriel for commit b61c46f 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F156819%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

Many thanks to the people and bots. This was fun.

pythongh-124697 : Represent inlined comprehensions as subscopes in the…

3245fd7

… symbol table

iritkatriel requested review from JelleZijlstra, carljm and markshannon as code owners September 2, 2026 11:10

bedevere-app Bot mentioned this pull request Sep 2, 2026

refactor the implementation of inlined comprehensions #124697

Closed

bedevere-app Bot added the awaiting core review label Sep 2, 2026

iritkatriel and others added 2 commits September 2, 2026 12:12

pythongh-124697 : Add NEWS entry for inlined comprehension symbol tables

8dfe966

Co-authored-by: Cursor <cursoragent@cursor.com>

pythongh-124697 : Add What's New entry for inlined comprehension symbo…

97ace7d

…l tables Co-authored-by: Cursor <cursoragent@cursor.com>

iritkatriel requested a review from AA-Turner as a code owner September 2, 2026 11:31

iritkatriel and others added 4 commits September 2, 2026 17:11

simpligy logic and edit docs

21ab5dd

redundant check

045913c

rename helper and use it in more places

0c2ad7e

Merge branch 'main' into subscope

a459371

carljm reviewed Sep 2, 2026

View reviewed changes

Comment thread Python/symtable.c Outdated

Comment thread Python/symtable.c Outdated

Comment thread Python/codegen.c

Comment thread Python/symtable.c Outdated

Comment thread Python/symtable.c Outdated

iritkatriel and others added 11 commits September 3, 2026 13:14

finalize nested against enclosing listcomp

3f7b483

do not splice

9d9ebfc

fix error handling in push/pop_inlined_comprehension_state

bee5d57

do not remove free variables from comprehension scope

0a13f59

Merge branch 'main' into subscope

28c532b

fix test

f4ad426

fix stale comment

ef2a6da

make the symtable honest about __class__ etc

8b37711

no more climbing in _PyST_GetSymbol - symbol table is honest now

ce7433e

Merge branch 'main' into subscope

48f9865

remove DEF_COMP_CELL

3515cc0

JelleZijlstra reviewed Sep 4, 2026

View reviewed changes

deduplicate frame locals

ea23c17

iritkatriel added 3 commits September 18, 2026 00:04

finalize inlined comp before analyze_cells

2c96ad6

add doc

3e173e3

propagate USED flag from inlined comp to enclosing scope

436861a

iritkatriel and others added 2 commits September 18, 2026 18:31

add whatsnew info

f99d475

Merge branch 'main' into subscope

9f2c07d

carljm self-requested a review September 21, 2026 14:50

iritkatriel and others added 2 commits September 21, 2026 16:12

more tests

ed24ee4

Merge branch 'main' into subscope

6bb04f1

carljm approved these changes Sep 21, 2026

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Sep 21, 2026

iritkatriel and others added 5 commits September 21, 2026 18:03

doc tweak

c05d147

avoid use-after-free

39b65fc

Merge branch 'main' into subscope

4ad011f

test needs socket

4c0bfd1

Merge branch 'main' into subscope

be7f556

iritkatriel added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 22, 2026

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 22, 2026

Merge branch 'main' into subscope

b61c46f

iritkatriel added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 22, 2026

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Sep 22, 2026

iritkatriel merged commit a9d42dc into python:main Sep 23, 2026
90 checks passed

bedevere-app Bot removed the awaiting merge label Sep 23, 2026

iritkatriel mentioned this pull request Oct 6, 2026

gh-124697: avoid duplicate names in the same frame #158924

Open

                       return NULL;
                   }
+                  // An inlined comprehension cell can share a name with a free var.
+                  PyObject *seen = PySet_New(NULL);

		@@ -0,0 +1,131 @@
		Inlined comprehensions

gh-124697: Represent inlined comprehensions as subscopes in the symbol table by iritkatriel · Pull Request #156819 · python/cpython · GitHub

Repository navigation

gh-124697: Represent inlined comprehensions as subscopes in the symbol table - #156819

gh-124697: Represent inlined comprehensions as subscopes in the symbol table#156819iritkatriel merged 41 commits intopython:mainpython/cpython:mainfrom iritkatriel:subscopeiritkatriel/cpython:subscopeCopy head branch name to clipboard

Conversation

iritkatriel commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

read-the-docs-community Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Sep 4, 2026

Uh oh!

carljm commented Sep 4, 2026

Uh oh!

iritkatriel commented Sep 18, 2026

Uh oh!

carljm commented Sep 21, 2026

Uh oh!

iritkatriel commented Sep 21, 2026

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carljm Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iritkatriel Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

iritkatriel Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carljm Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Sep 22, 2026

Uh oh!

bedevere-bot commented Sep 22, 2026

Uh oh!

Uh oh!

iritkatriel commented Sep 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gh-124697: Represent inlined comprehensions as subscopes in the symbol table#156819
iritkatriel merged 41 commits into
python:mainfrom
iritkatriel:subscope

iritkatriel commented Sep 2, 2026 •
edited
Loading

read-the-docs-community Bot commented Sep 2, 2026 •
edited
Loading

carljm Sep 21, 2026 •
edited
Loading