| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| Inlined comprehensions | ||
|
Comment thread
Copy link
Copy Markdown
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low QualityThis is super helpful, thank you
Sorry, something went wrong.
All reactions
|
||
| ====================== | ||
|
|
||
| Since [PEP 709](https://peps.python.org/pep-0709/), list, set, and dict | ||
| comprehensions are compiled into the enclosing compilation unit instead of | ||
| creating a nested function and calling it. Generator expressions are not | ||
| inlined; they still become their own code object. | ||
|
|
||
| The resulting bytecode lives in the enclosing unit, but the comprehension | ||
| still has its own locals: iteration variables must not leak into, or | ||
| overwrite, names in the enclosing scope. The symbol table models that as a | ||
| nested lexical scope; codegen then emits the comprehension inlined into its | ||
| containing compilation unit. | ||
|
|
||
| Which comprehensions are inlined | ||
| -------------------------------- | ||
|
|
||
| `symtable_handle_comprehension()` in | ||
| [`Python/symtable.c`](../Python/symtable.c) inlines a comprehension when it | ||
| is not a generator expression and the current block cannot see class scope | ||
| (`!ste_can_see_class_scope`). Annotation scopes that can see a class keep | ||
| the historical nested-function compilation so class-local names are not | ||
| treated as comprehension locals. | ||
|
|
||
| The outermost iterator expression is always evaluated in the enclosing | ||
| scope. The rest of the comprehension (targets, `if` clauses, inner | ||
| generators, and the element/value expressions) is visited in the | ||
| comprehension's own block. | ||
|
|
||
| Symbol table | ||
| ------------ | ||
|
|
||
| An inlined comprehension gets an `InlinedComprehensionBlock` entry | ||
| ([`pycore_symtable.h`](../Include/internal/pycore_symtable.h)). That entry | ||
| is a child of the enclosing block, with `ste_parent` pointing at the | ||
| enclosing `PySTEntryObject`. It is a lexical subscope, not a compilation | ||
| unit: there is no separate code object, `co_consts` entry, or compiler | ||
| scope for it. | ||
|
|
||
| Uses and bindings inside the comprehension are recorded on that child | ||
| table. Because those loads are in the enclosing compilation unit, a | ||
| `FREE` use is also marked `USE` on the parent table. For | ||
| `def inner(): return [x for y in ()]`, both `inner.lookup("x")` and the | ||
| inlined child's lookup are `FREE|USE`. | ||
|
|
||
| ### Analysis | ||
|
|
||
| `analyze_block()` records this block's declarations, analyzes children, | ||
| then classifies uses. `finalize_inlined_comprehension()` copies | ||
| `USE` from an inlined child onto the parent before that second pass, and | ||
| drops inlined-only frees so `analyze_cells()` does not promote those | ||
| names to cells. | ||
|
|
||
| A name that is `FREE` in the comprehension and bound in the parent is | ||
| dropped from the parent's free set unless: | ||
|
|
||
| * a real nested unit (function, lambda, or genexp) still needs it as a | ||
| cell, or | ||
| * a sibling nested scope already marked it free. | ||
|
|
||
| That keeps iteration variables as fast locals when they are only used by | ||
| nested inlined comprehensions. A nested lambda that captures the name | ||
| still forces a cell on the binding comprehension. | ||
|
|
||
| Class-closure names (`__class__` and friends) loaded from an inlined | ||
| comprehension do not require a class cell unless a nested function, | ||
| lambda, or genexp captures them. Compile treats the inlined loads as | ||
| implicit globals. `is_free_in_any_child()` walks through inlined children | ||
| and only counts `FREE` on non-inlined descendants. | ||
|
|
||
| Compiler | ||
| -------- | ||
|
|
||
| Codegen stays in the enclosing compiler unit. Around the inlined region, | ||
| `_PyCompile_EnterInlinedComprehensionScope()` / | ||
| `_PyCompile_ExitInlinedComprehensionScope()` in | ||
| [`Python/compile.c`](../Python/compile.c) swap `c->u->u_ste` so name | ||
| lookup uses the comprehension's symbol table. The saved `u_ste` is | ||
| restored on both success and error. | ||
|
|
||
| `_PyCompile_ResolveNameop()` calls `compiler_resolve_inlined_free()`, | ||
| which walks `ste_parent` while the current table is inlined and the name | ||
| is `FREE` or missing (scope `0`). Missing names include loads synthesized | ||
| by codegen, such as the implicit receiver for zero-argument `super()`. | ||
| The walk stops at a class: nested scopes do not see class locals. | ||
| Class-closure names that would otherwise be free through a class become | ||
| `GLOBAL_IMPLICIT`. | ||
|
|
||
| ### Isolating iteration variables | ||
|
|
||
| `codegen_push_inlined_comprehension_locals()` in | ||
| [`Python/codegen.c`](../Python/codegen.c) isolates names bound in the | ||
| comprehension: | ||
|
|
||
| * `LOAD_FAST_AND_CLEAR` saves the enclosing value (possibly `NULL`) and | ||
| clears the slot. | ||
| * `MAKE_CELL` runs if the name is a cell for this comprehension. | ||
| * In module and class units the name is added to `u_fasthidden` so | ||
| assemble can set `CO_FAST_HIDDEN`. | ||
|
|
||
| A `SETUP_FINALLY` / `COMPILE_FBLOCK_INLINED_COMPREHENSION` handler | ||
| restores those slots if the comprehension raises, so an enclosing `except` | ||
| or `finally` sees the original values. | ||
|
|
||
| Runtime | ||
| ------- | ||
|
|
||
| An inlined comprehension cell can share a localsplus name with an | ||
| enclosing free variable (for example `[lambda: x for x in x]` inside a | ||
| nested function). `FrameLocalsProxy` keys, values, items, and `len` | ||
| keep the first slot of each name so they agree with `getitem`. | ||
|
|
||
| Source | ||
| ------ | ||
|
|
||
| * [`Python/symtable.c`](../Python/symtable.c): | ||
| `symtable_handle_comprehension()`, `analyze_block()`, | ||
| `finalize_inlined_comprehension()`, `is_free_in_any_child()` | ||
| * [`Python/compile.c`](../Python/compile.c): | ||
| `compiler_resolve_inlined_free()`, | ||
| `_PyCompile_EnterInlinedComprehensionScope()`, | ||
| `compiler_cellvars()` | ||
| * [`Python/codegen.c`](../Python/codegen.c): | ||
| `codegen_comprehension()`, | ||
| `push_inlined_comprehension_state()`, | ||
| `codegen_push_inlined_comprehension_locals()` | ||
| * [`Include/internal/pycore_symtable.h`](../Include/internal/pycore_symtable.h): | ||
| `InlinedComprehensionBlock` | ||
| * [`Include/internal/pycore_compile.h`](../Include/internal/pycore_compile.h): | ||
| `_PyCompile_InlinedComprehensionState` | ||
| * [`Objects/frameobject.c`](../Objects/frameobject.c): | ||
| `FrameLocalsProxy` duplicate-name handling | ||
| Back | FazBrowse Home | New Git URL |
Uh oh!
There was an error while loading. Please reload this page.