PEP 695: Lazy evaluation, concrete scoping semantics, other changes by JelleZijlstra · Pull Request #3122 · python/peps · GitHub

This adds some clarifications and changes that I ran into while implementing the PEP.

The most important change is that TypeVar bounds and type alias values are lazily evaluated using PEP 649-like semantics. This will make the language more consistent once type annotations are evaluated using PEP 649.

Prior discussion:

PEP 649-like semantics: Lazy evaluation and PEP 649 erictraut/cpython#11
__type_params__ attribute replacing __type_variables__: Semantics of __type_variables__ erictraut/cpython#10, Clarification in the choice of __type_variables__ vs __type_param(eter)?s__ erictraut/cpython#8
Disallowing yield etc.: Prohibit yield, yield from, await, walrus in aliases and bounds? erictraut/cpython#13

I am marking this as draft for now in case I run into more things that need to be changed. We'll have to ask the SC for approval, but hopefully these changes are not controversial.

cc @erictraut

📚 Documentation preview 📚: https://pep-previews--3122.org.readthedocs.build/

LGTM

Some preliminary comments while I have this freshly in my head.

- Lazy evaluation means that referencing a later type variable works at runtime - Disallow walrus in TypeVar bounds, and also disallow yield/yield from/await in the same contexts

… using now

@erictraut @gvanrossum I pushed a few more changes; please take a look.

I am also considering adding more information about the implementation technique I'm currently using (a dummy immediately evaluated function, which I just learned is "lambda lifting"), but I'm not sure we need to specify that in the PEP.

Submitted python/steering-council#186 to the SC to approve these changes. Marking as "DO-NOT-MERGE" until the SC approves.

Thanks for taking care to explain all these details clearly!

Could you include the small AST change for TypeAlias.name here as well?

Couldn't comment inline unfortunately. It should be here:

peps/pep-0695.rst

Line 802 in 63a594f

TypeAlias(identifier name, typeparam* typeparams, expr value)

- TypeAlias(identifier name, typeparam* typeparams, expr value)
+ TypeAlias(expr name, typeparam* typeparams, expr value)

Pushed another change related to name mangling (I realized that the wording in the PEP would require severe complications in the symtable).

+1 on the mangling change.

Does the mangling change mean that inner scopes cannot refer to that type parameter without generating a runtime exception?

class Foo[__T]:
    def method(self, val: __T): # Does this work?
        x: __T = val # Does this work?

@erictraut yes, that will work. (Though note that the annotation within the function will always work without a runtime exception, because annotations on locals are never evaluated.) The name is mangled at both the definition site and all usage sites within the class. This was quite easy to implement: python/cpython@bd49622.

The PEP previously proposed not mangling the TypeVar names. That would require deep changes to the symtable to track whether a name refers to a TypeVar. The new proposed approach is more consistent with the rest of the language: names that are syntactically within a class are always mangled.

I think there are very few ways the mangling is even visible to user code. The only way I can think of is this:

>>> class A: pass
... 
>>> class Foo[__T](print(locals()) or A): pass
... 
{'_Foo__T': __T, '.generic_base': typing.Generic[__T], '.type_params': (__T,)}

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Definitely an improvement.

It still isn't clear to me which of the def695 "functions" exist to provide scopes for type variables, and which are to support lazy evaluation. Are the semantics of both identical?

It still isn't clear to me which of the def695 "functions" exist to provide scopes for type variables, and which are to support lazy evaluation. Are the semantics of both identical?

Yes, they have identical semantics. Let me push something to make that explicit.

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

The SC approved the proposed changes.

For reference: python/steering-council#186 (comment)

…ython#3122) - Lazy evaluation means that referencing a later type variable works at runtime - Disallow walrus in TypeVar bounds, and also disallow yield/yield from/await in the same contexts - Remove rejection of lambda lifting; that is the implementation we are using now - Change the AST - Change of direction on mangling - More precise scoping rules Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

* Improve PEP 649's description of its semantics. * Stipulate that name resolution for annotations under 649 must be *identical* to stock semantics. * *Don't* specify exactly the mechanism that conformant implementations must use to implement 649. Instead, *do* describe how CPython might do it, but leave the actual details of how to implement 649 up to each language implementation. * Incorporate Jelle's suggestion that the formats for inspect.get_annotations() be in an enum.IntEnum. * Fix lint. * PEP 693: Postpone 3.12.0b1 by two weeks (#3139) * PEP 695: Lazy evaluation, concrete scoping semantics, other changes (#3122) - Lazy evaluation means that referencing a later type variable works at runtime - Disallow walrus in TypeVar bounds, and also disallow yield/yield from/await in the same contexts - Remove rejection of lambda lifting; that is the implementation we are using now - Change the AST - Change of direction on mangling - More precise scoping rules Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> * Incorporate changes from feedback, mark Accepted. * Fix PEP 12 header *order compliance*. Wow. * Fix Sphinx complaints. * Make enum consistent, flesh out observed semantics. * Add "Resolution" header, as pointed out by Hugo. * Switch to other URL for Resolution header. * Apply ``global_enum`` to ``inspect.AnnotationFormat`` * Final? text / semantics cleanup pass. * "accept" -> "accepts". Bettering my Englishes. * Add new "post history" reflecting the updates. * Update post history with all conversations, courtesy CAM! Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM> * Fix typo. Thanks, Emily! Co-authored-by: Emily Morehouse <emilyemorehouse@gmail.com> * Add "Discussions-To" header. Thanks, CAM! Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM> * Attempt to satisfy "validate-post-history" hook. --------- Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com> Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com> Co-authored-by: C.A.M. Gerlach <CAM.Gerlach@Gerlach.CAM> Co-authored-by: Emily Morehouse <emilyemorehouse@gmail.com>

PEP 695: Changes proposed as a result of implementation work

5b3bbee

erictraut approved these changes Apr 24, 2023

View reviewed changes

gvanrossum reviewed Apr 24, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst

Comment thread pep-0695.rst

Comment thread pep-0695.rst

JelleZijlstra added 3 commits April 24, 2023 18:09

It does not all have to be C

23a6c4d

If PEP 649 is accepted, we should support formats

a20d27c

Two more updates:

bb3c770

- Lazy evaluation means that referencing a later type variable works at runtime - Disallow walrus in TypeVar bounds, and also disallow yield/yield from/await in the same contexts

JelleZijlstra mentioned this pull request Apr 25, 2023

gh-103763: Implement PEP 695 python/cpython#103764

Merged

Remove rejection of lambda lifting; that is the implementation we are…

af90161

… using now

JelleZijlstra commented Apr 25, 2023

View reviewed changes

Comment thread pep-0695.rst

erictraut reviewed Apr 25, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst

Longer discussion of lazy eval

8b43966

JelleZijlstra commented Apr 26, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

Update pep-0695.rst

6d80750

JelleZijlstra changed the title PEP 695: Changes proposed as a result of implementation work PEP 695: Lazy evaluation and other small changes Apr 26, 2023

Merge branch 'main' into pep695change

3037e47

JelleZijlstra marked this pull request as ready for review April 26, 2023 16:27

JelleZijlstra added the DO-NOT-MERGE label Apr 26, 2023

AlexWaygood reviewed Apr 26, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst Outdated

JelleZijlstra added 3 commits April 26, 2023 11:37

Some comments from Alex

e9969bd

Rewording

868bf7d

Another Alex suggestion

a009ae4

gvanrossum approved these changes Apr 26, 2023

View reviewed changes

Comment thread pep-0695.rst

JelleZijlstra added 2 commits April 26, 2023 15:18

Change the AST

608dedf

Change of direction on mangling

a064f11

gvanrossum approved these changes Apr 27, 2023

View reviewed changes

JelleZijlstra added 2 commits May 2, 2023 15:12

Merge branch 'main' into pep695change

7996ecc

More precise scoping rules

f446963

JelleZijlstra changed the title PEP 695: Lazy evaluation and other small changes PEP 695: Lazy evaluation, concrete scoping semantics, other changes May 4, 2023

AlexWaygood reviewed May 4, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

Comment thread pep-0695.rst Outdated

Update pep-0695.rst

368cc48

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

JelleZijlstra commented May 4, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

JelleZijlstra added 2 commits May 4, 2023 08:33

Update pep-0695.rst

607373c

Update pep-0695.rst

90101e1

markshannon reviewed May 4, 2023

View reviewed changes

Comment thread pep-0695.rst

Comment thread pep-0695.rst

JelleZijlstra added 2 commits May 4, 2023 08:53

Clarify where def695 scopes are used

62d4d50

GitHub downtime made me write more

0e5160f

hauntsaninja reviewed May 5, 2023

View reviewed changes

Comment thread pep-0695.rst Outdated

fix pep number typo

1654a1b

Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>

JelleZijlstra merged commit 5cea9b5 into python:main May 8, 2023

JelleZijlstra deleted the pep695change branch May 8, 2023 18:21

coderabbitai Bot mentioned this pull request Aug 19, 2026

fix(meaningless-vars): complete scope-aware rename planning alessio-locatelli/ruff-extra-rules#186

Merged

PEP 695: Lazy evaluation, concrete scoping semantics, other changes by JelleZijlstra · Pull Request #3122 · python/peps · GitHub

Uh oh!

Uh oh!

PEP 695: Lazy evaluation, concrete scoping semantics, other changes - #3122

PEP 695: Lazy evaluation, concrete scoping semantics, other changes#3122JelleZijlstra merged 21 commits intopython:mainpython/peps:mainfrom JelleZijlstra:pep695changeJelleZijlstra/peps:pep695changeCopy head branch name to clipboard

Conversation

JelleZijlstra commented Apr 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erictraut left a comment

Choose a reason for hiding this comment

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Apr 25, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented Apr 26, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cdce8p commented Apr 26, 2023

Uh oh!

JelleZijlstra commented Apr 27, 2023

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

erictraut commented Apr 27, 2023

Uh oh!

JelleZijlstra commented Apr 27, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JelleZijlstra commented May 4, 2023

Uh oh!

Uh oh!

JelleZijlstra commented May 8, 2023

Uh oh!

hugovk commented May 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

PEP 695: Lazy evaluation, concrete scoping semantics, other changes#3122
JelleZijlstra merged 21 commits into
python:mainfrom
JelleZijlstra:pep695change

JelleZijlstra commented Apr 24, 2023 •
edited
Loading