gh-114053: Fix another edge case involving `get_type_hints`, PEP 695 and PEP 563 by AlexWaygood · Pull Request #120272 · python/cpython · GitHub

This PR fixes a bug in #118009 that can be seen in the following REPL demo, where get_type_hints under from __future__ import annotations semantics won't detect if a variable defined in a type-parameter namespace has been overidden in a local namespace:

Python 3.14.0a0 (heads/getannotations-nameerror:ec4848f2c8b, Jun  8 2024, 15:28:33) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo[T]:
...     T = int
...     x: T
...     
>>> Foo.__annotations__
{'x': <class 'int'>}
>>> from __future__ import annotations
>>> class Foo[T]:
...     T = int
...     x: T
...     
>>> from typing import get_type_hints
>>> get_type_hints(Foo)
{'x': T}

The cause of the bug is that get_type_hints does this weird swapping of globals and locals, and the logic added in #118009 doesn't account for that -- it injects the type parameters into the dictionary called localns (which is actually the globals) when it should be injecting the type parameters into globalns (which is actually the locals).

I actually added a test for this exact edge case in #118009, which should have caught the bug. But there was a bug in the test, because the test was trying to be too clever, and I got mixed up between ^ and & when you're using sets. Had I not tried to write such a clever test, I would have caught the bug.

Issue: typing.get_type_hints fails when passed a class with PEP 695 type parameters and PEP 563 is enabled #114053

…P 695 and PEP 563

Can you add a test where T also exists in the global scope, ensuring that the type param is picked up over the global?

Can you add a test where T also exists in the global scope, ensuring that the type param is picked up over the global?

Thanks, good catch -- that test failed. I added it and fixed it.

…thon into gettypehints-bug

Okay, this still doesn't get things right in all cases :(

>>> from __future__ import annotations
>>> from typing import get_type_hints
>>> Eggs, Spam = int, str
>>> def f[Eggs, **Spam](x: Eggs, y: Spam): pass
... 
>>> get_type_hints(f)
{'x': <class 'int'>, 'y': <class 'str'>}

Okay, this still doesn't get things right in all cases :(

>>> from __future__ import annotations
>>> from typing import get_type_hints
>>> Eggs, Spam = int, str
>>> def f[Eggs, **Spam](x: Eggs, y: Spam): pass
... 
>>> get_type_hints(f)
{'x': <class 'int'>, 'y': <class 'str'>}

Should have been fixed in 2fbcd7b.

Why do we need the __forward_is_class__ condition? I don't understand why class-scoped forwardrefs are special here.

For functions, names in type_params should always take highest priority, and this is implemented by inserting the type params into the local namespace (which is here called globalns) and popping them from the global namespace (which is here called localns). For classes, we only want to give type_params highest priority if they haven't been overridden in the local scope, however.

Why do we need to do this? Shouldn't the local namespace override the type params?

Test case:

class X[T]:
    T = int
    x: T

I already have that test case included in this PR, and it fails unless we have this block of code here. The local namespace shoudl indeed override the type param, but the locals here are found in the globalns, and the globals here are found in the localns variable. This highly confusing situation is because the globals and locals are swapped higher up in the call stack here, for backwards compatibility reasons if you believe the comment:

cpython/Lib/typing.py

Lines 2419 to 2426 in a3711af

    
           if localns is None and globalns is None: 
        
               # This is surprising, but required.  Before Python 3.10, 
        
               # get_type_hints only evaluated the globalns of 
        
               # a class.  To maintain backwards compatibility, we reverse 
        
               # the globalns and localns order so that eval() looks into 
        
               # *base_globals* first rather than *base_locals*. 
        
               # This only affects ForwardRefs. 
        
               base_globals, base_locals = base_locals, base_globals

I saw that comment, maybe we should find a way to get rid of that compatibility hack.

I don't really see a way round it, short of deprecating get_type_hints altogether. Which... I'm not 100% against, but it would be quite disruptive.

There could be other approaches, like adding new keyword arguments. Will have to think about it more.

If you decide to add keywords arguments or deprecate things, please tell me in advance so that I can be reactive on Sphinx' side if needed.

@JelleZijlstra, are you okay with this going in? I'd love to get this into the next 3.13 beta if possible (but at the same time, the bug also exists in Python 3.12, so it wouldn't be a disaster if it waited until beta 4).

I agree that the implementation is kinda weird but I'm not sure there's a better way of doing it until we've sorted out the strange globals/locals swapping (which I think would probably be harder to backport to 3.12).

Thanks @AlexWaygood for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

…P 695 and PEP 563 (pythonGH-120272) (cherry picked from commit 2d3187b) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

GH-121003 is a backport of this pull request to the 3.13 branch.

…P 695 and PEP 563 (pythonGH-120272) (cherry picked from commit 2d3187b) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

GH-121004 is a backport of this pull request to the 3.12 branch.

…EP 695 and PEP 563 (GH-120272) (#121003) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

…EP 695 and PEP 563 (GH-120272) (#121004) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

…P 695 and PEP 563 (python#120272)

pythongh-114053 : Fix another edge case involving get_type_hints , PE…

879abeb

…P 695 and PEP 563

AlexWaygood added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Jun 8, 2024

AlexWaygood requested a review from JelleZijlstra as a code owner June 8, 2024 14:52

bedevere-app Bot mentioned this pull request Jun 8, 2024

typing.get_type_hints fails when passed a class with PEP 695 type parameters and PEP 563 is enabled #114053

Closed

bedevere-app Bot added the awaiting core review label Jun 8, 2024

AlexWaygood added the topic-typing label Jun 8, 2024

AlexWaygood mentioned this pull request Jun 8, 2024

gh-114053: Fix bad interaction of PEP 695, PEP 563 and inspect.get_annotations #120270

Merged

JelleZijlstra reviewed Jun 9, 2024

View reviewed changes

AlexWaygood and others added 4 commits June 9, 2024 11:29

Merge branch 'main' into gettypehints-bug

065914e

Add failing test

373b23c

Fix failing test

4f95b1c

remove redundant assertions in test

ffdba91

AlexWaygood requested a review from JelleZijlstra June 9, 2024 11:39

JelleZijlstra reviewed Jun 9, 2024

View reviewed changes

Comment thread Lib/typing.py Outdated

AlexWaygood added 2 commits June 9, 2024 13:27

Don't mutate globals!

5505d97

Merge branch 'gettypehints-bug' of https://github.com/AlexWaygood/cpy…

b05ca1a

…thon into gettypehints-bug

AlexWaygood requested a review from JelleZijlstra June 9, 2024 12:29

simplify

bccaea2

picnixz reviewed Jun 9, 2024

View reviewed changes

Comment thread Lib/typing.py Outdated

Fix yet another bug and add many more tests

2fbcd7b

JelleZijlstra reviewed Jun 13, 2024

View reviewed changes

Merge branch 'main' into gettypehints-bug

432e749

JelleZijlstra approved these changes Jun 25, 2024

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting core review labels Jun 25, 2024

AlexWaygood merged commit 2d3187b into python:main Jun 25, 2024

AlexWaygood deleted the gettypehints-bug branch June 25, 2024 15:53

bedevere-app Bot removed the awaiting merge label Jun 25, 2024

bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Jun 25, 2024

bedevere-app Bot removed the needs backport to 3.12 only security fixes label Jun 25, 2024

AlexWaygood added a commit that referenced this pull request Jun 25, 2024

[3.13] gh-114053 : Fix another edge case involving get_type_hints , P…

f4f8a71

…EP 695 and PEP 563 (GH-120272) (#121003) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood added a commit that referenced this pull request Jun 25, 2024

[3.12] gh-114053 : Fix another edge case involving get_type_hints , P…

a85241d

…EP 695 and PEP 563 (GH-120272) (#121004) Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

mrahtz pushed a commit to mrahtz/cpython that referenced this pull request Jun 30, 2024

pythongh-114053 : Fix another edge case involving get_type_hints , PE…

7357059

…P 695 and PEP 563 (python#120272)

noahbkim pushed a commit to hudson-trading/cpython that referenced this pull request Jul 11, 2024

pythongh-114053 : Fix another edge case involving get_type_hints , PE…

7dcc9e0

…P 695 and PEP 563 (python#120272)

estyxx pushed a commit to estyxx/cpython that referenced this pull request Jul 17, 2024

pythongh-114053 : Fix another edge case involving get_type_hints , PE…

f62761c

…P 695 and PEP 563 (python#120272)

JelleZijlstra added a commit to JelleZijlstra/cpython that referenced this pull request Jul 21, 2024

Reapply python#120272

1f3a63c

+                              globalns, localns = dict(globalns), dict(localns)
+                              for param in type_params:
+                                  param_name = param.__name__
+                                  if not self.__forward_is_class__ or param_name not in globalns:

+                                  param_name = param.__name__
+                                  if not self.__forward_is_class__ or param_name not in globalns:
+                                      globalns[param_name] = param
+                                      localns.pop(param_name, None)

	if localns is None and globalns is None:
	# This is surprising, but required. Before Python 3.10,
	# get_type_hints only evaluated the globalns of
	# a class. To maintain backwards compatibility, we reverse
	# the globalns and localns order so that eval() looks into
	# base_globals first rather than base_locals.
	# This only affects ForwardRefs.
	base_globals, base_locals = base_locals, base_globals

gh-114053: Fix another edge case involving `get_type_hints`, PEP 695 and PEP 563 by AlexWaygood · Pull Request #120272 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-114053: Fix another edge case involving get_type_hints, PEP 695 and PEP 563 - #120272

gh-114053: Fix another edge case involving get_type_hints, PEP 695 and PEP 563#120272AlexWaygood merged 10 commits intopython:mainpython/cpython:mainfrom AlexWaygood:gettypehints-bugAlexWaygood/cpython:gettypehints-bugCopy head branch name to clipboard

Conversation

AlexWaygood commented Jun 8, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

AlexWaygood commented Jun 9, 2024

Uh oh!

Uh oh!

Uh oh!

AlexWaygood commented Jun 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexWaygood commented Jun 9, 2024

Uh oh!

JelleZijlstra Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

AlexWaygood Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Jun 13, 2024

Choose a reason for hiding this comment

Uh oh!

picnixz Jun 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexWaygood commented Jun 25, 2024

Uh oh!

miss-islington-app Bot commented Jun 25, 2024

Uh oh!

bedevere-app Bot commented Jun 25, 2024

Uh oh!

bedevere-app Bot commented Jun 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gh-114053: Fix another edge case involving get_type_hints, PEP 695 and PEP 563#120272
AlexWaygood merged 10 commits into
python:mainfrom
AlexWaygood:gettypehints-bug

AlexWaygood commented Jun 8, 2024 •
edited by bedevere-app Bot
Loading

AlexWaygood commented Jun 9, 2024 •
edited
Loading

AlexWaygood Jun 13, 2024 •
edited
Loading

picnixz Jun 14, 2024 •
edited
Loading