bpo-41370: Evaluate strings as forward refs in PEP 585 generics by NiklasRosenstein · Pull Request #30900 · python/cpython · GitHub

This Pull Requests suggests a change to typing._eval_type() that considers strings in PEP 585 generics (ie. instances of types.GenericAlias) as forward references. This is necessary because ForwardRef is not and likely will not be implemented in C, and thus strings used as forward references in PEP 585 are not currently evaluated by typing.get_type_hints().

https://bugs.python.org/issue41370

A test case that uses assertIs() currently fails because in the current state _eval_type() creates a copy of the same generic alias with transformed arguments, thus for any PEP 585 generic alias x, the comparison x is typing.get_type_hints(func)['X'] will evaluate to False, assuming func has a field/argument annotation X: x. I think that this can be worked around, and happy to propose an updated implementation.

I created this PR as a proof of concept until a there is consent that in spirit this change to get_type_hints() is something that should happen. So for now I'll keep it as it is.

======================================================================
FAIL: test_get_type_hints_annotated (__main__.GetTypeHintTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/niklas/git/cpython/Lib/test/test_typing.py", line 3298, in test_get_type_hints_annotated
    self.assertIs(
    ^^^^^^^^^^^^^^
AssertionError: tuple[typing.Annotated[~T, (1, 0)], ...] is not tuple[typing.Annotated[~T, (1, 0)], ...]

Open todos/questions

Fix test_get_type_hints_annotated in test_typing.py
Do we actually need to treat types.GenericAlias separately or is it safe to assume that a string argument is always a forward reference?
Backport to 3.10 as a bugfix?

Co-Authored-By: Guido van Rossum guido@python.org

https://bugs.python.org/issue41370

…`ForwardRef` before evaluation (https://bugs.python.org/issue41370)

…ith forward refs

Looks pretty good! I have one small suggestion. This also needs a NEWS entry, which you can add with blurb.

This is the code that I posted to the issue. But I also posted a few questions there, and those should at least be answered by tests:

But would that be enough? The algorithm would have to recursively dive into args to see if there's a string hidden deep inside, e.g. list[tuple[int, list["N"]]].
And what if the user writes something hybrid, like List[list["N"]]? What other cases would we need to cover?
And can we sell this as a bugfix for 3.10, or will this be a new feature in 3.11?
How will it interact with from future import annotations?

But would that be enough? The algorithm would have to recursively dive into args to see if there's a string hidden deep inside, e.g. list[tuple[int, list["N"]]].

And what if the user writes something hybrid, like List[list["N"]]? What other cases would we need to cover?

The tests already cover the sort of case you list here (there's a dict[int, list[List[list["T"]]] I think). We could add more variants though.

And can we sell this as a bugfix for 3.10, or will this be a new feature in 3.11?

I think this can be considered a bugfix. get_type_hints() is supposed to evaluate forward references but it didn't in all cases.

How will it interact with from future import annotations?

That's another interesting test case. It would basically give us two levels of stringification, like x: 'list["N"]'. We should test that get_type_hints() resolves both levels of strings.

Would also be good to test a recursive forward ref:

X = list["X"]
def f(x: X): ...
get_type_hints(f)  # hopefully no RecursionError

We guard against this sort of thing already, but we should validate it works in this case too.

You can add a NEWS entry using https://blurb-it.herokuapp.com/ as an alternative to the CLI :)

[@gvanrossum]

This is the code that I posted to the issue.

Yes, sorry for not mentioning it. Same code, different location.

[@JelleZijlstra]

I've added two more unit tests for the cases Guido and you mentioned.

[@AlexWaygood]

Thanks, added it.

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Do we need to pass globals and locals to ForwardRef here? 🤔

I may not be knowledgable enough about the typing code base, but I don't see a globals/locals argument for ForwardRef. Do you mean the module? I couldn't find out how to use/what to pass to that argument exactly.

It's too late for that anyway. The ForwardRef class has an optional module= argument which may be used to resolve references, but the best we could do at this point is passing globalns.get("__name__"), which would just end up with the same globalns as we are using anyway.

This suggests there could still be scenarios where this will fail, esp. when a type alias defined in one module is used in another. But that would only be solvable by recording the module at the time the alias is being defined, and we've already said we wouldn't go that far (since it would require a ForwardRef implementation in C).

Actually I'm not sure if we can find a good way to make the Annotated equality unit test work (test_get_type_hints_annotated) with this new behaviour. Maybe this is a side effect of this change that has to be accepted, but I'm happy to hear suggestions.

Actually I'm not sure if we can find a good way to make the Annotated equality unit test work (test_get_type_hints_annotated) with this new behaviour

We can just change those tests from assertIs to assertEquals. It doesn't seem important that the same object is returned.

LGTM.

I'll wait landing this until 3.11a6 is out, so as not to disturb the release process (which is already stressed by a last-minute blocker).

It's too late for that anyway. The ForwardRef class has an optional module= argument which may be used to resolve references, but the best we could do at this point is passing globalns.get("__name__"), which would just end up with the same globalns as we are using anyway.

This suggests there could still be scenarios where this will fail, esp. when a type alias defined in one module is used in another. But that would only be solvable by recording the module at the time the alias is being defined, and we've already said we wouldn't go that far (since it would require a ForwardRef implementation in C).

There's no C code here, and 3.11a6 seems to be still delayed, so I'm just landing this. Sorry for the confusion.

Update: This didn't make 3.11a6. The release process was farther along than I realized.

NiklasRosenstein added 2 commits January 25, 2022 23:11

convert strings in PEP 585 generic aliases (types.GenericAlias) to …

e820e3a

…`ForwardRef` before evaluation (https://bugs.python.org/issue41370)

add unittest for new get_type_hints() behaviour on PEP 585 generics w…

d7f57c7

…ith forward refs

NiklasRosenstein requested review from Fidget-Spinner and gvanrossum as code owners January 25, 2022 22:39

This comment has been minimized.

Sign in to view

the-knights-who-say-ni added the CLA not signed label Jan 25, 2022

bedevere-bot added the awaiting review label Jan 25, 2022

gvanrossum marked this pull request as draft January 25, 2022 23:03

JelleZijlstra self-requested a review January 26, 2022 17:40

JelleZijlstra requested changes Jan 27, 2022

View reviewed changes

Comment thread Lib/typing.py Outdated

bedevere-bot added awaiting core review and removed awaiting review labels Jan 27, 2022

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Jan 27, 2022

gvanrossum reviewed Jan 27, 2022

View reviewed changes

NiklasRosenstein added 2 commits January 27, 2022 12:44

add unit test for from __future__ import annotations

26ab740

add test for recursive type

a04bd5d

📜🤖 Added by blurb_it.

3416713

NiklasRosenstein and others added 2 commits January 27, 2022 03:56

Double backticks because RST.

30a3564

Update Lib/typing.py

13a3505

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

AlexWaygood reviewed Jan 27, 2022

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2022-01-27-11-54-16.bpo-41370.gYxCPE.rst Outdated

Update Misc/NEWS.d/next/Library/2022-01-27-11-54-16.bpo-41370 .gYxCPE.rst

33d286a

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

JelleZijlstra approved these changes Feb 19, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Feb 19, 2022

JelleZijlstra self-assigned this Feb 19, 2022

sobolevn reviewed Feb 19, 2022

View reviewed changes

Merge branch 'main' into issue41370

6cd96b1

NiklasRosenstein marked this pull request as ready for review February 22, 2022 23:11

Replace assertIs with assertEqual in test_get_type_hints_annotated

39c5b06

NiklasRosenstein requested review from AlexWaygood, JelleZijlstra and gvanrossum March 3, 2022 16:19

gvanrossum approved these changes Mar 3, 2022

View reviewed changes

JelleZijlstra approved these changes Mar 4, 2022

View reviewed changes

gvanrossum merged commit b465b60 into python:main Mar 7, 2022

bedevere-bot removed the awaiting merge label Mar 7, 2022

JelleZijlstra mentioned this pull request Apr 13, 2022

Add typing to 3.11 What's New #91491

Closed

This was referenced Apr 17, 2022

typing.get_type_hints() works for Callable[[], None] from the typing module, but not collections.abc #91621

Closed

gh-91621: Fix typing.get_type_hints for collections.abc.Callable #91656

Merged

wyfo mannequin mentioned this pull request Apr 20, 2022

PEP 585 and ForwardRef #85542

Closed

This was referenced May 8, 2022

Support subcriptable types and new Union Type tefra/xsdata#681

Closed

Add option to use union operator and subscriptable types for type annotations tefra/xsdata#680

Closed

AlexWaygood removed their request for review June 5, 2022 11:03

egparedes mentioned this pull request Jan 9, 2024

Feature request: backport typing fix in 3.11 to evaluate strings as forward refs in PEP 585 generics alexmojaki/eval_type_backport#10

Open

Viicos mentioned this pull request Oct 17, 2024

Support usage of type with typing.Self and type aliases pydantic/pydantic#10621

Merged

5 tasks

Viicos mentioned this pull request Apr 16, 2025

Only mark model as complete once all fields are complete pydantic/pydantic#11759

Merged

5 tasks

Viicos mentioned this pull request Dec 4, 2025

No ForwardRef in model_fields annotations with list['Model'] field type with legacy-style annotations (w/o from __future__ import annotations) pydantic/pydantic#12610

Open

1 task

Viicos mentioned this pull request Aug 4, 2026

Require types passed to GenerateSchema.generate_schema() to be evaluated pydantic/pydantic#13583

Draft

5 tasks

                   if isinstance(t, (_GenericAlias, GenericAlias, types.UnionType)):
+                      if isinstance(t, GenericAlias):
+                          args = tuple(
+                              ForwardRef(arg) if isinstance(arg, str) else arg

                   if isinstance(t, (_GenericAlias, GenericAlias, types.UnionType)):
+                      if isinstance(t, GenericAlias):
+                          args = tuple(
+                              ForwardRef(arg) if isinstance(arg, str) else arg

bpo-41370: Evaluate strings as forward refs in PEP 585 generics by NiklasRosenstein · Pull Request #30900 · python/cpython · GitHub

Uh oh!

Uh oh!

bpo-41370: Evaluate strings as forward refs in PEP 585 generics - #30900

bpo-41370: Evaluate strings as forward refs in PEP 585 generics#30900gvanrossum merged 10 commits intopython:mainpython/cpython:mainfrom NiklasRosenstein:issue41370Copy head branch name to clipboard

Conversation

NiklasRosenstein commented Jan 25, 2022 • edited by gvanrossum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra commented Jan 27, 2022

Uh oh!

JelleZijlstra commented Jan 27, 2022

Uh oh!

AlexWaygood commented Jan 27, 2022

Uh oh!

NiklasRosenstein commented Jan 27, 2022

Uh oh!

Uh oh!

sobolevn Feb 19, 2022

Choose a reason for hiding this comment

Uh oh!

NiklasRosenstein Feb 22, 2022

Choose a reason for hiding this comment

Uh oh!

gvanrossum Mar 3, 2022

Choose a reason for hiding this comment

Uh oh!

NiklasRosenstein commented Feb 22, 2022

Uh oh!

JelleZijlstra commented Mar 2, 2022

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

gvanrossum Mar 3, 2022

Choose a reason for hiding this comment

Uh oh!

gvanrossum commented Mar 7, 2022

Uh oh!

gvanrossum commented Mar 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

bpo-41370: Evaluate strings as forward refs in PEP 585 generics#30900
gvanrossum merged 10 commits into
python:mainfrom
NiklasRosenstein:issue41370

NiklasRosenstein commented Jan 25, 2022 •
edited by gvanrossum
Loading

gvanrossum commented Mar 7, 2022 •
edited
Loading