Respect explicit return type of new() by ilevkivskyi · Pull Request #21441 · python/mypy · GitHub

Fixes #8330
Fixes #15182
Closes #16020
Closes #14471
Fixes #13824
Fixes #14502

With this PR will still give an error if the explicit return __new__() is not a subtype of current class, but now we will actually use it. There are two exceptions (to preserve backwards compatibility):

If the return type is Any with still use current class as the return type.
If the explicit return type comes from a superclass and is a supertype of implicit return type.

class A:
    def __new__(cls): ...
reveal_type(A())  # still __main__.A

class B:
    def __new__(cls) -> B:
        return cls()
class C(B): ...
reveal_type(C())  # still __main__.C

This uses a more principled implementation than some earlier attempts: adding a new dedicated attribute to CallableType for this purpose. Some comments:

This PR has a bit for boilerplate, but this is expected. When adding a new attribute to a type, one needs to update most visitors.
While doing the above I noticed that CallableType.type_guard and CallableType.type_is were not handled in dependency visitors (neither coarse-grained nor fine-grained). IMO they definitely should be handled there, so I now handle them.
I try to reduce the size of CallableType a bit to compensate for new attribute by removing (rarely used) min_args attribute. I also use compact flags serialization.
I skimmed the code base and updated all places where we "casually" use CallableType.ret_type for various type object edge cases.
Note that I don't assert that instance_type is set when is_type_obj() returns True. This is mostly to avoid breaking 3rd party plugins.
I didn't add a test case for each edge case, but I added (improved) test cases from Believe more __new__ return types #16020 plus some more. Suggestions for more test cases are welcome.
It looks like this exposes a pre-existing bug where we leaked type variables in type[T]. It is relatively niche edge case, but it looks important for NumPy stubs, so I am trying to fix it here.

mypy_primer looks sad, but I guess this is what people want. I spot-checked the new errors, and they all look correct. The only one suspicious thing in one new Any error in pandas-stubs.

cc @Dr-Irv for all the new errors in pandas. IIUC these are because of things like this https://github.com/pandas-dev/pandas/blob/47f6ccdbec98998bd675324fb6f049794ad03acf/pandas/_libs/tslibs/timestamps.pyi#L44-L60

class Timestamp(datetime):
    ...
    def __new__(  # type: ignore[misc]
        cls: type[Self],
        # ...
    ) -> Self | NaTType: ...

mypy_primer looks sad, but I guess this is what people want. I spot-checked the new errors, and they all look correct. The only one suspicious thing in one new Any error in pandas-stubs.

Yes, I agree that is suspicious.

cc @Dr-Irv for all the new errors in pandas. IIUC these are because of things like this https://github.com/pandas-dev/pandas/blob/47f6ccdbec98998bd675324fb6f049794ad03acf/pandas/_libs/tslibs/timestamps.pyi#L44-L60
class Timestamp(datetime):
    ...
    def __new__(  # type: ignore[misc]
        cls: type[Self],
        # ...
    ) -> Self | NaTType: ...

We should probably copy the PYI for Timestamp and Timedelta and NAType from pandas-stubs into the pandas source at some point to address this.

I have on my TODO list to first make the pandas source compatible with the latest release of mypy. So when these changes in this PR get released, I'll do a similar task then.

If the return type is Any with still use current class as the return type.

If the explicit return type comes from a superclass and is a supertype of implicit return type.

Would the following alternatives also preserve backwards compatibility just as well, but reflect explicit intention better?

If the return type is unannotated (i.e. an explicit Any is not used):

class A:
    def __new__(cls, arg: object): ...

class B:
    def __new__(cls, arg: object) -> Any: ...

class C:
    def __new__(cls, arg: object) -> "bad + annotation": ...

reveal_type(A())  # `__main__.A`
reveal_type(B())  # `Any`
reveal_type(C())  # `Any`

If the explicit return type comes from a superclass and the superclass's __new__'s return type is the same as the class:
```
class A:
    def __new__(cls) -> A: ...
class B(A):
    def __new__(cls) -> A: ...

class C(B): ...
class D(A): ...

reveal_type(C())  # Would it be possible for this to be `__main__.A`?
reveal_type(D())  # `__main__.D`
```
This pattern came up when I was trying to write stubs for symengine.py a while back (maybe it affects SymPy too), where they have a root Expr class, and most of Expr's subclasses have a __new__ that frequently returns something that is not an instance of the subclass.

I'm hoping that "__new__'s return type being the same as a class" would handle the vast majority of backwards compatibility cases because it was common before typing.Self was introduced.

@bzoracler Yeah, I actually was thinking about both these options myself. Reasons why I didn't do this:

We have a specific test checking that explicit def __new__() -> Any is ignored, see testNewReturnType2. I don't know the motivation, so decided to simply not touch that.
The definition of "the same as the class" gets a little blurry in case of generic classes. For example, is plain -> C (and not say -> C[T]) considered the same, etc.

However, I don't have strong feelings on either, so if others will chime in, I am happy reconsider.

It looks like this exposes a pre-existing bug where we leaked type variables in type[T]. It is relatively niche edge case, but it looks important for NumPy stubs, so I am trying to fix it here.

OK, mypy_primer looks good now. @hauntsaninja @JukkaL this is ready for review now. You may also want to say what do you think about this comment #21441 (comment)

I mostly agree with bzoracler:

if there is an explicit return Any, ideally we should reveal Any
the biggest reason for back compat is code that predates Self, so the heuristic for return typing matching current class makes sense to me

Will look at the diff in a little bit!

OMG! We are actually handling arbitrary class object with return type Any as builtins.type, see

mypy/mypy/checkexpr.py

Line 1847 in e53693b

and is_equivalent(callee.ret_type, self.named_type("builtins.type"))

I am surprised this didn't cause a ton of issues.

@hauntsaninja I tried your (or really @bzoracler's) suggestions. Looking at the primer I am slightly worried this may cause false negatives. It looks like there is still some code doing this:

class C:
    def __new__(cls) -> Any:  # to silence a mypy error from some old version
        return super().__new__(cls)

Otherwise however the primer looks as expected.

class C:
    def __new__(cls) -> Any:  # to silence a mypy error from some old version
        return super().__new__(cls)

In this example, are we saying that return super().__new__(cls) returns something that an old version of mypy rejected, so the user put Any as the return type (instead of, say, C)?

Hmm, I can only think of this being an issue if C was a generic type and the user (or the old mypy) could not properly get the generic parameterisation of -> C[...] to match return super().__new__(...). In this case, I can see why keeping the current behaviour, where the return type is C[...] even if the user wrote -> Any, would work. I don't recall coming across this situation (to be fair it sounds quite easily missable if there's no inline comment/docstring to explain -> Any), so I don't have strong opinions on this.

@bzoracler OK, I did some digging and it looks like mypy (until v0.920) used to infer super().__new__(cls) as Any which resulted in Returning Any from function declared to return "C" (e.g. for people with --strict or --warn-return-any). This is why some repos still have this.

@hauntsaninja

Will look at the diff in a little bit!

Ping on this one.

Did a quick pass, not a full review.

What about partial signature, but implicit Any return type? For example, def __new__(self, x: int = 0): ...

Added a test (works as expected, we infer C as constructor return type).

Test if A is compatible with Callable[[], Foo] or Callable[[], A]? Also similarly against callback protocol.

Is it worth testing type joins involving __new__ return type overrides?

Added a bunch of tests for subtyping (btw this uncovered one type combination I missed). Also added few basic things for joins. All works as expected.

This feels a bit inconsistent and causes false negatives. Would changing cause many issues in existing code?

What do you mean by inconsistent? How is this different from say:

class B:
    x: str
class C(B):
    x: int  # Error
class D(C): ...

reveal_type(C().x)  # int
reveal_type(D().x)  # int

Also this is kind of the whole point of what people want: respect the explicit return type.

Ok, I'm convinced that the new behavior is good.

Looks good -- nice to have this working finally. Left some minor comments, feel free to merge once you've addressed those that make sense.

Ok, I'm convinced that the new behavior is good.

Test Callable[[], T] -- T should be inferred from the __new__ return type when relevant.

Yep, there was the same issue in constraints.py as in subtypes.py.

What about testing isinstance narrowing with a type that has an explicit __new__ return type?

Add an incremental test case?

I briefly searched the tracker for issues with __new__ and found some more that this will fix. I didn't look too deep however. I am going to add a regression test that seems important.

Diff from mypy_primer, showing the effect of this PR on open source code:

zope.interface (https://github.com/zopefoundation/zope.interface)
- src/zope/interface/common/tests/test_collections.py:88: error: Incompatible types in assignment (expression has type "set[type[array[int]]]", base class "VerifyClassMixin" defined the type as "tuple[()]")  [assignment]
+ src/zope/interface/common/tests/test_collections.py:88: error: Incompatible types in assignment (expression has type "set[type[array[_T]]]", base class "VerifyClassMixin" defined the type as "tuple[()]")  [assignment]

comtypes (https://github.com/enthought/comtypes)
- comtypes/server/register.py:226: error: Incompatible types in assignment (expression has type "reversed[tuple[int, str]]", variable has type "list[tuple[int, str]]")  [assignment]
+ comtypes/server/register.py:226: error: Incompatible types in assignment (expression has type "Iterator[tuple[int, str]]", variable has type "list[tuple[int, str]]")  [assignment]

antidote (https://github.com/Finistere/antidote)
+ src/antidote/lib/interface_ext/predicate.py:118: error: Returning Any from function declared to return "NeutralWeight"  [no-any-return]
+ src/antidote/lib/interface_ext/qualifier.py:134: error: Returning Any from function declared to return "NeutralWeight"  [no-any-return]
+ tests/lib/interface/test_custom.py:205: error: Returning Any from function declared to return "NeutralWeight"  [no-any-return]

discord.py (https://github.com/Rapptz/discord.py)
- discord/abc.py:1815: error: Incompatible types in assignment (expression has type "reversed[MessagePin]", variable has type "list[MessagePin]")  [assignment]
+ discord/abc.py:1815: error: Incompatible types in assignment (expression has type "Iterator[MessagePin]", variable has type "list[MessagePin]")  [assignment]

sympy (https://github.com/sympy/sympy)
+ sympy/core/evalf.py:997: error: Argument 1 to "evalf_add" has incompatible type "Expr"; expected "Add"  [arg-type]
+ sympy/core/function.py:449: error: Unused "type: ignore" comment  [unused-ignore]
+ sympy/printing/smtlib.py:188: error: Argument 1 to "_print_Unequality" of "SMTLibPrinter" has incompatible type "Unequality | BooleanFalse | BooleanTrue"; expected "Unequality"  [arg-type]

tornado (https://github.com/tornadoweb/tornado)
+ tornado/test/ioloop_test.py:433: error: Unused "type: ignore" comment  [unused-ignore]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/tseries/frequencies.py:336: error: Incompatible return value type (got "Timestamp | NaTType", expected "Timestamp")  [return-value]
- pandas/core/dtypes/cast.py:698: error: Non-overlapping identity check (left operand type: "Timestamp", right operand type: "NaTType")  [comparison-overlap]
- pandas/core/dtypes/cast.py:711: error: Non-overlapping identity check (left operand type: "Timedelta", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/dtypes/cast.py:159: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "str | bytes | date | datetime | timedelta | <7 more items> | Interval[Any] | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]")  [assignment]
+ pandas/core/dtypes/cast.py:161: error: Incompatible types in assignment (expression has type "Timedelta | NaTType", variable has type "str | bytes | date | datetime | timedelta | <7 more items> | Interval[Any] | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]")  [assignment]
+ pandas/core/arrays/_ranges.py:147: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Timestamp | None")  [assignment]
+ pandas/core/arrays/datetimes.py:832: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "unit"  [union-attr]
+ pandas/core/arrays/datetimes.py:3177: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Timestamp | None")  [assignment]
+ pandas/core/arrays/datetimes.py:3177: note: Error code "assignment" not covered by "type: ignore[arg-type]" comment
- pandas/core/arrays/datetimes.py:3178: error: Non-overlapping identity check (left operand type: "Timestamp", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/arrays/datetimes.py:3178: error: Non-overlapping identity check (left operand type: "Timestamp | None", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/arrays/datetimes.py:3179: error: Item "None" of "Timestamp | None" has no attribute "as_unit"  [union-attr]
+ pandas/core/arrays/datetimes.py:3185: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Timestamp | None")  [assignment]
+ pandas/core/arrays/datetimes.py:3185: note: Error code "assignment" not covered by "type: ignore[arg-type]" comment
- pandas/core/arrays/datetimes.py:3186: error: Non-overlapping identity check (left operand type: "Timestamp", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/arrays/datetimes.py:3186: error: Non-overlapping identity check (left operand type: "Timestamp | None", right operand type: "NaTType")  [comparison-overlap]
+ pandas/core/arrays/datetimes.py:3187: error: Item "None" of "Timestamp | None" has no attribute "as_unit"  [union-attr]
+ pandas/core/arrays/period.py:385: error: Incompatible return value type (got "Period | NaTType", expected "Period")  [return-value]
+ pandas/core/arrays/period.py:1228: error: Argument 1 to "_add_timedelta_arraylike" of "PeriodArray" has incompatible type "ndarray[tuple[Any, ...], dtype[generic[date | int | timedelta | None]]]"; expected "TimedeltaArray | ndarray[tuple[Any, ...], dtype[timedelta64[timedelta | int | None]]]"  [arg-type]
+ pandas/core/arrays/period.py:1343: error: Argument 1 to "delta_to_tick" has incompatible type "Timedelta | NaTType"; expected "timedelta"  [arg-type]
+ pandas/core/window/rolling.py:2023: error: Incompatible types in assignment (expression has type "int | signedinteger[_64Bit]", variable has type "int | None")  [assignment]
+ pandas/core/tools/datetimes.py:633: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_julian_date"  [union-attr]
+ pandas/core/indexes/timedeltas.py:324: error: Argument 1 to "to_offset" has incompatible type "Timedelta | NaTType"; expected "timedelta | str | PeriodDtypeBase"  [arg-type]
+ pandas/core/indexes/timedeltas.py:324: note: Error code "arg-type" not covered by "type: ignore[operator]" comment
+ pandas/core/indexes/period.py:673: error: Incompatible return value type (got "Period | NaTType", expected "Period")  [return-value]
+ pandas/core/indexes/period.py:700: error: Item "NaTType" of "Period | NaTType" has no attribute "asfreq"  [union-attr]
+ pandas/core/indexes/datetimes.py:943: error: Unsupported left operand type for % ("NaTType")  [operator]
+ pandas/core/indexes/datetimes.py:943: note: Left operand is of type "Timedelta | NaTType"
+ pandas/core/indexes/datetimes.py:1120: error: Item "NaTType" of "Period | NaTType" has no attribute "start_time"  [union-attr]
+ pandas/core/indexes/datetimes.py:1123: error: Unsupported operand types for + ("NaTType" and "int")  [operator]
+ pandas/core/indexes/datetimes.py:1123: note: Left operand is of type "Period | NaTType"
+ pandas/core/indexes/datetimes.py:1148: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "datetime")  [assignment]
+ pandas/core/indexes/datetimes.py:1154: error: "datetime" has no attribute "tz_localize"  [attr-defined]
+ pandas/core/indexes/datetimes.py:1156: error: Incompatible return value type (got "tuple[datetime, Resolution]", expected "tuple[Timestamp, Resolution]")  [return-value]
+ pandas/core/indexes/datetimes.py:1735: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "unit"  [union-attr]
+ pandas/core/indexes/datetimes.py:1736: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "unit"  [union-attr]
+ pandas/core/indexes/datetimelike.py:1139: error: Argument 1 to "to_offset" has incompatible type "Timedelta | NaTType"; expected "timedelta | str | PeriodDtypeBase"  [arg-type]
+ pandas/core/resample.py:2524: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Literal['epoch', 'start', 'start_day', 'end', 'end_day'] | Timestamp")  [assignment]
+ pandas/core/resample.py:2648: error: Argument "offset" to "_get_timestamp_range_edges" has incompatible type "Timedelta | NaTType | None"; expected "Timedelta | None"  [arg-type]
+ pandas/core/resample.py:2840: error: Argument "offset" to "_get_period_range_edges" has incompatible type "Timedelta | NaTType | None"; expected "Timedelta | None"  [arg-type]
+ pandas/core/resample.py:2959: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Literal['epoch', 'start', 'start_day', 'end', 'end_day'] | Timestamp")  [assignment]
+ pandas/core/resample.py:2975: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Timestamp")  [assignment]
+ pandas/core/resample.py:2977: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Timestamp")  [assignment]
+ pandas/core/resample.py:2979: error: Incompatible types in assignment (expression has type "Timestamp | NaTType", variable has type "Timestamp")  [assignment]
+ pandas/core/resample.py:3091: error: No overload variant of "__mul__" of "signedinteger" matches argument type "Tick"  [operator]
+ pandas/core/resample.py:3091: note: Possible overload variants:
+ pandas/core/resample.py:3091: note:     def __mul__(self, int | signedinteger[_8Bit] | numpy.bool[builtins.bool] | signedinteger[_64Bit], /) -> signedinteger[_64Bit]
+ pandas/core/resample.py:3091: note:     def __mul__(self, float, /) -> float64
+ pandas/core/resample.py:3091: note:     def __mul__(self, complex, /) -> complex128
+ pandas/core/resample.py:3091: note:     def __mul__(self, signedinteger[Any], /) -> signedinteger[Any]
+ pandas/core/resample.py:3091: note:     def __mul__(self, integer[Any], /) -> Any
+ pandas/core/resample.py:3091: note: Left operand is of type "int | signedinteger[_64Bit]"
+ pandas/core/resample.py:3139: error: Incompatible return value type (got "tuple[Timestamp | NaTType, Timestamp | NaTType]", expected "tuple[Timestamp, Timestamp]")  [return-value]
+ pandas/io/excel/_odfreader.py:220: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "time"  [union-attr]
+ pandas/tseries/holiday.py:262: error: Incompatible types in assignment (expression has type "Timestamp | NaTType | None", variable has type "Timestamp | None")  [assignment]
+ pandas/tseries/holiday.py:264: error: Incompatible types in assignment (expression has type "Timestamp | NaTType | None", variable has type "Timestamp | None")  [assignment]
+ pandas/tests/tslibs/test_timedeltas.py:24: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:277: error: Unsupported operand types for - ("NaTType" and "Nano")  [operator]
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:277: note: Left operand is of type "Timestamp | NaTType"
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:278: error: Unsupported operand types for - ("NaTType" and "Nano")  [operator]
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:278: note: Left operand is of type "Timestamp | NaTType"
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:288: error: Unsupported operand types for - ("NaTType" and "Nano")  [operator]
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:288: note: Left operand is of type "Timestamp | NaTType"
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:289: error: Unsupported operand types for - ("NaTType" and "Nano")  [operator]
+ pandas/tests/tseries/offsets/test_custom_business_hour.py:289: note: Left operand is of type "Timestamp | NaTType"
+ pandas/tests/tseries/offsets/test_business_day.py:78: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/tseries/offsets/test_business_day.py:79: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/series/indexing/test_setitem.py:512: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/series/indexing/test_setitem.py:513: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/scalar/test_nat.py:557: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/scalar/test_nat.py:558: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/scalar/timedelta/test_timedelta.py:266: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "max"  [union-attr]
+ pandas/tests/scalar/interval/test_arithmetic.py:116: error: Value of type variable "_OrderableT" of "Interval" cannot be "Timestamp | NaTType"  [type-var]
+ pandas/tests/scalar/interval/test_arithmetic.py:119: error: Value of type variable "_OrderableT" of "Interval" cannot be "Timedelta | NaTType"  [type-var]
+ pandas/tests/indexes/test_engines.py:62: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/indexes/test_engines.py:63: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:72: error: List item 0 has incompatible type "list[Timestamp | NaTType]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:79: error: List item 1 has incompatible type "list[Timestamp | NaTType]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:86: error: List item 2 has incompatible type "list[Timedelta | NaTType]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:93: error: List item 3 has incompatible type "list[Period | Any]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:94: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:95: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:96: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:97: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:98: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:202: error: List item 0 has incompatible type "list[float | Timestamp | NaTType]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:212: error: List item 1 has incompatible type "list[float | Timestamp | NaTType]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:222: error: List item 2 has incompatible type "list[float | Timedelta | NaTType]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:232: error: List item 3 has incompatible type "list[float | Period | Any]"; expected "ndarray[tuple[Any, ...], dtype[Any]]"  [list-item]
+ pandas/tests/groupby/methods/test_rank.py:233: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:234: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:236: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:237: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/groupby/methods/test_rank.py:238: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/arrays/test_timedeltas.py:215: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/arrays/test_timedeltas.py:216: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/arrays/test_timedeltas.py:238: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/arrays/test_datetimes.py:669: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "to_period"  [union-attr]
+ pandas/tests/arrays/test_datetimelike.py:187: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "time"  [union-attr]
+ pandas/tests/arithmetic/conftest.py:70: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/arithmetic/conftest.py:72: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/arithmetic/conftest.py:87: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/arithmetic/conftest.py:106: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/arithmetic/conftest.py:131: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/plotting/_matplotlib/converter.py:153: error: Item "NaTType" of "Timestamp | NaTType" has no attribute "time"  [union-attr]
+ pandas/plotting/_matplotlib/converter.py:309: error: Item "NaTType" of "Period | NaTType" has no attribute "ordinal"  [union-attr]
+ pandas/conftest.py:1575: error: Incompatible return value type (got "Timestamp | NaTType", expected "Timestamp")  [return-value]
+ pandas/tests/arithmetic/test_timedelta64.py:87: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/arithmetic/test_period.py:42: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/arithmetic/test_period.py:59: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]
+ pandas/tests/arithmetic/test_numeric.py:227: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_timedelta64"  [union-attr]
+ pandas/tests/arithmetic/test_numeric.py:228: error: Item "NaTType" of "Timedelta | NaTType" has no attribute "to_pytimedelta"  [union-attr]

... (truncated 8 lines) ...

jax (https://github.com/google/jax)
- jax/experimental/pallas/ops/tpu/splash_attention/splash_attention_mask_info.py:729: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]]", variable has type "list[ndarray[tuple[Any, ...], dtype[Any]]] | None")  [assignment]
+ jax/experimental/pallas/ops/tpu/splash_attention/splash_attention_mask_info.py:729: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[bool[Any]]]", variable has type "list[ndarray[tuple[Any, ...], dtype[Any]]] | None")  [assignment]

starlette (https://github.com/encode/starlette)
+ starlette/testclient.py:143: error: Incompatible types in assignment (expression has type "tuple[MemoryObjectSendStream[MutableMapping[str, Any]], MemoryObjectReceiveStream[MutableMapping[str, Any]]]", variable has type "create_memory_object_stream[MutableMapping[str, Any]]")  [assignment]
+ starlette/testclient.py:145: error: Incompatible types in assignment (expression has type "tuple[MemoryObjectSendStream[MutableMapping[str, Any]], MemoryObjectReceiveStream[MutableMapping[str, Any]]]", variable has type "create_memory_object_stream[MutableMapping[str, Any]]")  [assignment]
+ starlette/testclient.py:692: error: Incompatible types in assignment (expression has type "tuple[MemoryObjectSendStream[MutableMapping[str, Any] | None], MemoryObjectReceiveStream[MutableMapping[str, Any] | None]]", variable has type "create_memory_object_stream[MutableMapping[str, Any] | None]")  [assignment]
+ starlette/testclient.py:694: error: Incompatible types in assignment (expression has type "tuple[MemoryObjectSendStream[MutableMapping[str, Any]], MemoryObjectReceiveStream[MutableMapping[str, Any]]]", variable has type "create_memory_object_stream[MutableMapping[str, Any]]")  [assignment]
+ starlette/middleware/base.py:189: error: Incompatible types in assignment (expression has type "tuple[MemoryObjectSendStream[MutableMapping[str, Any]], MemoryObjectReceiveStream[MutableMapping[str, Any]]]", variable has type "create_memory_object_stream[MutableMapping[str, Any]]")  [assignment]

This is awesome; thanks!

## Mypy 2.2 ### Support for Closed TypedDicts (PEP 728) Mypy now supports closed TypedDicts as specified in PEP 728. A closed TypedDict cannot have extra keys beyond those explicitly defined. This allows the type checker to determine that certain operations are safe when they otherwise wouldn't be due to the potential presence of unknown keys. You can use the `closed` keyword argument with `TypedDict`: ```python HasName = TypedDict("HasName", {"name": str}) HasOnlyName = TypedDict("HasOnlyName", {"name": str}, closed=True) Movie = TypedDict("Movie", {"name": str, "year": int}) movie: Movie = {"name": "Nimona", "year": 2023} has_name: HasName = movie # OK: HasName is open (default) has_only_name: HasOnlyName = movie # Error: HasOnlyName is closed and Movie has extra "year" key ``` Closed TypedDicts enable more precise type checking because the type checker knows exactly which keys are present. This is particularly useful when working with TypedDict unions or when you want to ensure that a TypedDict conforms to an exact shape. The `closed` keyword also enables safe type narrowing with `in` checks: ```python Book = TypedDict('Book', {'book': str}, closed=True) DVD = TypedDict('DVD', {'dvd': str}, closed=True) type Inventory = Book | DVD def print_type(inventory: Inventory) -> None: if "book" in inventory: # Type is narrowed to Book here - safe because DVD is closed print(inventory["book"]) else: # Type is narrowed to DVD here print(inventory["dvd"]) ``` The `closed` keyword is also supported in class-based syntax: ```python class HasOnlyName(TypedDict, closed=True): name: str ``` Note that closed TypedDicts are structural types, so a closed TypedDict is assignable to an open TypedDict with the same keys, but not vice versa. Contributed by Alice (PR [21382](python/mypy#21382)). ### Complete Support for Type Variable Defaults (PEP 696) Mypy now has complete support for type variable defaults as specified in PEP 696. This allows you to specify default values for type parameters in generic classes, functions, and type aliases. Traditional syntax (Python 3.11 and earlier): ```python T = TypeVar("T", default=int) # This means that if no type is specified T = int @DataClass class Box(Generic[T]): value: T | None = None reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] ``` New syntax (Python 3.12+): ```python class Box[T = int]: def __init__(self, value: T) -> None: self.value = value reveal_type(Box()) # type is Box[int] reveal_type(Box(value="Hello World!")) # type is Box[str] ``` Type variable defaults work with all forms of generics, including classes, functions, and type aliases. This release completes the implementation by fixing various edge cases involving recursive defaults, dependencies between type variables, and interactions with variadic generics. Contributed by Ivan Levkivskyi (PRs [21491](python/mypy#21491), [21526](python/mypy#21526), [21544](python/mypy#21544)). ### Respect Explicit Return Type of `__new__()` Mypy now respects explicitly annotated return types in `__new__()` methods. Previously, mypy would always assume that `__new__()` returns an instance of the current class, ignoring explicit annotations. With this change, if you explicitly annotate a return type that differs from the implicit type, mypy will use the explicit annotation: ```python class Factory: def __new__(cls) -> Product: return Product() reveal_type(Factory()) # type is Product, not Factory ``` Note that mypy still gives an error at the definition site if the explicit annotation is not a subtype of the current class, since this is technically not type-safe. For backwards compatibility, there are two exceptions: - If the return type is `Any`, mypy will still use the current class as the return type. - If the explicit return type comes from a superclass and is a supertype of the implicit return type, mypy will use the implicit (more specific) type: ```python class A: def __new__(cls) -> A: ... reveal_type(A()) # type is A class B: def __new__(cls) -> B: return cls() class C(B): ... reveal_type(C()) # type is C ``` This fixes several long-standing issues where explicit `__new__()` return types were ignored. Contributed by Ivan Levkivskyi (PR [21441](python/mypy#21441)). ### TypeForm Support No Longer Experimental Support for `TypeForm` is no longer experimental. `TypeForm` (introduced in Python 3.14) allows you to annotate parameters that accept type expressions, providing better type checking for functions that work with types as values. ```python from typing import TypeForm def make_list(tp: TypeForm[T]) -> list[T]: ... # Correctly typed as list[int] int_list = make_list(int) ``` `TypeForm` support was previously reverted from mypy 2.1 due to a performance regression, but this has now been mitigated. Contributed by Ivan Levkivskyi and Jelle Zijlstra (PRs [21262](python/mypy#21262), [21591](python/mypy#21591), [21459](python/mypy#21459)). ### Experimental WASM Wheel for Python 3.14 Mypy now ships an experimental WebAssembly (WASM) wheel for Python 3.14. This allows mypy to run in WASM environments such as Pyodide and browser-based Python implementations. The WASM wheel is considered experimental and may have limitations compared to native builds. Please report any issues you encounter when using mypy in WASM environments. Contributed by Ivan Levkivskyi (PR [21671](python/mypy#21671)). ### Mypyc Free-threading Improvements - Make function wrappers thread-safe on free-threaded builds (Jukka Lehtosalo, PR [21620](python/mypy#21620)) - Make list remove and index thread-safe on free-threaded builds (Jukka Lehtosalo, PR [21614](python/mypy#21614)) - Fix dict iteration memory safety on free-threaded builds (Jukka Lehtosalo, PR [21617](python/mypy#21617)) - Make some dict primitives thread-safe on free-threading builds (Jukka Lehtosalo, PR [21616](python/mypy#21616)) - Fix free-threading race condition in argument parsing (Jukka Lehtosalo, PR [21613](python/mypy#21613)) - Document free threading and other doc updates (Jukka Lehtosalo, PR [21494](python/mypy#21494)) ### `librt.strings` Updates - Add `librt.strings.toupper` and `librt.strings.tolower` codepoint primitives (Vaggelis Danias, PR [21553](python/mypy#21553)) - Add `librt.strings.isidentifier` codepoint primitive (Vaggelis Danias, PR [21522](python/mypy#21522)) - Add `librt.strings.isalpha` codepoint primitive (Vaggelis Danias, PR [21521](python/mypy#21521)) - Add `librt.strings.isalnum` codepoint primitive (Vaggelis Danias, PR [21509](python/mypy#21509)) - Add `librt.strings.isdigit` codepoint primitive (Vaggelis Danias, PR [21504](python/mypy#21504)) - Add `librt.strings.isspace` char primitive (Vaggelis Danias, PR [21462](python/mypy#21462)) ### Mypyc Improvements - Fix name lookup when class var and module var have the same name (Jukka Lehtosalo, PR [21594](python/mypy#21594)) - Report file and line number on uncaught exceptions (Jukka Lehtosalo, PR [21584](python/mypy#21584)) - Use `other` arg instead of `self` for RHS type (Ryan Heard, PR [21569](python/mypy#21569)) - Use `method_sig` to get the method signature (Ryan Heard, PR [21567](python/mypy#21567)) - Preserve inherited attribute defaults under `separate=True` (Jo, PR [21547](python/mypy#21547)) - Fix missing cross-group header deps in incremental builds (Jo, PR [21490](python/mypy#21490)) - Fix cross-group call to inherited `__mypyc_defaults_setup` (Jo, PR [21481](python/mypy#21481)) - Fix non-deterministic class struct layout under `separate=True` (Vaggelis Danias, PR [21530](python/mypy#21530)) - Specialize `s[i] == 'x'` to a codepoint int compare (Vaggelis Danias, PR [21579](python/mypy#21579)) - Fix reference leak in mypyc bytes concatenation (Colinxu2020, PR [21469](python/mypy#21469)) ### Fixes to Crashes - Fix crash on invalid recursive variadic alias (Ivan Levkivskyi, PR [21572](python/mypy#21572)) - Fix crashes on variadic unpacking in synthetic types (Ivan Levkivskyi, PR [21555](python/mypy#21555)) - Fix crash on unhandled meet variadic tuple vs instance (Ivan Levkivskyi, PR [21558](python/mypy#21558)) - Fix crash on deferred generic class nested in function (Ivan Levkivskyi, PR [21557](python/mypy#21557)) - Fix crash in new-style type alias with variadic unpack (Ivan Levkivskyi, PR [21551](python/mypy#21551)) - Fix various crashes on recursive type variable defaults (Ivan Levkivskyi, PR [21491](python/mypy#21491)) - Fix crash for empty `Annotated` type application (Rayan Salhab, PR [21503](python/mypy#21503)) - Fix crash on `Unpack` used without arguments in class bases (Sai Asish Y, PR [21470](python/mypy#21470)) ### Performance Improvements - Memoize the options snapshot (Kevin Kannammalil, PR [21354](python/mypy#21354)) - Don't include `not_ready_deps` tracking as relating to mypy internals (Kevin Kannammalil, PR [21389](python/mypy#21389)) - Speed up transitive dependency hash for singleton SCCs (Kevin Kannammalil, PR [21390](python/mypy#21390)) - Optimize typeform checks (Jelle Zijlstra, PR [21459](python/mypy#21459)) ### Improvements to the Native Parser - Support `--shadow-file` with `--native-parser` (Jukka Lehtosalo, PR [21623](python/mypy#21623)) - Add Python version checks to native parser (Kevin Kannammalil, PR [21539](python/mypy#21539)) - Allow nativeparse to parse source code directly (bzoracler, PR [21260](python/mypy#21260)) ### Other Notable Fixes and Improvements - Add function definition notes for `too many positional arguments` errors (Kevin Kannammalil, PR [21410](python/mypy#21410)) - Fix the exportjson tool (.ff cache to .json conversion) (Jukka Lehtosalo, PR [21628](python/mypy#21628)) - Support floats in JSON in fixed-format cache (Ivan Levkivskyi, PR [21603](python/mypy#21603)) - Update `TypedDictType.__init__` signature to preserve backward compat (Jukka Lehtosalo, PR [21590](python/mypy#21590)) - Fix constructor calls for union-bounded `TypeVar`s (Jingchen Ye, PR [21571](python/mypy#21571)) - Fix `TypedDict` indexing with literal keys in comprehensions (Jingchen Ye, PR [21556](python/mypy#21556)) - Correctly handle empty tuple index when unpacked (Ivan Levkivskyi, PR [21545](python/mypy#21545)) - Support protocol checks for self-types in tuple types (Ivan Levkivskyi, PR [21535](python/mypy#21535)) - Fix edge cases in variadic tuple subclasses (Ivan Levkivskyi, PR [21518](python/mypy#21518)) - Special-case constructor for tuple types (Ivan Levkivskyi, PR [21502](python/mypy#21502)) - Fix false positive "Expected TypedDict key to be string literal" for `Union[TypedDict, dict[K, V]]` (Zakir Jiwani, PR [21511](python/mypy#21511)) - Use explicit `Never` for type inference (Ivan Levkivskyi, PR [21497](python/mypy#21497)) - Narrow membership in statically known containers (Shantanu, PR [21461](python/mypy#21461)) - Improve negative narrowing for membership checks on tuples (Shantanu, PR [21456](python/mypy#21456)) - Analyze `TypedDict` decorators (Pranav Manglik, PR [21267](python/mypy#21267)) - Start testing Python 3.15 (Marc Mueller, PR [21439](python/mypy#21439)) - Improved handling of `NamedTuple`, `TypedDict`, `Enum`, and regular classes nested in functions (Ivan Levkivskyi, PR [21478](python/mypy#21478))

ilevkivskyi added 7 commits May 5, 2026 23:51

Preparatory refactor

b5ea90c

Add basic logic

a43ecc1

Add boilerplate

c930a95

Do not use ret_type in various special cases

8c3e017

Store precise type

b01ffea

Some more tweaks

b76c746

Update docstring

73bb465