Typecheck unreachable code by A5rocks · Pull Request #18707 · python/mypy · GitHub

Mypy doesn't typecheck unreachable code and this is surprising to many. This is an attempt at fixing that. It's a draft until:

I list out the issues fixed (done)
I fix the bugs raised by the test cases (done)
I fix the bugs raised by mypy primer (done)
I get feedback that this is fine (skipped for now)
I put this behind a flag (we can bundle this in --strict maybe and hopefully enable it by default in 2.0) (skipped for now, I will add after reviews)

Problem areas based on test cases:

currently we use None for unreachable type maps but we want to know what is unreachable to mark it as Never (fixed what was relevant in tests. narrowing is not guaranteed by mypy and we can fix things in the future -- I'd rather have context for any fixes)
- it doesn't seem too bad to mark unreachability based on if a variable has a Never type
analyzing a function multiple times for type var values (fixed)
- I think I can disable this new behavior based on if we're in a function like that?
reporting unreachability multiple times (block in a block) (fixed)
- blocks could store unreachability report status
exports could change (see check-incremental.test's changes)
- I think this is fine?
Never doesn't support operations (e.g. <Never>.f(), <Never> > 5) (fixed)
- it should return Never for these, I think?

Also maybe reveal_type(<Never>) should be non-trivial. (fixed)

Generally I tried enabling --warn-unreachable in a test case if it looked like people were relying on un-checked unreachable code as a test. Also obviously many of the test case changes are wrong I just did not have the energy to fix all the bugs (and honestly maybe should think through the two things above -- the first especially was a source of issues).

I also have no clue what mypyc does so the test change there could be awful.

Except things important enough we marked them before, which include: - `if not typing.TYPE_CHECKING` - platform checks that do not match

…bility

Thanks for the PR! Not a full review yet, just a few comments. I probably won't have time to review this in the next few days at least, so if anybody else has bandwidth, please have a look.

Also there's now a merge conflict.

…heck-unreachable

Diff from mypy_primer, showing the effect of this PR on open source code:

beartype (https://github.com/beartype/beartype)
+ beartype/_util/cache/utilcachecall.py:575: error: Incompatible return value type (got "Callable[..., Any]", expected "CallableT")  [return-value]

manticore (https://github.com/trailofbits/manticore)
+ manticore/wasm/types.py:56: error: Unused "type: ignore" comment  [unused-ignore]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/server/models/deployments.py:744: error: "Never" has no attribute "model_dump"  [attr-defined]

pandas (https://github.com/pandas-dev/pandas)
- pandas/io/formats/style.py:1903: error: "Never" has no attribute "shape"  [attr-defined]
- pandas/io/formats/style.py:1906: error: "Never" has no attribute "shape"  [attr-defined]

spark (https://github.com/apache/spark)
+ python/pyspark/core/context.py:818: error: Unused "type: ignore" comment  [unused-ignore]
+ python/pyspark/core/context.py:819: error: Unused "type: ignore" comment  [unused-ignore]

xarray (https://github.com/pydata/xarray)
+ xarray/core/variable.py:900: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/core/variable.py:902: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/core/variable.py:2786: error: Unused "type: ignore" comment  [unused-ignore]
+ xarray/core/variable.py:2788: error: Unused "type: ignore" comment  [unused-ignore]
- xarray/computation/apply_ufunc.py: note: In function "apply_variable_ufunc":
- xarray/computation/apply_ufunc.py:845: error: "Never" has no attribute "ndim"  [attr-defined]
- xarray/computation/apply_ufunc.py:848: error: "Never" has no attribute "ndim"  [attr-defined]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/runtime.py:903: error: Statement is unreachable  [unreachable]

websockets (https://github.com/aaugustin/websockets)
+ src/websockets/legacy/protocol.py:1077: error: Argument 1 to "join" of "str" has incompatible type "list[str | bytes]"; expected "Iterable[str]"  [arg-type]
+ src/websockets/legacy/protocol.py:1077: error: Argument 1 to "join" of "bytes" has incompatible type "list[str | bytes]"; expected "Iterable[Buffer]"  [arg-type]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/parsing.py:588: error: "Never" has no attribute "__iter__" (not iterable)  [attr-defined]
- tanjun/parsing.py:1632: error: Statement is unreachable  [unreachable]
- tanjun/parsing.py:1642: error: Statement is unreachable  [unreachable]
- tanjun/dependencies/limiters.py:851: error: "Never" has no attribute "unlock"  [attr-defined]
+ tanjun/dependencies/limiters.py:852: error: Statement is unreachable  [unreachable]
- tanjun/commands/slash.py:820: error: Statement is unreachable  [unreachable]
- tanjun/commands/slash.py:1787: error: Statement is unreachable  [unreachable]
- tanjun/clients.py:2294: error: "Never" has no attribute "__iter__" (not iterable)  [attr-defined]
+ tanjun/clients.py:2295: error: Statement is unreachable  [unreachable]

ibis (https://github.com/ibis-project/ibis)
- ibis/common/bases.py:231: error: Need type annotation for "field"  [var-annotated]

jax (https://github.com/google/jax)
+ jax/_src/interpreters/partial_eval.py:97: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/interpreters/partial_eval.py:100: error: No overload variant of "get" of "dict" matches argument types "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx", "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx"  [call-overload]
+ jax/_src/interpreters/partial_eval.py:100: note: Possible overload variants:
+ jax/_src/interpreters/partial_eval.py:100: note:     def get(self, Name, None = ..., /) -> DBIdx | None
+ jax/_src/interpreters/partial_eval.py:100: note:     def get(self, Name, DBIdx, /) -> DBIdx
+ jax/_src/interpreters/partial_eval.py:100: note:     def [_T] get(self, Name, _T, /) -> DBIdx | _T
+ jax/_src/interpreters/batching.py:235: error: Unused "type: ignore" comment  [unused-ignore]
+ jax/_src/interpreters/batching.py:237: error: No overload variant of "get" of "dict" matches argument types "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx", "int | DArray | Tracer | Var | DBIdx | InDBIdx | OutDBIdx"  [call-overload]
+ jax/_src/interpreters/batching.py:237: note: Possible overload variants:
+ jax/_src/interpreters/batching.py:237: note:     def get(self, Name, None = ..., /) -> DBIdx | None
+ jax/_src/interpreters/batching.py:237: note:     def get(self, Name, DBIdx, /) -> DBIdx
+ jax/_src/interpreters/batching.py:237: note:     def [_T] get(self, Name, _T, /) -> DBIdx | _T
+ jax/_src/interpreters/batching.py:261: error: Unused "type: ignore" comment  [unused-ignore]

core (https://github.com/home-assistant/core)
+ homeassistant/components/rfxtrx/__init__.py:455: error: Statement is unreachable  [unreachable]

graphql-core (https://github.com/graphql-python/graphql-core)
+ tests/execution/test_subscribe.py:212: error: Unused "type: ignore" comment  [unused-ignore]

pytest (https://github.com/pytest-dev/pytest)
+ testing/code/test_source.py:238: error: Unused "type: ignore" comment  [unused-ignore]

In the future, it would be nice to have an option to continue to check unreachable code. See #18707 from A5rocks. This PR lays some semantics preserving groundwork for that. This is effectively a subset of #18707 , but I go just a little bit further and change the type of TypeMap to exclude None Btw A5rocks if you are willing to rebase your PR, that would be great. I think `--check-unreachable` will be a very useful feature. Co-authored-by: A5rocks

Wow those conflicts are nasty. I should I probably just remake the PR rather than try to resolve them...

Yeah, this is a remake situation 😅
Several of the test changes could also be in their own PR.

I'm just going to close this! I kept forgetting to remake it. (I'll put the very basic changes into a new PR and... maybe I'll figure out how to make a stacked PR? Who knows...)

Ok, check out #21834!

Replaces #18707, references #21835. This is very unbaked, but I figure it's best to get the smallest bit in. I'll see if I can figure out stacked PRs to add extra stuff to this (namely decreasing the number of false errors in unreachable code!), but this is probably super easy to review and a good start!

A5rocks added 10 commits February 19, 2025 15:21

Start type checking unreachable code

be1071e

Except things important enough we marked them before, which include: - `if not typing.TYPE_CHECKING` - platform checks that do not match

Pass self-check

d996918

Hacky fix to narrow unreachable literals

07de5bf

20% of tests

bac268b

25% of tests

2da4b7c

30% of tests

ba9dab2

40% of tests

7c39efa

50% of tests

18642d0

60% of tests

879b274

Pass all tests

c62093e