Send early errors to workers by ilevkivskyi · Pull Request #20822 · python/mypy · GitHub

Ref #933

When parallel parsing is ready, we will not run load_graph() in each worker. So, we need to send early errors (mostly import errors) to the relevant workers. Couple notes here:

Now only_once messages are shown only once per worker. It is hard to fix, and I don't think we should really bother. It doesn't affect correctness, and may only slightly increase noise. Btw they were never truly only_once, when reading errors from cache we can get multiple occurrences of the same error/note.
While I am at it, I switch few more error to the standard file.py: error: some message format.

Btw @hauntsaninja mypy_primer output looks like we aren't really checking attrs now (but it is probably important). To clear, that error is a blocker.

I have some questions, mostly about things that I don't fully understand. Otherwise looks good.

This comment wasn't clear to me. I'm not sure which extra step this refers to (parse_file or the state.tree is None check?).

It refers to the whole if/else block. It can be avoided if we would not call load_graph() in the worker.

Why are we parsing the file here? Is this related to error processing?

Yes, we want to get the side effect of setting ignored/skipped lines.

Typo: sn

Should we only have the del statement behind the not temporary condition?

No, we need the tree until it is used by the caller who created temporary state to check for is_partial_stub_package, I will delete it there to make it more clear.

Actually, I decided to add a little cache for partial packages, otherwise it looks like a lot of potential overhead for a niche use case.

Unrelated: Add a docstring here?

Is this change related to error processing or the earlier changes to how we process suppressed modules?

This is just a minor perf optimization (we never check for imports of suppressed deps after graph is loaded). I was scrolling past this line and decided it is not worth a separate PR.

Is this because we now delete modules in the middle of a build, or is there another reason for maintaing this here?

This is mostly for convenience, to avoid passing graph around just for the purpose of mapping id to xpath. In most cases we identify modules by id, but in error-tracking-related code it is always xpath.

It looks like we might calculate this on demand using graph, and it wouldn't be much slower than a set lookup. Why are we storing this separately here?

The receiver will not be able to do this after we stop calling load_graph() there. This is because we don't serialize State.meta (it would double the size). The worker only needs to know whether the state is from cache or was already parsed by coordinator. I will try to restructure some code in other places to make this more obvious.

…rers

Btw @hauntsaninja mypy_primer output looks like we aren't really checking attrs now (but it is probably important). To clear, that error is a blocker.

hauntsaninja/mypy_primer#241 should fix this.

Diff from mypy_primer, showing the effect of this PR on open source code:

attrs (https://github.com/python-attrs/attrs)
- mypy: can't read file 'tests/typing_example.py': No such file or directory
+ mypy: error: cannot read file 'tests/typing_example.py': No such file or directory

Send early errors to workers

1dc730a

This comment has been minimized.

Sign in to view

Tweak

83f283e

ilevkivskyi marked this pull request as ready for review February 16, 2026 13:45

This comment has been minimized.

Sign in to view

JukkaL reviewed Feb 16, 2026

View reviewed changes

ilevkivskyi added 4 commits February 16, 2026 18:47

Address CR

f5efcf0

Fix typo

89f12ec

Merge remote-tracking branch 'upstream/master' into send-errors-to-wo…

0278aae

…rers

Fix merge

a9ecc04

JukkaL approved these changes Feb 17, 2026

View reviewed changes

ilevkivskyi merged commit 70de10e into python:master Feb 17, 2026
25 checks passed

ilevkivskyi deleted the send-errors-to-worers branch February 17, 2026 10:42

hauntsaninja mentioned this pull request Feb 24, 2026

dmypy not reporting an issue that mypy finds #15677

Open

                       try:
+                          for id in scc.mod_ids:
+                              state = graph[id]
+                              # Extra step below is needed only because we are using local graph.

+                              # Extra step below is needed only because we are using local graph.
+                              # TODO: clone options when switching to coordinator graph.
+                              if state.tree is None:
+                                  state.parse_file()

                   source_set = BuildSourceSet(sources)
                   cached_read = fscache.read
                   errors = Errors(options, read_source=lambda path: read_py_file(path, cached_read))
+                  # Record import errors so that they sn be replayed by the workers.

                               # We don't need parsed trees in coordinator process, we parse only to
                               # compute dependencies.
-                              state.tree = None
+                              if not temporary:

                       # Cache for transitive dependency check (expensive).
                       self.transitive_deps_cache: dict[tuple[int, int], bool] = {}
+                      # Resolved paths for each module in build.
+                      self.path_by_id: dict[str, str] = {}

		util.hard_exit(0)


		def serve(server: IPCServer, ctx: ServerContext) -> None:

                   for id in graph:
-                      manager.import_map[id] = set(graph[id].dependencies + graph[id].suppressed)
+                      manager.import_map[id] = graph[id].dependencies_set

+                  def __init__(self, *, graph: Graph, missing_modules: set[str]) -> None:
                       self.graph = graph
+                      self.missing_modules = missing_modules
+                      self.from_cache = {mod_id for mod_id in graph if graph[mod_id].meta}

Send early errors to workers by ilevkivskyi · Pull Request #20822 · python/mypy · GitHub

Repository navigation

Send early errors to workers - #20822

Send early errors to workers#20822ilevkivskyi merged 6 commits intopython:masterpython/mypy:masterfrom ilevkivskyi:send-errors-to-worersilevkivskyi/mypy:send-errors-to-worersCopy head branch name to clipboard

Conversation

ilevkivskyi commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JukkaL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi commented Feb 16, 2026

Uh oh!

github-actions Bot commented Feb 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Send early errors to workers#20822
ilevkivskyi merged 6 commits into
python:masterfrom
ilevkivskyi:send-errors-to-worers

ilevkivskyi commented Feb 16, 2026 •
edited
Loading

ilevkivskyi commented Feb 16, 2026 •
edited
Loading