| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
|
@JukkaL the test failures on Windows look real. |
Sorry, something went wrong.
|
Yeah, I'm investigating the Windows failures. |
Sorry, something went wrong.
| PyObject *file = PyObject_GetAttrString(modobj, "__file__"); | ||
| if (file != NULL) { | ||
| // __file__ already set, nothing to do. | ||
| Py_DECREF(file); | ||
| return 0; | ||
| } | ||
| if (!PyErr_ExceptionMatches(PyExc_AttributeError)) { | ||
| return -1; | ||
| } | ||
| PyErr_Clear(); |
There was a problem hiding this comment.
could use PyObject_GetOptionalAttrString to not have to deal with the exception here and in other new functions.
Sorry, something went wrong.
| def func() -> int: | ||
| return 42 | ||
| [file driver.py] | ||
| import native | ||
| assert native.f() == 42 |
There was a problem hiding this comment.
test if _file is as expected?
Sorry, something went wrong.
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Sorry, something went wrong.
Fixes #21808 Native-to-native imports within the same compilation group were sped up in #21101 to improve mypy cold-start performance on macOS, since regular imports going through the shim significantly slowed it down. With this change a native-to-native import does not go through the regular Python import machinery. This means that native-to-native imports are not synchronized by the module lock used by the regular Python import process, which guarantees that only one thread can import a given module at a time. Missing this synchronization means that a thread importing a native module might see it in a partially-initialized state, leading to errors. More details about this are available in comments to the linked issue. To add synchronization, we use the module lock from the Python import library. Writing a custom native lock might be possible, but it would be difficult because of edge cases such as circular imports, modules waiting at top-level on threads importing other native modules, or interpreted and native modules concurrently importing a native module. Using the Python module lock does not regress performance, as the mypy self-check comparison does not produce a significant difference (table below). mypy cold-start time on macOS is also pretty much the same before and after this change. mypy self-check comparison: Python | Baseline | PR | Difference | - | - | - | - 3.13, GIL | 2.124s ± 0.020 | 2.118s ± 0.024 | -0.3% 3.14t, no GIL | 2.756s ± 0.039 | 2.768s ± 0.042 | +0.4% The module lock API is obtained when initializing a group shared library by importing from `importlib._bootstrap`. When importing a native module, this API is used to obtain the lock associated with the module. This is the same lock that would be used when importing the module in an interpreted module. To ensure compatibility in case the native module is imported both through Python and natively at the same time, the native import function now sets `module.__spec__._initializing` to `True` when the module is being imported. There's a fast path in case the native module is imported multiple times. Each native module defines a boolean flag that is set to `true` after it has been imported, so future imports can immediately return and not take the module lock after reading the flag. The flag is set in a wrapper over the module exec function so it will be set even if the first import is not native-to-native.
| Back | FazBrowse Home | New Git URL |
When compiling multiple modules, mypyc generally creates one big shared library with all the code, and also tiny shim shared libraries for each compiled module so that Python import machinery can find the modules. This is inefficient at least on macOS, since each shared library that is loaded into the process seems to have a non-trivial cost, including each shim. On the first run, this cost is much higher, and the first mypy run after pip install can take 30s or more on macOS.
This PR addresses the slow imports on macOS by adding a custom implementation of native-to-native imports within the same compilation group that avoids using the shim. We directly construct the module object, populate sys.modules, and set an attribute in the parent package, without using Python import machinery.
This speeds up a minimal mypy run (mypy -c 'import os') on macOS by up to 10x (first cold run after installation), but even small warm runs are significantly faster. The measurements were all over the place, but at least in one measurement the minimal warm run was over 1.5x faster with these changes. Impact on Linux should be small (an earlier version of this PR was slightly faster on Linux, but didn't measure the current one). I haven't measured the impact on Windows.
Some notes about the implementation:
I used Claude Code and Codex to implement much of the code in small increments (based on a manually written core implementation). I also iterated on the code quite significantly after the basic implementation was done.