gh-155695: Remove resolved names from sys.lazy_modules more consistently by brittanyrey · Pull Request #157714 · python/cpython · GitHub

summary

Improve sys.lazy_modules to address the following issues:

A lazy import of a module that has already been loaded does not get added to sys.lazy_modules .
The "pkg.attr" entry for lazy from pkg import attr is cleaned up after reification.

perf

End-to-end (hyperfine, --warmup 50, 1000 runs; 400 for reify-only)

scenario	base	patched	control	patched/base	control/base
bare startup (~0 lazy stmts)	12.1 ± 0.6 ms	12.0 ± 1.0 ms	12.1 ± 0.7 ms	0.99×	1.00×
synthetic: 243k lazy stmts	43.9 ± 3.0 ms	46.4 ± 2.9 ms	43.7 ± 2.8 ms	1.06×	1.00×
synthetic: 162k reifications	223.7 ± 11.8 ms	228.1 ± 10.7 ms	225.1 ± 11.9 ms	1.02×	1.01×
realistic -X lazy_imports=all app	68.0 ± 3.2 ms	68.3 ± 3.1 ms	67.4 ± 2.4 ms	1.00×	0.99×

Issue: Names that have been resolved are not consistently removed from sys.lazy_modules #155695

Found one tracking regression when a module fails during initialization, see the inline comment.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@pablogsal Addressed comments.

cc @encukou

Thanks! The initializing-module case is covered now. I left one follow-up comment about the new __spec__ lookup.

Names only left sys.lazy_modules through _imp._set_lazy_attributes(), which the import machinery calls from _find_and_load_unlocked(). Two cases never reached it, so their names were recorded and then kept forever: - A lazy import of a module already in sys.modules. _find_and_load() returns early, so nothing ever discards the name. Do not record it in the first place. - The "pkg.attr" entry for `lazy from pkg import attr`. The import machinery only discards module names, and attr is often not a module. Discard it when the lazy object is reified, where the name is already known and has been resolved either way. Submodules that are not yet loaded are still tracked: loading a package does not load its submodules, so those imports can still fire. Names whose reification failed also stay tracked, since the import can still happen.

A module is in sys.modules while its body runs, so lazy_modules_add() counted it as loaded and skipped recording the name. If the body then raises, the module is removed from sys.modules again and the lazy import is left pending under no name at all. Check __spec__._initializing so such a module does not count as loaded. A name added while a module initializes is still discarded once the import completes, since _set_lazy_attributes() runs after the body.

🤖 New build scheduled with the buildbot fleet by @hugovk for commit ad36b56 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F157714%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit ad36b56 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F157714%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

Added a rebase @encukou
I think the PR can be slimmed down.

Thank you! You got roughly to the same place I did, but faster as I keep double-checking.
I'll look for places to slimming down.
One thing that bothers me is using the human-friendly display name (with a PyUnicode_FromFormat("%U...", m->lz_from); case) when discarding, but addressing that won't slim things down.

Enjoy your PTO! Should I push into this PR, or do you want to continue working on it?

LGTM. I left some suggestions at brittanyrey#1; feel free to merge or cherry-pick if those look good.

The PR can be slimmed down by removing the _PyImport_DiscardLazyModule indirection, but keeping all uses of LAZY_MODULES in import.c has its own advantages.

🤖 New build scheduled with the buildbot fleet by @hugovk for commit 95198ab 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F157714%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

If #158521 is merged first, this will change to lazy_import_path since only the root will be resolved here:

Suggested change

      
                    PyObject *name = lazy_import_name(lz);
          
                    PyObject *name = lazy_import_path(lz);

…_from_lazy_on_parent

Buildbot failures are pre-existing/unrelated.

I've pushed my suggestions here, except human-friendly display name which will be obsolete if #158521 is merged.

Thank you @encukou!

Thanks @brittanyrey for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

GH-158607 is a backport of this pull request to the 3.15 branch.

…nsistently (GH-157714) (#158607) gh-155695: Remove resolved names from sys.lazy_modules more consistently (GH-157714) (cherry picked from commit be87a85) Co-authored-by: Brittany Reynoso <breynoso@meta.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>

…ore consistently (pythonGH-157714) (python#158607) pythongh-155695: Remove resolved names from sys.lazy_modules more consistently (pythonGH-157714) (cherry picked from commit be87a85) Co-authored-by: Brittany Reynoso <breynoso@meta.com> Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com> Co-authored-by: Petr Viktorin <encukou@gmail.com>

brittanyrey requested review from DinoV, FFY00, Yhg1s, brettcannon, ericsnowcurrently, kumaraditya303, ncoghlan, pablogsal and warsaw as code owners September 17, 2026 22:22

bedevere-app Bot mentioned this pull request Sep 17, 2026

Names that have been resolved are not consistently removed from sys.lazy_modules #155695

Closed

bedevere-app Bot added the awaiting review label Sep 17, 2026

pablogsal requested changes Sep 18, 2026

View reviewed changes

Comment thread Python/import.c Outdated

bedevere-app Bot added awaiting changes and removed awaiting review labels Sep 18, 2026

brittanyrey requested a review from pablogsal September 18, 2026 17:42

brettcannon removed their request for review September 21, 2026 21:03

pablogsal reviewed Sep 25, 2026

View reviewed changes

Comment thread Python/import.c Outdated

brittanyrey added 4 commits September 26, 2026 22:14

Empty commit to re-run CI

8ef4ae0

Read __spec__ from the module dict when checking for initialization

0456068

pablogsal added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Sep 26, 2026

pablogsal force-pushed the b-lazy-modules-stale-entries branch from 374d85f to 44ebbaa Compare September 26, 2026 21:39

pablogsal approved these changes Sep 26, 2026

View reviewed changes

pablogsal enabled auto-merge (squash) September 26, 2026 21:39

bedevere-app Bot removed the awaiting changes label Sep 26, 2026

github-project-automation Bot added this to Release and Deferred blockers 🚫 Sep 29, 2026

github-project-automation Bot moved this to Todo in Release and Deferred blockers 🚫 Sep 29, 2026

encukou mentioned this pull request Sep 30, 2026

gh-142349: Simplify lazy import resolution #158282

Merged

hugovk added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 30, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Sep 30, 2026

pablogsal added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 1, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 1, 2026

brittanyrey changed the title gh-155695: Remove resolved names from sys.lazy_modules consistently gh-155695: Remove resolved names from sys.lazy_modules more consistently Oct 1, 2026

Rebase.

95198ab

encukou approved these changes Oct 2, 2026

View reviewed changes

hugovk added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 2, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 2, 2026

encukou reviewed Oct 2, 2026

View reviewed changes

encukou added 2 commits October 2, 2026 13:46

Reuse return value from PySet_Discard

84a8e95

Share already-imported module between lazy_modules_add & all register…

45d23cd

…_from_lazy_on_parent

hugovk merged commit be87a85 into python:main Oct 2, 2026
54 checks passed

bedevere-app Bot removed the awaiting merge label Oct 2, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Oct 2, 2026

encukou mentioned this pull request Oct 2, 2026

gh-158140: Resolve lazily imported sibling submodules independently #158521

Merged

                   }
                   assert(obj == NULL || !PyLazyImport_CheckExact(obj));
+                  if (obj != NULL) {
+                      PyObject *name = lazy_import_name(lz);

	PyObject *name = lazy_import_name(lz);
	PyObject *name = lazy_import_path(lz);

gh-155695: Remove resolved names from sys.lazy_modules more consistently by brittanyrey · Pull Request #157714 · python/cpython · GitHub

Repository navigation

gh-155695: Remove resolved names from sys.lazy_modules more consistently - #157714

gh-155695: Remove resolved names from sys.lazy_modules more consistently#157714hugovk merged 8 commits intopython:mainpython/cpython:mainfrom brittanyrey:b-lazy-modules-stale-entriesbrittanyrey/cpython:b-lazy-modules-stale-entriesCopy head branch name to clipboard

Conversation

brittanyrey commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

summary

perf

Uh oh!

pablogsal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bedevere-app Bot commented Sep 18, 2026

Uh oh!

brittanyrey commented Sep 21, 2026

Uh oh!

brittanyrey commented Sep 24, 2026

Uh oh!

Uh oh!

pablogsal commented Sep 25, 2026

Uh oh!

bedevere-bot commented Sep 30, 2026

Uh oh!

bedevere-bot commented Oct 1, 2026

Uh oh!

brittanyrey commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented Oct 1, 2026

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Oct 2, 2026

Uh oh!

encukou Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou commented Oct 2, 2026

Uh oh!

brittanyrey commented Oct 2, 2026

Uh oh!

Uh oh!

miss-islington-app Bot commented Oct 2, 2026

Uh oh!

bedevere-app Bot commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gh-155695: Remove resolved names from sys.lazy_modules more consistently#157714
hugovk merged 8 commits into
python:mainfrom
brittanyrey:b-lazy-modules-stale-entries

brittanyrey commented Sep 17, 2026 •
edited
Loading

pablogsal left a comment •
edited
Loading

brittanyrey commented Oct 1, 2026 •
edited
Loading

encukou Oct 2, 2026 •
edited
Loading