gh-69605: Check for already imported modules in PyREPL module completion by loic-simon · Pull Request #139461 · python/cpython · GitHub

This PR handles an edge-case when suggesting module imports in PyREPL (see this comment):

If a module is already imported, the import machinery will only get submodules from it, even if a module of the same name is higher in the path. This can happen if eg. sys.path has been updated since the module was originaly imported.

The easiest way to reproduce this issue (and how I stumbled on it) is with stdlib modules imported during interpreter startup (before site customisation, so ignoring potential shadowing):

% mkdir collections
% touch collections/__init__.py collections/foo.py
% python
Python 3.14.0rc2 [...]
>>> from collections import <TAB>
>>> from collections import foo
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    from collections import foo
ImportError: cannot import name 'foo' from 'collections' (/Users/loic/cpython/Lib/collections/__init__.py)
>>>

This PR add a special case in _pyrepl.ModuleCompleter._find_modules that

check if the module to complete imports from is already imported
if yes, search for imports from the imported module location only, instead of using the global cache

It was a little tricky to get right, I'm not sure about everything 😅 but it pass all tests I could think off!

cc @tomasr8

Issue: Readline completion of module names in import statements #69605

Ok, one of the new test fails on Windows x64/32 (but pass on arm)

I just setup and built Python on a Win 11 x64 to try to reproduce, the test pass 😵‍💫 (and the fix works)
If anyone has an idea of what may interfere? (Python 3.15.0a0 (main, Sep 30 2025, 23:36:53) [MSC v.1944 64 bit (AMD64)] on win32)

All commit authors signed the Contributor License Agreement.

…dule

…mported-modules

I'm going to convert it into a draft until you deem this PR ready. That will also reduce the probability of someone clicking on the PR to review it (like me...) while it's not yet ready for.

I'm going to convert it into a draft until you deem this PR ready. That will also reduce the probability of someone clicking on the PR to review it (like me...) while it's not yet ready for.

Thanks, I just fixed the failing test, this should be ready for review!

Sorry again for the noise, I was dealing with what turned out to be a importer cache issue, which I couldn't reproduce outside of the CI buildbots 😓

[edit] I just noticed I can convert to draft / mark ready myself, will do it next times!

…ules' of https://github.com/loic-simon/cpython into pyrepl-module-completion-check-for-already-imported-modules

…ready-imported-modules

…mported-modules

I'm not sure if this is correct, it's going to find submodules but we actually want this to be the top-level modules

I believe it's OK, because os.path.dirname make us search in the folder containing the module origin, not on the origin itself (that's why we need to do it twice for a package):

>>> import os, pkgutil, typing, concurrent
>>>
>>> typing.__spec__.origin  # single-file module
'<venv>/lib/python3.14/typing.py'
>>> concurrent.__spec__.origin  # package
'<venv>/lib/python3.14/concurrent/__init__.py'
>>>
>>> loc = os.path.dirname(typing.__spec__.origin)  # or do it twice for concurrent
>>> [mod for mod in pkgutil.iter_modules([loc]) if mod.name in ("typing", "concurrent")]
[ModuleInfo(module_finder=FileFinder('<venv>/lib/python3.14'), name='concurrent', ispkg=True), 
ModuleInfo(module_finder=FileFinder('<venv>/lib/python3.14'), name='typing', ispkg=False)]

While refactoring this into a separate function I ended up rewriting the whole thing, it should be more explicit now (and it checks module.__package__ instead of the os.path.basename(imported_path) == "__init__.py" hack)

…mported-modules

I also realized we will need the same logic for my other PR about attributes completion, so I factored the logic into a private method, I suggest we merge this one first so I can rebase merge main into the other and call it!

EDIT: ok that's not true, since attributes completion checks directly the module object, I forgot 😅 but the refactor is a good thing anyway I think

I believe it's OK, because os.path.dirname make us search in the folder containing the module origin, not on the origin itself (that's why we need to do it twice for a package):

>>> import os, pkgutil, typing, concurrent
>>>
>>> typing.__spec__.origin  # single-file module
'<venv>/lib/python3.14/typing.py'
>>> concurrent.__spec__.origin  # package
'<venv>/lib/python3.14/concurrent/__init__.py'
>>>
>>> loc = os.path.dirname(typing.__spec__.origin)  # or do it twice for concurrent
>>> [mod for mod in pkgutil.iter_modules([loc]) if mod.name in ("typing", "concurrent")]
[ModuleInfo(module_finder=FileFinder('<venv>/lib/python3.14'), name='concurrent', ispkg=True), 
ModuleInfo(module_finder=FileFinder('<venv>/lib/python3.14'), name='typing', ispkg=False)]

While refactoring this into a separate function I ended up rewriting the whole thing, it should be more explicit now (and it checks module.__package__ instead of the os.path.basename(imported_path) == "__init__.py" hack)

This is not always going to be a real path right? e.g. _frozen_importlib.. in any case it does not seem that useful to create the full path when we strip it away with os.path.dirname right after.

Perhaps we can have a smaller diff if instead of reverse-engineering the location, we simply look for the right ModuleInfo in the global cache. After all if the module has already been imported, it should be visible to pkgutil.iter_modules. Something like this:

modules: Iterable[pkgutil.ModuleInfo] = self.global_cache
imported_module = sys.modules.get(path.split('.')[0])
if imported_module is not None:
    # Filter modules to those whose name and spec matches the imported module
    modules = ...

We can compare the module name and origin (maybe with something like mod_info.module_finder.find_spec(mod_info.name)) to make sure we have the right one.

We can compare the module name and origin (maybe with something like mod_info.module_finder.find_spec(mod_info.name)) to make sure we have the right one.

Clever! That will however provide no completions if the imported module exists, but is not the one returned by pkgutil.iter_modules anymore. But it's a quite degenerate case, and already half-supported because of the global_cache not updating... So I think having no suggestions in this case is fine, and the diff is much smaller indeed! I was probably a bit over zealous 😅

(frozen packages are also left out (__phello__.__spec__.origin == 'frozen' vs mod_info.find_spec('__phello__').origin == '<venv>/Lib/__phello__/__init__.py'), but that's an even more niche issue -- I'm not ever sure there is frozen packages except __phello__!)

That will however provide no completions if the imported module exists, but is not the one returned by pkgutil.iter_modules anymore

I considered it, but I think that's an edge case not worth trying to work around..

frozen packages are also left out

damn, that's unfortunate, we might need some special handling for frozen modules then. Note that there's also another special origin for built-in modules:

>>> import builtins
>>> builtins.__spec__.origin
'built-in'

I'm not ever sure there is frozen packages except phello

There are quite a bit. Here's an example of those that are imported at startup:

>>> frozen = {name for name, m in sys.modules.items() if m.__spec__ and m.__spec__.origin == 'frozen'}
>>> frozen
{'posixpath', 'site', 'importlib._bootstrap_external', 'importlib._bootstrap', 'io', 'zipimport', '_sitebuiltins', '__phello__', '_frozen_importlib', 'collections.abc', '_collections_abc', 'os', 'stat', 'genericpath', 'abc', 'os.path', '_frozen_importlib_external', 'codecs', 'importlib.util', 'importlib.machinery'}

This is good, confirmed on macOS and Windows it's doing the right thing!

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 LTO + PGO 3.x (tier-3) has failed when building commit bfac54d.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1578/builds/3864) and take a look at the build logs.
Check if the failure is related to this commit (bfac54d) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1578/builds/3864

Failed tests:

test_pyrepl

Failed subtests:

test_already_imported_module_without_origin_or_spec - test.test_pyrepl.test_pyrepl.TestPyReplModuleCompleter.test_already_imported_module_without_origin_or_spec

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto-pgo/build/Lib/test/test_pyrepl/test_pyrepl.py", line 1195, in test_already_imported_module_without_origin_or_spec
    module = importlib.import_module(mod)
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto-pgo/build/Lib/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1303, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1276, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1240, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'no_spec'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto-pgo/build/Lib/test/test_pyrepl/test_pyrepl.py", line 1195, in test_already_imported_module_without_origin_or_spec
    module = importlib.import_module(mod)
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto-pgo/build/Lib/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1303, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1276, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1240, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'not_has_location'

The buildbot failure is related. I'm on it.

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 LTO 3.x (tier-3) has failed when building commit bfac54d.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1587/builds/3876) and take a look at the build logs.
Check if the failure is related to this commit (bfac54d) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1587/builds/3876

Failed tests:

test_pyrepl

Failed subtests:

test_already_imported_module_without_origin_or_spec - test.test_pyrepl.test_pyrepl.TestPyReplModuleCompleter.test_already_imported_module_without_origin_or_spec

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto/build/Lib/test/test_pyrepl/test_pyrepl.py", line 1195, in test_already_imported_module_without_origin_or_spec
    module = importlib.import_module(mod)
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto/build/Lib/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1303, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1276, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1240, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'not_has_location'


Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto/build/Lib/test/test_pyrepl/test_pyrepl.py", line 1195, in test_already_imported_module_without_origin_or_spec
    module = importlib.import_module(mod)
  File "/home/buildbot/buildarea/3.x.cstratak-rhel9-s390x.lto/build/Lib/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1303, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1276, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1240, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'no_spec'

I suspect this is about file system timestamp resolution. Lemme attempt a fix.

@ambv Shall we backport this PR (& and the buildbot fix)? I think this can be considered a bugfix.

…ompletion (pythonGH-139461) Co-authored-by: Tomas R. <tomas.roun8@gmail.com>

Yup, gh-143438.

…ompletion (pythonGH-139461) Co-authored-by: Tomas R. <tomas.roun8@gmail.com>

loic-simon added 2 commits September 30, 2025 21:38

PyREPL module completion: check for already imported modules

6ed2776

Add blurb

48fd43f

loic-simon requested review from ambv, lysnikolaou and pablogsal as code owners September 30, 2025 20:00

bedevere-app Bot added the awaiting review label Sep 30, 2025

bedevere-app Bot mentioned this pull request Sep 30, 2025

Readline completion of module names in import statements #69605

Closed

loic-simon mentioned this pull request Sep 30, 2025

gh-69605: Hardcode some stdlib submodules in PyREPL module completion (os.path, collections.abc...) #138268

Merged

ashm-dev suggested changes Oct 1, 2025

View reviewed changes

Comment thread Lib/_pyrepl/_module_completer.py Outdated

bedevere-app Bot added awaiting core review and removed awaiting review labels Oct 1, 2025

loic-simon added 2 commits October 2, 2025 01:48

Better convey intent

7ac428e

[TEMP] debug tests on windows using modern technology (print statements)

6515e2f

loic-simon added 7 commits October 2, 2025 23:07

[TEMP] More debugging, where is my module??

7dbb906

[TEMP] More debugging, where is my module?? (bis)

ac3065a

[TEMP] Day 57, deep into debugging, I still don't know where is my mo…

75a33da

…dule

[TEMP] Moar logs

ce124b1

Merge branch 'main' into pyrepl-module-completion-check-for-already-i…

ee7047f

…mported-modules

[TEMP] Is it a FileFinder cache issue??

3f362cd

[TEMP] Looks like a cache issue indeed

ed8ce73

picnixz marked this pull request as draft October 3, 2025 13:21

bedevere-app Bot removed the awaiting core review label Oct 3, 2025

loic-simon added 2 commits October 3, 2025 15:41

Tests: clean FileFinder cache

19c49bb

Remove all debugging junk

16e44af

loic-simon marked this pull request as ready for review October 5, 2025 13:38

bedevere-app Bot added the awaiting review label Oct 5, 2025

loic-simon and others added 5 commits October 5, 2025 15:42

Small if refactor

14f6175

Merge branch 'pyrepl-module-completion-check-for-already-imported-mod…

bdd7bdf

…ules' of https://github.com/loic-simon/cpython into pyrepl-module-completion-check-for-already-imported-modules

Full test coverage for new code

78e4737

Merge branch 'python:main' into pyrepl-module-completion-check-for-al…

e3f1ddb

…ready-imported-modules

Merge branch 'main' into pyrepl-module-completion-check-for-already-i…

2644400

…mported-modules

tomasr8 self-requested a review December 28, 2025 15:29

tomasr8 reviewed Dec 30, 2025

View reviewed changes

loic-simon added 2 commits January 1, 2026 18:09

Merge branch 'main' into pyrepl-module-completion-check-for-already-i…

5fa70cf

…mported-modules

Check __spec__.has_location + refactor

d332e14

loic-simon commented Jan 1, 2026

View reviewed changes

Rename private helper

1a5327c

loic-simon requested a review from tomasr8 January 1, 2026 18:14

tomasr8 reviewed Jan 1, 2026

View reviewed changes

loic-simon added 3 commits January 2, 2026 16:56

Simplify implementation

d48f243

Fix find_spec call

e235e20

Remove unused import

f6757fe

ambv merged commit bfac54d into python:main Jan 5, 2026
55 checks passed

bedevere-app Bot removed the awaiting review label Jan 5, 2026

ambv pushed a commit to ambv/cpython that referenced this pull request Jan 5, 2026

pythongh-69605 : Check for already imported modules in PyREPL module c…

7db209b

…ompletion (pythonGH-139461) Co-authored-by: Tomas R. <tomas.roun8@gmail.com>

thunder-coding pushed a commit to thunder-coding/cpython that referenced this pull request Feb 15, 2026

pythongh-69605 : Check for already imported modules in PyREPL module c…

55f5b8b

…ompletion (pythonGH-139461) Co-authored-by: Tomas R. <tomas.roun8@gmail.com>

+                          if os.path.basename(imported_path) == "__init__.py":  # package
+                              imported_path = os.path.dirname(imported_path)
+                          import_location = os.path.dirname(imported_path)
+                          modules = list(pkgutil.iter_modules([import_location]))

+                          if os.path.basename(imported_path) == "__init__.py":  # package
+                              imported_path = os.path.dirname(imported_path)
+                          import_location = os.path.dirname(imported_path)
+                          modules = list(pkgutil.iter_modules([import_location]))

+                          return None
+                      if not spec.has_location:
+                          if spec.origin == "frozen":  # See Tools/build/freeze_modules.py
+                              return os.path.join(self._stdlib_path, f"{spec.name}.py")

gh-69605: Check for already imported modules in PyREPL module completion by loic-simon · Pull Request #139461 · python/cpython · GitHub

Uh oh!

Uh oh!

gh-69605: Check for already imported modules in PyREPL module completion - #139461

Conversation

loic-simon commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

loic-simon commented Sep 30, 2025

Uh oh!

Uh oh!

python-cla-bot Bot commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

picnixz commented Oct 3, 2025

Uh oh!

loic-simon commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomasr8 Dec 30, 2025

Choose a reason for hiding this comment

Uh oh!

loic-simon Jan 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

loic-simon left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

loic-simon Jan 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tomasr8 Jan 1, 2026

Choose a reason for hiding this comment

Uh oh!

loic-simon Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

tomasr8 Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ambv commented Jan 5, 2026

Uh oh!

bedevere-bot commented Jan 5, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

ambv commented Jan 5, 2026

Uh oh!

bedevere-bot commented Jan 5, 2026

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

ambv commented Jan 5, 2026

Uh oh!

tomasr8 commented Jan 5, 2026

Uh oh!

ambv commented Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

loic-simon commented Sep 30, 2025 •
edited
Loading

python-cla-bot Bot commented Oct 1, 2025 •
edited
Loading

loic-simon commented Oct 3, 2025 •
edited
Loading

loic-simon Jan 1, 2026 •
edited
Loading

loic-simon left a comment •
edited
Loading

loic-simon Jan 1, 2026 •
edited
Loading