FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-121111: Fix `zipimport` incorrectly dealing with namespace packages by ZeroIntensity · Pull Request #121161 · python/cpython · GitHub

Repository navigation

gh-121111: Fix zipimport incorrectly dealing with namespace packages - #121161

Closed
ZeroIntensity wants to merge 8 commits into
python:mainfrom
ZeroIntensity:fix-zipimport
Closed

ZeroIntensity wants to merge 8 commits into
python:mainfrom
ZeroIntensity:fix-zipimport

Conversation

mod = importlib.util.module_from_spec(spec)
importer.exec_module(mod)
self.assertEqual(mod.foo(), "foo")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

If here test the namespace module in zip, mkdir() must be removed, this bug is triggered by no directory entry in zip.

And don't do test via zipimport, that always fail/succeed, because find functons depend the loader's path prefix. Direct import namespace module can cover all cases.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'm a bit confused on what you want here - the test passes, why should mkdir() be removed?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Again, this bug is triggered by no directory entry in zip. If the directory entry is already exists, everything is OK.

Comment thread Lib/zipimport.py
# OK, we found it the old way. Let's return it.
return path

return self.prefix + fullname.replace(".", path_sep)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

As https://github.com/python/cpython/pull/121161/files#r1660062444 described, this patch has no effect, shoud patch _is_dir or _read_directory.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

As https://github.com/python/cpython/pull/121161/files#r1660062444 described, this patch has no effect, shoud patch _is_dir or _read_directory.

From what I found, there's no problem with _read_directory, and _is_dir is only relevant to find_spec (the problem still occurs in get_code, for example). This patch seems to fix the issue for both. Have you tried this fix on this issue?

SeaHOH Jul 1, 2024 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

For namespace package's dealing and operation, there has already clearly explained in the source code comment. You can see them in find_loader (<=py311), find_spec and _is_dir.

And, why your test with get_code (and get_filename, get_source, is_package also) was fail? Because find functons depend the loader's path prefix, different level modules have different level loaders, skip levels will be failed. Yes, this is _get_module_path which you patched, then it broken, return the full path at once instead of stepwise lookup.

If you like to test into zipimport functions, it should be get_data:

    def testImportSubmodulesInZip(self):
        with ZipFile(TEMP_ZIP, "w") as z:
            z.writestr("a/__init__.py", b'')
            z.writestr("a/b/c/__init__.py", b'def foo(): return "foo"')

        importer = zipimport.zipimporter(TEMP_ZIP + "/a/")
        spec = importer.find_spec("a.b")
        self.assertEqual(spec.loader, None)
        self.assertEqual(spec.submodule_search_locations,
                         [(TEMP_ZIP + "/a/b").replace("/", zipimport.path_sep)])
        self.assertRaises(zipimport.ZipImportError, importer.get_code, "a.b")
        self.assertEqual(importer.get_data("a/b/"), b"")

        sys.path.insert(0, TEMP_ZIP)
        try:
            import a.b.c
            self.assertIn('namespace', str(a.b).lower())
            self.assertEqual(a.b.c.foo(), "foo")
        finally:
            del sys.path[0]
            sys.modules.pop('a.b.c', None)
            sys.modules.pop('a.b', None)
            sys.modules.pop('a', None)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I agree with @SeaHOH that the problem is rather in _read_directory(). I created #121233 for this approach.

AraHaan commented Jul 1, 2024

Copy link
Copy Markdown
Contributor

This fix worked, Thank you so much.

SeaHOH commented Jul 1, 2024

Copy link
Copy Markdown

This fix worked, Thank you so much.

@AraHaan It just work, did not be fixed. Run the test to confirm.

AraHaan commented Jul 1, 2024

Copy link
Copy Markdown
Contributor

This fix worked, Thank you so much.

@AraHaan It just work, did not be fixed. Run the test to confirm.

It seems to work when backported to 3.12 for me.

SeaHOH commented Jul 1, 2024

Copy link
Copy Markdown

@serhiy-storchaka Could you take look at the testcase and patch?

ZeroIntensity commented Jul 1, 2024 •
edited
Loading

Copy link
Copy Markdown
Member Author

Can confirm that the test written by @SeaHOH continues to fail on the new patch. There were two bugs at play here, I only patched one of them (my original test case fails on the main branch, as of now).

The underlying problem is not directly with _is_dir, but it's that _read_directory doesn't have a way to represent empty directories. AFAIK, find_spec is only affected by this now.

SeaHOH commented Jul 1, 2024

Copy link
Copy Markdown

Sorry for my English is so bad. Seems #121233 fixed the bug.

brettcannon removed their request for review July 4, 2024 22:20
ZeroIntensity deleted the fix-zipimport branch July 9, 2024 22:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants


Back | FazBrowse Home | New Git URL