bpo-46425: fix direct invocation of `test_importlib` by sobolevn · Pull Request #30682 · python/cpython · GitHub

Multiple tests in test_importlib were not suited to direct invocation. Why?

Some tests did miss unittest.main() call
Some tests used relative import style, here's how it were:

» ./python.exe ./Lib/test/test_importlib/test_open.py
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 4, in <module>
    from . import data01
    ^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

In several places relative imports were used in setUp(). It was also problematic:

» ./python.exe ./Lib/test/test_importlib/test_open.py
..................../Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py:71: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import namespacedata01
EEEEEE............
======================================================================
ERROR: test_open_binary (__main__.OpenDiskNamespaceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 71, in setUp
    from . import namespacedata01
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_open_binary_FileNotFoundError (__main__.OpenDiskNamespaceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 71, in setUp
    from . import namespacedata01
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_open_text_FileNotFoundError (__main__.OpenDiskNamespaceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 71, in setUp
    from . import namespacedata01
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_open_text_default_encoding (__main__.OpenDiskNamespaceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 71, in setUp
    from . import namespacedata01
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_open_text_given_encoding (__main__.OpenDiskNamespaceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 71, in setUp
    from . import namespacedata01
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_open_text_with_errors (__main__.OpenDiskNamespaceTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/./Lib/test/test_importlib/test_open.py", line 71, in setUp
    from . import namespacedata01
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package

----------------------------------------------------------------------
Ran 38 tests in 0.017s

FAILED (errors=6)

Now, all test_importlib modules pass the local check 🎉

Size of the PR

This PR is quite big. If desireable, I can split it into multiple parts.

CC @corona10 as my mentor.

https://bugs.python.org/issue46425

So I'm of two minds on this. One, this was actually done on purpose as the expectation is to run the tests via -m test which does not have any of these issues, e.g. python -m test test_importlib.test_open. And so the entire test suite is kind of viewed as a single unit. Plus I prefer relative imports. 😁

But none of this is also important, so I don't object to this PR either. 🙂 I'll let @corona10 or someone else handle reviewing and merging.

I'm not quite sure what the objective here is. Is it only so you can run any of the files directly as a script? For something as extensive as the importlib test suite, what's the advantage of running individual modules as scripts? Is this something someone needs or is your goal only consistency? (We generally discourage blanket changes motivated by a desire for consistency, due to the costs of churn in a large project like this.)

Almost always, you would want to run all of the importlib tests:

./python -m test test_importlib
./python -m unittest test.test_importlib
(rarely) ./python -m test.test_importlib

If you are actually working on importlib and want to run a specific subset of the tests then you'd usually use "-m", for example:

./python -m unittest test.test_importlib.builtin.test_loader
(specific test) ./python -m unittest test.test_importlib.builtin.test_loader.LoaderTests.test_module (which you can't do as cleanly when invoking with a script)

So I'd be interested in hearing the use case for running one of the individual importlib test files as a script. Also, you should definitely get @brettcannon's feedback since he wrote nearly all these tests and likely has a clear opinion on the matter.

In case there is a good enough use case for run-as-script then the change mostly LGTM. I've left some comments noting files that should be left alone.

Thanks, Brett.

I think that the main thing here is that most modules are working fine and only some are broken when invoked directly as python Lib/test/*. It might be confusing.

I think that the main thing here is that most modules are working fine and only some are broken when invoked directly as python Lib/test/*. It might be confusing.

That's fair, but I want to be clear that nothing is "broken". There is no general rule that sub-tests need to function as separate execution targets. As I have said, I don't object to the change specifically, but I'm also not going to be the one who merges it either. 😁

For me, on Windows, with fresh 3.11, python -m test.test_importlib.test_open runs the open tests. python Lib/test/test_import/test_open.py (or py ...) in a directory containing Lib fails with the import error above. If I load test_open in an IDLE editor and run it (F5), it also fails, similarly. To me, this is broken. I would feel the same if using any other editor that can run a saved file with an appropriate program.

That's fair, but I want to be clear that nothing is "broken".

Yeah, this word might a bit harsh. Let's not use it 🙂
Instead, I am going to illustrate my motivation behind this change.

Let's say I am newcomer to CPython's source code (which I +- am). And I want to work on some feature/bugfix in importlib. I obviously need to run my tests. Let's say I need Lib/test/test_importlib/builtin/test_finder.py. I read this file and I see unittest.main() in the end. Oh, I know what it does!

cpython/Lib/test/test_importlib/builtin/test_finder.py

Lines 97 to 98 in cfadcc3

    
           if __name__ == '__main__': 
        
               unittest.main()

But, if I try to run it directly (which this module declare to support) - tests will fail.

I think, it can be improved! And an it is easy to do: change imports, add unittest.main() in some files where it was missing. Done ✨

This change causes problems within the backports (importlib_metadata, importlib_resources). This code is kept in sync with those projects and thus (a) relies on the relative imports to find the fixtures in a relative location and (b) would never expect the modules to be invoked as a script. As primary maintainer of importlib.metadata and importlib.resources, I'd prefer to keep the code portable and simple (encapsulated). It's conceivable that the backport could maintain a forked version of these files that doesn't include these behaviors, but that adds a cognitive and maintenance burden to the effort.

In my effort, the lack of __name__ == '__main__' was intentional and represented a signal that these files were not meant to be executed, but were meant to be invoked through a test runner (i.e. python -m test, python regrtest, python -m unittest discover, or pytest (in the case of the backports)). I'd like to restore that expectation and back out these changes for tests relating to importlib.metadata and importlib.resources. I have less of an opinion on the changes to other importlib modules.

@jaraco thanks! I will send a new PR with restored changes in importlib_metadata and importlib_resources in 10 mins.

…t_importlib` (GH-30682)" This reverts commit 57316c5 for files pertaining to importlib.metadata and importlib.resources.

@jaraco thanks! I will send a new PR with restored changes in importlib_metadata and importlib_resources in 10 mins.

Thanks sobolevn. I missed your message prior to preparing and submitting #30799. Since it's not clear the demarcation of the tests, I figured it would be better if I helped out with that part. I went through and flagged what I think are the relevant files. I'm going to use that PR to validate that I got the correct files as well (by performing a dry-run sync into the backports).

…t_importlib` (GH-30682)" (GH-30799) This reverts commit 57316c5 for files pertaining to importlib.metadata and importlib.resources.

bpo-46425: fix direct invocation of test_importlib

6d24205

sobolevn requested review from brettcannon, encukou, ericsnowcurrently, ncoghlan and warsaw as code owners January 19, 2022 10:30

the-knights-who-say-ni added the CLA signed label Jan 19, 2022

bedevere-bot added the awaiting review label Jan 19, 2022

sobolevn added the skip news label Jan 19, 2022

brettcannon removed their request for review January 19, 2022 20:06

ericsnowcurrently reviewed Jan 19, 2022

View reviewed changes

Comment thread Lib/test/test_importlib/__main__.py

Comment thread Lib/test/test_importlib/resources/util.py

Do not touch __main__ and util

8cb7df8

asvetlov approved these changes Jan 22, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Jan 22, 2022

asvetlov merged commit 57316c5 into python:main Jan 22, 2022

bedevere-bot removed the awaiting merge label Jan 22, 2022

jaraco added a commit that referenced this pull request Jan 22, 2022

bpo-46425 : Partially revert "bpo-46425 : fix direct invocation of `tes…

1e0419f

…t_importlib` (GH-30682)" This reverts commit 57316c5 for files pertaining to importlib.metadata and importlib.resources.

jaraco mentioned this pull request Jan 22, 2022

bpo-46425: Partially revert GH-30682 #30799

Merged

sobolevn mentioned this pull request Jan 22, 2022

bpo-46425: revert changes to importlib.metadata and importlib.resources tests #30800

Closed

jaraco mentioned this pull request Jan 22, 2022

bpo-46474: sync with importlib_metadata 4.10.0 #30802

Merged

jaraco mentioned this pull request Jan 22, 2022

bpo-46474: Avoid REDoS in EntryPoint.pattern (sync with importlib_metadata 4.10.1) #30803

Merged

jaraco added a commit that referenced this pull request Jan 23, 2022

bpo-46425 : Partially revert "bpo-46425 : fix direct invocation of `tes…

d888ff5

…t_importlib` (GH-30682)" (GH-30799) This reverts commit 57316c5 for files pertaining to importlib.metadata and importlib.resources.

sobolevn mentioned this pull request Apr 16, 2022

Multiple test modules fail to run if invoked directly #90583

Closed

bpo-46425: fix direct invocation of `test_importlib` by sobolevn · Pull Request #30682 · python/cpython · GitHub

Repository navigation

bpo-46425: fix direct invocation of test_importlib - #30682

bpo-46425: fix direct invocation of test_importlib#30682
asvetlov merged 2 commits into
python:mainfrom
sobolevn:issue-46425-4

sobolevn commented Jan 19, 2022 •
edited
Loading

Uh oh!

Uh oh!

brettcannon commented Jan 19, 2022

Uh oh!

ericsnowcurrently left a comment

Uh oh!

Uh oh!

Uh oh!

sobolevn commented Jan 19, 2022

Uh oh!

brettcannon commented Jan 19, 2022

Uh oh!

terryjreedy commented Jan 20, 2022

Uh oh!

sobolevn commented Jan 21, 2022

Uh oh!

jaraco commented Jan 22, 2022 •
edited
Loading

Uh oh!

Uh oh!

sobolevn commented Jan 22, 2022

Uh oh!

jaraco commented Jan 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bpo-46425: fix direct invocation of `test_importlib` by sobolevn · Pull Request #30682 · python/cpython · GitHub

Repository navigation

bpo-46425: fix direct invocation of test_importlib - #30682

bpo-46425: fix direct invocation of test_importlib#30682asvetlov merged 2 commits intopython:mainpython/cpython:mainfrom sobolevn:issue-46425-4sobolevn/cpython:issue-46425-4Copy head branch name to clipboard

Conversation

sobolevn commented Jan 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Size of the PR

Uh oh!

brettcannon commented Jan 19, 2022

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sobolevn commented Jan 19, 2022

Uh oh!

brettcannon commented Jan 19, 2022

Uh oh!

terryjreedy commented Jan 20, 2022

Uh oh!

sobolevn commented Jan 21, 2022

Uh oh!

jaraco commented Jan 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sobolevn commented Jan 22, 2022

Uh oh!

jaraco commented Jan 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bpo-46425: fix direct invocation of test_importlib#30682
asvetlov merged 2 commits into
python:mainfrom
sobolevn:issue-46425-4

sobolevn commented Jan 19, 2022 •
edited
Loading

jaraco commented Jan 22, 2022 •
edited
Loading