Repository navigation
gh-62432: unittest runner: Exit code 5 if no tests were run - #102051
Merged
gpshead merged 13 commits intoApr 27, 2023
Merged
gpshead merged 13 commits into
gpshead merged 13 commits into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
stefanor
force-pushed
the
fail-zero-tests
branch
from
eab4b21 to
d8b54a3
Compare
February 19, 2023 16:37
As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7 It is common for test runner misconfiguration to fail to find any tests, this should be an error. Fixes: python#62432
stefanor
force-pushed
the
fail-zero-tests
branch
from
d8b54a3 to
0687ca8
Compare
February 19, 2023 16:41
Copy link
Copy Markdown
Member
|
Tests/macOS is routinely failing; ignore currently. This is a design bug fix, hence enhancement (feature change), as much as I would like to pretend otherwise. |
Sorry, something went wrong.
hugovk
reviewed
Feb 20, 2023
Comment thread
Lib/unittest/main.py
Outdated
Comment thread
Lib/unittest/runner.py
Outdated
Comment thread
Lib/test/test_unittest/test_program.py
Outdated
Comment thread
Lib/test/test_unittest/test_runner.py
Outdated
Comment thread
Misc/NEWS.d/next/Library/2023-02-19-12-37-08.gh-issue-62432.GnBFIB.rst
Outdated
Comment thread
Lib/test/test_unittest/test_program.py
Outdated
stefanor
requested review from
hugovk
and removed request for
ezio-melotti and
gpshead
February 20, 2023 11:51
Copy link
Copy Markdown
Contributor
Author
Urgh, that removal wasn't intentional, I just hit the re-request review button. |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
@gpshead: Ping? (it's been 2 months) |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
|
Thank you! |
Sorry, something went wrong.
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 14, 2023
- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 15, 2023
- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 15, 2023
- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 15, 2023
- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992
copybara-service Bot
pushed a commit
to abseil/abseil-py
that referenced
this pull request
Sep 15, 2023
- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565771935
Copy link
Copy Markdown
Member
|
The new Python 3.12 message surprises users when all tests are skipped: see issue #113661. |
Sorry, something went wrong.
Copy link
Copy Markdown
|
The PR description says:
Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running. |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
Author
Yeah, I'd agree with that. |
Sorry, something went wrong.
texhnolyze
added a commit
to bit-bots/bitbots_main
that referenced
this pull request
Dec 12, 2024
to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. See: colcon/colcon-core#678 See: python/cpython#102051
texhnolyze
added a commit
to bit-bots/udp_bridge
that referenced
this pull request
Dec 16, 2024
to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. Additionally removed old ros 1 integration tests. See: colcon/colcon-core#678 See: python/cpython#102051
3 of 6 tasks
texhnolyze
added a commit
to bit-bots/dynamic_stack_decider
that referenced
this pull request
Dec 16, 2024
to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. See: colcon/colcon-core#678 See: python/cpython#102051
3 of 6 tasks
texhnolyze
added a commit
to bit-bots/udp_bridge
that referenced
this pull request
Dec 16, 2024
to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. Additionally removed old ros 1 integration tests. See: colcon/colcon-core#678 See: python/cpython#102051
mrg123
added a commit
to mrg123/nanojev-mlx
that referenced
this pull request
Sep 19, 2026
Python 3.10 is in the test matrix, and on 3.10 unittest still exits 0 when it discovers no tests at all -- exit code 5 for "NO TESTS RAN" only landed in 3.12 (python/cpython#102051). So if tests/ were ever renamed or moved, the 3.10 leg would report success while running nothing at all. Capture the runner output and fail explicitly when zero tests were collected. Once tests are actually discovered the new branch cannot trigger. Verified both paths locally under `bash -eo pipefail`, the shell GitHub Actions uses: a simulated zero-test run (exit 0, "Ran 0 tests") is rejected with exit 1, and a normal 19-test run passes through untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Reviewers
Assignees
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7
It is common for test runner misconfiguration to fail to find any tests,
this should be an error.
Fixes: #62432