Fix pytest result parsing counting more tests than were run - #5
Open
KTanmay1 wants to merge 1 commit into
Open
Conversation
Copy link
Copy Markdown
Sorry, something went wrong.
`test_pattern` uses `\s+` between the test id and the status token. `\s`
matches newlines, so a test id ending one line pairs with a status token
beginning the next. pytest's short summary emits exactly that shape for
setup and teardown errors, whose lines carry no trailing reason:
ERROR tests/test_smaz.py::test_one
ERROR tests/test_smaz.py::test_two
Line 1 ends in `::test_one`, line 2 begins with `ERROR`, and the pair
matches. Chained across the block, N entries are counted as 2N-1.
Observed on antirez_smaz (C -> Python) against an empty tree: 18 tests
run, 35 counted. Reproduced standalone at 3 -> 5.
Only entries with no trailing reason chain. Collection errors print
`ERROR tests/x.py` with no `::` and never matched. Failures print
`FAILED tests/x.py::test_y - AssertionError: ...`, so nothing chains onto
them. A fully passing run is unaffected.
Because Ni inflates while Ti does not, APR and AMPR are depressed on
affected tasks; SR is unchanged.
Fix: anchor the match to the start of a line and restrict the separator
to spaces and tabs.
Verified against PythonTestAnalyzer.analyze_output:
3 setup errors total=3 (was 5) rate=0.0%
2 pass 1 fail total=3 rate=66.7%
all passing total=2 rate=100.0%
This changes reported APR/AMPR on tasks with setup or teardown errors.
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RepoTransAgent/test_analyzer.py:126
\s matches newlines, so a test id ending one line pairs with a status token beginning the next. pytest's short summary emits exactly that shape for setup and teardown errors, whose lines carry no trailing reason:
Line 1 ends in ::test_one, line 2 begins with ERROR, and the pair matches. Chained across the block, N entries are counted as 2N - 1.
Observed on antirez_smaz (C → Python) against an empty tree: 18 tests run, 35 counted.
Reproduction
Scope
Only entries with no trailing reason chain:
Because Ni inflates while Ti does not, APR and AMPR are depressed on affected tasks. A fully passing run is unaffected, so SR is unchanged.
Fix
Anchor to line start and restrict the separator to spaces and tabs.
Verification
Through PythonTestAnalyzer.analyze_output, not the regex alone:
This changes reported APR/AMPR on tasks with setup or teardown errors. Flagging that explicitly since it affects published results — happy to hold if you would rather land it alongside a re-run.