Repository navigation
Split up JUnit XML output per file #4233
Description
Hello! I've recently started running mypy with --junit-xml since our CI system supports parsing JUnit output (and it's been a really useful feature!).
Currently, mypy's JUnit output lumps together all of the error output into one "testcase", like so:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="1" time="0.826">
<testcase classname="mypy" file="mypy" line="1" name="mypy" time="0.826">
<failure message="mypy produced messages">testmodule/bar.py:2: error: The return type of "__init__" must be None
testmodule/bar.py:2: error: Missing return statement
testmodule/foo.py:4: error: Self argument missing for a non-static method (or an invalid type for self)
testmodule/foo.py:5: error: Incompatible return value type (got "str", expected "int")</failure>
</testcase>
</testsuite>Would anyone have any objection to breaking this output up into 1 testcase per file? I imagine it would look something like this:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="mypy" skips="0" tests="3" time="0.826">
<testcase classname="testmodule.bar" file="testmodule/bar.py" line="0" name="testmodule.bar">
<failure message="mypy produced messages">testmodule/bar.py:2: error: The return type of "__init__" must be None
testmodule/bar.py:2: error: Missing return statement</failure>
</testcase>
<testcase classname="testmodule.foo" file="testmodule/foo.py" line="0" name="testmodule.foo">
<failure message="mypy produced messages">testmodule/foo.py:4: error: Self argument missing for a non-static method (or an invalid type for self)
testmodule/foo.py:5: error: Incompatible return value type (got "str", expected "int")</failure>
</testcase>
<testcase classname="testmodule.file_with_no_issues" file="testmodule/file_with_no_issues.py" line="0" name="testmodule.file_with_no_issues"></testcase>
</testsuite>There are a few benefits to splitting it up by file:
- CI systems that support the JUnit format can nicely format each error message per file, making it easier for developers to see which files need fixing
- CI systems that collect and aggregate metrics about which test cases fail most often / how many tests are run can have more detailed information to play with
- We can account for files that don't have typing issues (like testmodule/file_with_no_issues.py in the example above).
As an alternative, I considered having each typing error surface as a failed test case, but that makes the total number of tests and the number of failing tests vary wildly, making it difficult for CI systems that try to aggregate test case data.
I would love to implement this, just wanted to make sure no one had any objections to this before starting on it!
For implementation, I'm thinking of pulling in https://github.com/kyrus/python-junit-xml to actually generate the XML.
Activity
gvanrossum commented on Nov 10, 2017
kennydo commented on Nov 10, 2017
I put the sample per-file JUnit XML file through our CI system (Circle CI) with tests="3" in the testsuite, but with the testcase with no errors removed, and the UI displayed "Your build ran 2 tests with 2 failures", not 3. I think other CI systems would also expect each test to have an associated testcase, so omitting them might not be an option.
In general, I'd really like to keep the number of testcases equal to the number of typechecked files. There are a few benefits, like being able to correlate the change in number of files type-checked with the change in execution time. Also, we can see when a previously failing file type later checks successfully (instead of having that file blink into and out of existince in the test suite). I think adding these successful testcases to the output XML won't be excessively large (the JUnit XML from our is also pretty massive).
Adding that newline sounds good to me!
gvanrossum commented on Nov 11, 2017
dragongame commented on May 25, 2018
Has any progress been made on this?
What is missing for this to be included?
gvanrossum commented on May 25, 2018
Well there's a PR out but there are some questions raised in code review that haven't been answered AFAICT. Please follow up there if you have questions or feedback.
gvanrossum commented on May 30, 2018
So, before you spend more time on my feedback, I showed the result to some coworkers and so far the only comment I got was "Hmm, I like it being more condensed myself. I wouldn't find that any more readable, just more clicks to see all the errors." I kind of sympathize. Note that this was on a diff where there happened to be errors in about 30 different files. Maybe if you really want this it could be enabled by a separate flag???
kennydo commented on May 31, 2018
With files spread over 30 different files, having to do all that clicking does sound tedious.
I'd be OK with having a separate flag for this, since the utility of this change is pretty dependent on each developer/codebase's workflow. For our codebase, we're still incrementally adding typing, which is why we care about how many files successfully typechecked. We also generally run mypy locally a few times on our changed files before CI is run on our branches, so the number of failing files in CI would drop off quickly.
gvanrossum commented on May 31, 2018
Honestly I'm torn. We already have too many flags. And in our (internal) CI the same problem occurs for failing tests -- if you have a lot of failing tests you must click on each one separately. Let's just proceed.
jisaacstone commented on Apr 4, 2019
Yeah honestly having the whole thing be a single blob is less than ideal. I think most people would expect the report to have the correct metadata such as line number, file, etc.
However, since the output of mypy is nearly identical go flake8 the tools for flake8 such as flake8-junit-report pretty much just work with the mypy output. Anyone who desires a quick solution can just do that for now.
But I'm more than happy to get the --junit-xml flag to work the same way if there is a consensus that it is what people want?
gvanrossum commented on Apr 4, 2019
Let’s close this issue. I think it would be better to leave the xml generation to a separate tool.
dundee commented on Feb 22, 2020
I just created simple tool for this task: https://pypi.org/project/mypy2junit/
Feedback welcomed!
Mattwmaster58 commented on Mar 2, 2020
@dundee mypy can already generate JUnit XML, it's just that it isn't split up per file
dundee commented on Mar 2, 2020
@Mattwmaster58 Yes, this is why I created this tool.
RJPercival commented on Feb 8, 2024
This ended up being implemented by #16388.