Part of #2722.
The CI: Coverage workflow has failed every night since Aug 15 (last full green: run 31757710811, Aug 14), with a single incidental pass on Aug 23.
Symptom
The "Combine Coverage and Deploy" job fails at the step named "Combine coverage data":
=== Combining coverage data ===
Combined 2 files
=== Generating HTML, XML, and text reports ===
Combined 2 files
Plugin 'Cython.Coverage.Plugin' did not provide a file reporter for 'cuda/bindings/_lib/cyruntime/cyruntime.pxi'.
##[error]Process completed with exit code 1.
On other nights the same error names a different file — e.g. run 32539916399 (Aug 22) and run 32676345017 (Aug 24) both report cuda/core/_memory/_location.pxd.
The step name is misleading. coverage combine succeeds — the log prints Combined 2 files. The failure is coverage html, four commands later in the same run: block, surfacing under the step's name because the shell is bash -e. This cost real time during diagnosis.
Root cause
Cython has two file types that are not standalone modules: .pxd (declaration/header) and .pxi (textually included into another file). Neither compiles to its own translation unit, so Cython.Coverage.Plugin cannot construct a FileReporter for them. When coverage.py is handed one, the plugin returns nothing and coverage.py treats that as fatal, aborting the whole report.
This is already known — .coveragerc carries an ignore list with the comment "Omits specific definition files that causes plugin errors". The problem is that it enumerates three files by hand:
omit =
*/windll.pxd
*/_lib/windll.pxd
*/_lib/utils.pxd
Two new files of that kind have since landed and were never added:
Nothing warns you when a new .pxd/.pxi lands. Every one is a latent break.
Why it passed once (Aug 23): whether a given file reaches the report depends on whether any of its traced lines actually executed that night, and which offender trips first depends on iteration order. This is a class of failure, not a single regression — which is why the timeline looks flaky and there is no single "breaking commit".
Fix
Replace the enumeration with a pattern, in [run]:
[run]
plugins = Cython.Coverage
core = ctrace
branch = False
relative_files = True
# Cython's plugin cannot build a FileReporter for declaration/include files.
# Pattern, not an enumeration: every new .pxd/.pxi otherwise breaks `coverage html`.
omit =
*/*.pxd
*/*.pxi
It must be [run], not [report]. Verified against coverage 7.15.4:
- report_core.get_analysis_to_report() calls coverage._get_file_reporters(morfs) before applying the report_omit matcher, and the PluginError is raised inside that call (control.py:1051). A [report] omit entry is evaluated too late to suppress it.
- The two sections are independent — report_omit does not fall back to run_omit — so moving the list would silently drop the protection that currently works for windll.pxd/utils.pxd.
- The existing three entries work precisely because they are in [run]: they keep the files out of the measured set at collection time, and both collection jobs pass --cov-config=$REPO_ROOT/.coveragerc (coverage.yml:183-195, :430-449), so the data arrives at the combining runner already filtered.
Glob behaviour confirmed against coverage.files.GlobMatcher: */*.pxd and */*.pxi match both offenders and do not match .pyx or .py files.
Also worth doing
- Split the mega-step. coverage combine, coverage html, coverage xml and coverage report should be separate steps so the failing command is named in the UI.
- Consider coverage html -i as belt-and-braces — but only after the omit fix. On its own it would have hidden this for months.
Severity
Not release-blocking; no product code is affected and no test signal is lost. But coverage reporting has been dark since Aug 14, so any coverage regression in that window is invisible.
Part of #2722.
The CI: Coverage workflow has failed every night since Aug 15 (last full green: run 31757710811, Aug 14), with a single incidental pass on Aug 23.
Symptom
The "Combine Coverage and Deploy" job fails at the step named "Combine coverage data":
On other nights the same error names a different file — e.g. run 32539916399 (Aug 22) and run 32676345017 (Aug 24) both report cuda/core/_memory/_location.pxd.
Root cause
Cython has two file types that are not standalone modules: .pxd (declaration/header) and .pxi (textually included into another file). Neither compiles to its own translation unit, so Cython.Coverage.Plugin cannot construct a FileReporter for them. When coverage.py is handed one, the plugin returns nothing and coverage.py treats that as fatal, aborting the whole report.
This is already known — .coveragerc carries an ignore list with the comment "Omits specific definition files that causes plugin errors". The problem is that it enumerates three files by hand:
omit = */windll.pxd */_lib/windll.pxd */_lib/utils.pxdTwo new files of that kind have since landed and were never added:
Nothing warns you when a new .pxd/.pxi lands. Every one is a latent break.
Why it passed once (Aug 23): whether a given file reaches the report depends on whether any of its traced lines actually executed that night, and which offender trips first depends on iteration order. This is a class of failure, not a single regression — which is why the timeline looks flaky and there is no single "breaking commit".
Fix
Replace the enumeration with a pattern, in [run]:
It must be [run], not [report]. Verified against coverage 7.15.4:
Glob behaviour confirmed against coverage.files.GlobMatcher: */*.pxd and */*.pxi match both offenders and do not match .pyx or .py files.
Also worth doing
Severity
Not release-blocking; no product code is affected and no test signal is lost. But coverage reporting has been dark since Aug 14, so any coverage regression in that window is invisible.