FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

compute multiline maps cheaply in the sysmon core by reaperhulk · Pull Request #2220 · coveragepy/coveragepy · GitHub

compute multiline maps cheaply in the sysmon core - #2220

Merged
nedbat merged 6 commits into
coveragepy:mainfrom
reaperhulk:claude/sysmon-multiline-map
Jul 12, 2026
Merged

nedbat merged 6 commits into
coveragepy:mainfrom
reaperhulk:claude/sysmon-multiline-map

Conversation

Copy link
Copy Markdown
Contributor

(This PR was generated via claude but has undergone significant review and changes based on that review before submission)

When measuring branch coverage under sys.monitoring, the tracer needs each file's multiline map to resolve branch events. It currently gets it by running a full PythonParser.parse_source(). This change extracts the map computation into a shared multiline_map_from_tokens(). This is refactored to be the single place the map is built, used by both PythonParser._raw_parse and the sysmon core.

On the cryptography test suite (Python 3.14, branch mode), sysmon overhead drops from +35.2% to +30.0%; the remaining overhead is mostly branch_trails() analysis, which is the next PR. 😄

claude and others added 2 commits July 7, 2026 20:04
The sysmon core needs the multiline map (line -> first line of its
multi-line statement) to resolve branch events.  It got it by running a
full PythonParser.parse_source() per file — an ast.parse, a full
tokenization, a compile() via ByteParser, and AST walks — behind a
module-level functools.lru_cache(maxsize=20).  Test suites tracing more
than 20 files thrash that cache and re-parse files repeatedly (the
pyca/cryptography suite traces 192 files; see issue coveragepy#2172 for a report
of sysmon branch mode being slow on such suites).

Three changes:

- Make multiline_map_from_tokens() in parser.py the one place the map
  is computed: PythonParser._raw_parse tokenizes once into a list, gets
  the map from the shared builder, and its own token loop keeps only
  the exclusion and indent bookkeeping (its in-flight first_line
  tracking stays, since the exclusion logic needs the statement start
  before the map entry for the current statement exists).

- Use multiline_map_from_text() in the sysmon core instead of a full
  parse.  This is ~4x cheaper (0.34s vs 1.48s for the 192 files of the
  cryptography suite) and produces identical maps (verified on 445
  files: coverage's own source and tests, the stdlib, cryptography and
  its tests).

- Cache the maps in a plain dict on the SysMonitor instance, unbounded,
  so each traced file is tokenized at most once per run.  The cache
  dies with the tracer, which also removes the cross-run staleness a
  module-level cache can have.

Measured on the cryptography suite (Python 3.14.2, branch mode, wall
time best of 3, base 35.44s): 47.90s (+35.2%) before, 46.07s (+30.0%)
after, with byte-identical coverage data.  The parse/report phase pays
one extra pure-Python pass over the already-materialized token list
(parse_source() over the 192 files: 1.75s -> 1.95s, ~+1ms per file),
which the reporting-phase caches already amortize.  Most of the
remaining measurement overhead is the branch_trails() analysis,
addressed separately by the sysmon-lazy-branch-resolver branch; the
two changes compose to ~0-2%.

Verified: parser results (statements, excluded, raw_excluded,
multiline_map) identical to released main on 445 files with a realistic
exclusion regex; coverage's own test suite failure set identical to
released main.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137DLbSXfm5v5bhCcz7xEKM
claude and others added 4 commits July 8, 2026 21:17
Add directed tests of multiline_map_from_text() for each statement shape, plus a parametrized check that it always matches the map PythonParser produces, since branch arcs are attributed with one and reports are keyed by the other.  Also test compute_multiline_map()'s fallbacks for missing, non-Python, and badly indented files, and that SysMonitor computes each file's map at most once per tracer instance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137DLbSXfm5v5bhCcz7xEKM
nedbat merged commit ee271ee into coveragepy:main Jul 12, 2026
42 checks passed

nedbat commented Jul 12, 2026

Copy link
Copy Markdown
Member

This is now released as part of coverage 7.15.1.

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL