Adds perf ab so two flag/DI-reachable configurations can be compared in one hermetic process and
database. This is a harness-only PR; it does not optimize product code.
Pre-registered prediction
Recorded in this PR before implementation:
identical default configurations: reads 6, writes 1, queries 9, embedding requests 1, retrieved
items 43, and access-tracked items 25 on both sides; Recall@K/MRR 1.000; elapsed ratio CI contains
1.0, so the verdict is no significant difference;
Recall.MaxEntities 10 → 2: entities 10 → 2, retrieved items 43 → 35, access-tracked items
25 → 17; reads/writes/queries/embedding requests unchanged; Recall@K approximately 0.982 and MRR
1.000; no timing win unless the bootstrap interval's upper bound is below 1.0.
What changed
perf ab --control <spec> --candidate <spec> parses default, Recall.Max*, and
Recall.MinSimilarityScore configurations.
Order is counterbalanced as AB, BA; each configuration crosses over two equivalent,
owner-isolated fixture copies.
Six consecutive paired iterations are one bootstrap unit, preserving observed Docker/driver
autocorrelation.
The generated Markdown and JSON put exact counter ranges, retrieval quality, extraction quality,
p50 timings, candidate/control ratios, and bootstrap 95% CIs together.
iteration total is the pre-registered timing headline. Subspan intervals are explicitly
exploratory because they are not corrected for multiple comparisons.
State-mutating scenarios are rejected; PERF-W-02 cannot be causally paired without reset/restore.
Why the design changed from literal ABAB
The original strict-ABAB/shared-fixture implementation failed the required null test in both
directions. PERF-R-04 updates access timestamps, so one side changed the state the next side measured,
and bootstrapping correlated adjacent ratios independently made intervals too narrow. The corrected
state-isolated, dataset-crossover, counterbalanced cluster design passed the null twice.
Results
Null experiment — default vs default
Final-form runs:
20260726T014053Z__ab__hermetic-S-zero
20260726T021504Z__ab__hermetic-S-zero
Both report PASS — no significant difference, identical structural-work ranges, and zero quality
delta. Latest headline:
Metric
Control
Candidate
Delta / ratio
Iteration p50
30.22 ms
32.24 ms
1.029
Bootstrap 95% CI
—
—
0.992–1.086
Reads / writes / queries
6 / 1 / 9
6 / 1 / 9
0 / 0 / 0
Retrieved / access-tracked
43 / 25
43 / 25
0 / 0
Recall@K / MRR
1.000 / 1.000
1.000 / 1.000
0.000 / 0.000
Positive control — Recall.MaxEntities 10 → 2
Run 20260726T014215Z__ab__hermetic-S-zero:
Metric
Control
Candidate
Change
items.entities
10
2
-8 (-80.0%)
items.retrieved
43
35
-8 (-18.6%)
access_tracking.items
25
17
-8 (-32.0%)
Reads / writes / queries
6 / 1 / 9
6 / 1 / 9
0 / 0 / 0
Embedding requests
1
1
0
Iteration p50
24.65 ms
24.79 ms
ratio 0.908
Bootstrap 95% CI
—
—
0.809–1.009: no significant difference
Recall@K
1.000
0.982
-0.018
MRR / forbidden cases
1.000 / 0
1.000 / 0
unchanged
Extraction precision/recall
1.000 / 1.000
1.000 / 1.000
unchanged
The exact counter prediction held. The timing direction was lower but not significant, so this PR
does not claim a speed win. The zero-tolerance quality report correctly flags the Recall@K tradeoff.
Validation
New tests were run before implementation and failed to compile because the A/B command,
configuration parser, and bootstrap did not exist; focused suite now passes 13/13.
Unit suite: 3,403/3,403 passed.
Integration suite: 311 passed; the same 29 Nams.* tests fail because the configured workspace is
deprovisioned (workspace_not_provisioned), as documented before this PR.
Standard perf regression run 20260726T021600Z__m21-regression__hermetic-S-zero: quality 1.000
throughout and every published structural counter unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
M-21 — trustworthy in-process A/B measurement
Adds perf ab so two flag/DI-reachable configurations can be compared in one hermetic process and
database. This is a harness-only PR; it does not optimize product code.
Pre-registered prediction
Recorded in this PR before implementation:
items 43, and access-tracked items 25 on both sides; Recall@K/MRR 1.000; elapsed ratio CI contains
1.0, so the verdict is no significant difference;
25 → 17; reads/writes/queries/embedding requests unchanged; Recall@K approximately 0.982 and MRR
1.000; no timing win unless the bootstrap interval's upper bound is below 1.0.
What changed
Recall.MinSimilarityScore configurations.
owner-isolated fixture copies.
autocorrelation.
p50 timings, candidate/control ratios, and bootstrap 95% CIs together.
exploratory because they are not corrected for multiple comparisons.
Why the design changed from literal ABAB
The original strict-ABAB/shared-fixture implementation failed the required null test in both
directions. PERF-R-04 updates access timestamps, so one side changed the state the next side measured,
and bootstrapping correlated adjacent ratios independently made intervals too narrow. The corrected
state-isolated, dataset-crossover, counterbalanced cluster design passed the null twice.
Results
Null experiment — default vs default
Final-form runs:
Both report PASS — no significant difference, identical structural-work ranges, and zero quality
delta. Latest headline:
Positive control — Recall.MaxEntities 10 → 2
Run 20260726T014215Z__ab__hermetic-S-zero:
The exact counter prediction held. The timing direction was lower but not significant, so this PR
does not claim a speed win. The zero-tolerance quality report correctly flags the Recall@K tradeoff.
Validation
configuration parser, and bootstrap did not exist; focused suite now passes 13/13.
deprovisioned (workspace_not_provisioned), as documented before this PR.
throughout and every published structural counter unchanged.