Pin each conformance leg to a spec-revision wire - #3304
Conversation
Sorry, something went wrong.
Replace the server job's `--suite active` and `--suite draft` legs, and reshape the client job to match, so each job runs three `--suite all` legs: one forced to the 2025-11-25 wire, one forced to 2026-07-28, and the existing bare leg at the harness's default wire. Each pinned leg selects every scenario the harness marks applicable at that revision, which is a superset of the frozen requirement set alpha.11's `tier-check` scores for it, and its wire no longer depends on which revision the harness treats as latest. At this pin `active` was already a subset of the bare leg and `draft` a subset of the 2026-07-28 leg at identical wires, so nothing that was judged before stops being judged; the 2025-11-25 leg additionally holds server-session-lifecycle, server-sse-polling, json-schema-2020-12 and json-schema-2020-12-preservation to that wire explicitly. Baselines become one file per leg: a new (empty) expected-failures.2025-11-25.yml, the existing 2026-07-28 file, and expected-failures.yml for the bare leg, whose headers now say which leg reads them and how they overlap. While here: give both jobs a 15-minute timeout, upload the per-check results on every run (always(), so a timed-out job still uploads what finished), and have run-client.sh emit a workflow warning when it rescues a contention flake now that an artifact on a green run no longer signals one. No-Verification-Needed: CI config and conformance fixture scripts only
There was a problem hiding this comment.
LGTM — CI-only restructuring of the conformance workflow, with the legs themselves validating the change on this PR's own run.
What was reviewed:
- The leg swap in conformance.yml: removed active/draft server legs vs. the new pinned legs, and that each leg pairs with the right baseline file and output dir.
- The new ::warning line in run-client.sh for annotation injection — scenario names are validated against ^[A-Za-z0-9/_-]+$ before reaching it, and output_dir is workflow-controlled.
- run-server.sh argument pass-through for the new --spec-version flags — verbatim "$@", no parsing to break.
- The if: always() upload and timeout-minutes: 15 — consistent with the stated goal of preserving partial results on a timeout.
Overview
The PR touches only CI assets: .github/workflows/conformance.yml (replaces the server active/draft legs and the bare client leg layout with three --suite all legs per job — pinned to 2025-11-25, pinned to 2026-07-28, and bare at the harness default wire), a new empty baseline expected-failures.2025-11-25.yml, header rewrites in the two existing baseline files, and a small addition to run-client.sh that emits a ::warning annotation when a flake is rescued. No src/ or test code changes.
Security risks
None of substance. The workflow keeps permissions: contents: read and persist-credentials: false; all actions remain SHA-pinned and unchanged. The only new string interpolation is the ::warning echo in run-client.sh; the ${scenarios[*]} values it embeds are extracted from harness output but rejected earlier in the script unless they match ^[A-Za-z0-9/_-]+$, which cannot carry annotation-command syntax, and output_dir comes from workflow-authored arguments.
Level of scrutiny
Low-to-moderate. This is CI-only with no effect on the published package or runtime behavior. The main thing that could go wrong is a coverage regression in what conformance CI measures — the PR's own description addresses this (active ⊂ bare all, draft ⊂ the pinned 2026-07-28 leg at identical wires) and backs it with local run counts for all six legs. Crucially, the change is self-validating: the reworked workflow executes on this PR, so a mispaired baseline file, a bad flag, or a stale entry fails the run visibly rather than silently. The failure mode of an empty baseline file being passed is also handled — the runners key off the harness's own baseline output.
Other factors
The baselines follow the established pattern from #3282 (one file per leg, stale-entry enforcement, reconcile-on-pin-bump headers now referencing all three files consistently). run-server.sh forwards arguments verbatim, so the new --spec-version flags need no script changes there. The bug hunting system found no issues, and there are no outstanding reviewer comments on the timeline. The if: always() upload and timeout-minutes additions are standard CI hygiene with clear inline justification.
Sorry, something went wrong.
Follow-up to #3282. Each conformance job now runs three --suite all legs — one forced to the 2025-11-25 wire, one forced to 2026-07-28, and the existing bare leg at the harness's default wire — with one baseline file per leg. The server job's active and draft legs go away.
Motivation and Context
alpha.11 froze a per-revision requirement set (conformance#447) that conformance tier-check --sdk python-sdk scores at each revision's own wire (stateful handshake for 2025-11-25, stateless per-request _meta for 2026-07-28). Our legs selected by suite (active, draft, bare all) and inherited whichever wire the harness defaults each scenario to, which is due to move when 2026-07-28 stops being the harness's draft version (conformance#426). So what CI measured at the 2025-11-25 wire was implicit and would silently change on a future pin bump.
--suite all --spec-version <rev> selects every scenario the harness marks applicable at <rev> and forces that wire. That selection is a superset of the frozen requirement set for <rev> (npx @modelcontextprotocol/conformance@0.2.0-alpha.11 list --requirements <rev> prints it), so the pinned legs give the tier signal directly: an expected-failures.<rev>.yml entry for a scenario in that set means tier-check is below 100% for that revision while CI is green. Both files are empty today (30/30 + 18/18 at 2025-11-25, 37/37 + 32/32 at 2026-07-28).
I went with pinned --suite all legs rather than --requirements <rev> legs because a requirements leg only ever reports the scenarios outside its frozen list (extensions, pending, added-after-release, and anything the harness adds later) — it never judges them — so it would still need an all leg beside it at the same wire, and would be a strict subset of that leg. The pinned all leg is the one leg that does both.
Nothing that was judged before stops being judged: at this pin active ⊂ bare all and draft ⊂ all --spec-version 2026-07-28, at identical wires. The new 2025-11-25 legs additionally hold server-session-lifecycle, server-sse-polling, json-schema-2020-12 and json-schema-2020-12-preservation to that wire explicitly rather than via the harness default. The bare leg stays because it is the only one that reaches extension-tagged scenarios (tasks-*, auth/dpop*, …), which never match a --spec-version filter; the tasks-* and per-check DPoP/WIF entries therefore stay in expected-failures.yml only. Where the bare leg overlaps a pinned leg, a failure needs an entry in both files — the headers say so.
While here, three small CI-evidence changes: both jobs get timeout-minutes: 15 (the harness's server runner has no per-scenario timeout, conformance#427); results upload on every run with if: always() so a timed-out job still uploads what finished; and since an artifact on a green run no longer means "flake rescued", run-client.sh now emits a workflow ::warning when it rescues one.
How Has This Been Tested?
All six legs run locally through run-server.sh / run-client.sh against conformance c321dd3 (== 0.2.0-alpha.11):
Plus the rescue path on the new client 2025-11-25 leg with a wrapper client that fails sse-retry once: suite reports it unexpected → solo re-run 3/3 → exit 0, FLAKE_RESCUED written, warning annotation emitted. A wrapper that fails tools_call every time still exits 1 ("real failure, not suite contention").
Breaking Changes
None. CI-only.
Types of changes
Checklist
Additional context
Step names and output directories are now <mode>-2025-11-25, <mode>-2026-07-28, <mode>-all; the server-active / server-draft artifact directories no longer exist.
AI Disclaimer