Surface skipped conformance scenarios as baselined known failures - #3030
Surface skipped conformance scenarios as baselined known failures#3030maxisbey merged 3 commits into
Conversation
Sorry, something went wrong.
…wn failures The tasks-* scenarios are extension-tagged: they never match a --spec-version filter and sit on the harness's pending list, so none of the existing server legs can select them — they were silently absent from CI with no skip line and no waiver. server-sse-polling (pending, removed at 2026-07-28) was equally unreachable. Add a bare --suite all server leg, which selects every scenario shipped with the pinned harness, and baseline the nine runnable tasks-* scenarios in expected-failures.yml. The stale-entry rule turns the baseline into a forced burn-down list as tasks support lands. server-sse-polling turns out to already pass. tasks-status-notifications is deliberately not baselined: the harness skips it unconditionally, and a baseline entry for a scenario with no failing checks is itself flagged stale.
…header fixture tool All five SEP-2243 Mcp-Param-* server-validation checks in http-custom-header-server-validation skipped silently because no everything-server tool carried an x-mcp-header annotation — the scenario printed a green 0-passed/0-failed in every leg that selected it. Add a test_x_mcp_header tool whose region parameter carries the annotation via json_schema_extra (mirroring the typescript-sdk fixture). The checks now execute: the accept-path checks pass and the reject-path checks fail because server-side Mcp-Param validation is not implemented yet, so the scenario is baselined in both expected-failures files. The stale-entry rule forces the entries out when validation lands.
There was a problem hiding this comment.
No issues found across 4 files
Tip: cubic could auto-approve low-risk PRs like this, if it thinks it's safe to merge. Learn more
Re-trigger cubic
Sorry, something went wrong.
There was a problem hiding this comment.
LGTM — CI conformance config plus an example-server fixture tool only; no SDK library code touched.
Extended reasoning...Overview
This PR adds a fourth server conformance leg (bare --suite all) to .github/workflows/conformance.yml, baselines the newly-reachable scenarios (9 tasks-* extension scenarios and http-custom-header-server-validation) in the two expected-failures YAML files, and adds a test_x_mcp_header fixture tool to the everything-server example so the SEP-2243 Mcp-Param server-validation checks actually execute. No SDK library code is modified.
Security risks
None of substance. The new workflow step invokes the existing run-server.sh helper with different flags — no new actions, secrets, permissions, or untrusted inputs are introduced, and the harness package remains pinned by SHA-256 as before. The fixture tool is a trivial echo-style tool in an example conformance server, not in the published package.
Level of scrutiny
Low. The blast radius is limited to CI signal (a wrong baseline entry would fail the conformance job, not affect the shipped SDK) and a conformance fixture in an example server. The harness's stale-entry rule means over-broad waivers are self-correcting as features land. The changes follow the established pattern already used by the existing legs and mirror typescript-sdk's CI.
Other factors
The bug hunting system found no issues, the author documents local replication of all four legs against the pinned harness, and the YAML/workflow edits are mechanical and well-commented. The only operational risk noted (possible server-sse-polling flakiness on small runners) is acknowledged in the PR description and would only require a CI rerun, not a code change.
Sorry, something went wrong.
…cks it surfaces conformance 4944b268 (0.2.0-alpha.8, unpublished on npm — same sha256-verified pkg.pr.new mechanics as the previous pin) includes upstream #372: checks whose prerequisite is missing now fail with a 'Not testable' message instead of skipping outside the pass/fail denominator. For this repo that surfaces one new known failure: server-stateless fails its three subscriptions/listen MUST checks, because the everything-server's legacy resources/subscribe handlers make it advertise resources.subscribe in server/discover while subscriptions/listen answers -32601 — advertised-but-rejected is now a failure, not a skip. Baseline the scenario in both expected-failures files until the listen runtime lands. The two listChanged SHOULD checks remain legitimately skipped (declared false). The tasks-* scenarios fail with higher check counts (their cascade skips now fail too) but were already baselined; client legs are unaffected.
There was a problem hiding this comment.
I didn't find any bugs — the changes are confined to CI configuration and a conformance fixture tool — but the new bare --suite all leg, the baseline additions, and the harness pin bump to an unpublished pkg.pr.new build are CI policy decisions worth a maintainer's sign-off.
Extended reasoning...Overview
This PR touches only conformance CI infrastructure: .github/workflows/conformance.yml (bumps the pinned harness from b18aa918 to 4944b268/0.2.0-alpha.8 and adds a fourth server leg running bare --suite all), both expected-failures*.yml baselines (adds server-stateless, http-custom-header-server-validation, and 9 tasks-* scenarios as known failures), and the example everything-server (adds a trivial test_x_mcp_header fixture tool with an x-mcp-header schema annotation). No SDK runtime code is modified.
Security risks
Low. The conformance harness is fetched from a pkg.pr.new URL, but it is pinned by commit SHA and verified against a recorded SHA-256 digest before use, continuing the pattern already established in the workflow; the workflow runs with contents: read only. The new fixture tool is an example-server change with no security surface. I cannot independently verify that the new digest corresponds to a genuine build of the upstream commit, which is one reason a maintainer should glance at the pin bump.
Level of scrutiny
Moderate. Nothing here ships in the SDK package, and a mistake would surface as a red (or misleadingly green) CI run rather than a user-facing defect. However, the PR encodes several policy choices a maintainer typically owns: pinning CI to an unpublished alpha build, adding a fourth conformance leg (extra CI time, mostly duplicate coverage per the author's own description), and adopting per-scenario baseline waivers as a burn-down ledger — including the caveat the author documents that a server-stateless waiver masks regressions in that scenario's other ~25 currently-passing checks.
Other factors
The bug-hunting pass found no issues, the author reports replicating all four server legs locally against the pinned harness, and the comments in the workflow and baseline files are unusually thorough about the trade-offs. The remaining open questions are judgment calls (suite choice, baseline granularity, flake risk of server-sse-polling on 2-core runners) rather than correctness bugs, so this is a quick review for someone who owns the conformance setup.
Sorry, something went wrong.
Conformance CI's green currently overstates coverage in two ways: 11 of the harness's server scenarios are never selected at all (the 10 extension-tagged tasks-* scenarios never match a --spec-version filter and sit on the harness's pending list, so the active, draft, and 2026 legs all exclude them; server-sse-polling is pending-listed and removed at 2026-07-28, so no leg can reach it), and several checks inside running scenarios skip silently when a prerequisite is missing — all 5 SEP-2243 Mcp-Param-* server-validation checks skip because no fixture tool carries an x-mcp-header annotation, and the SEP-2575 subscriptions/listen checks skip on -32601.
This PR makes those surfaces visible as baselined known failures instead of silent exclusions, in three steps (one commit each):
Net effect on the baseline: zero waivers for implemented surfaces is unchanged — every entry added here tracks a not-yet-built surface that previously produced no signal at all.
Motivation and Context
A green conformance run currently overstates coverage: unimplemented surfaces (tasks extension, server-side Mcp-Param validation, subscriptions/listen) produce no signal because their scenarios are filtered or their checks skip upstream of the expected-failures mechanism. Visible baselined failures are strictly better — they show up in the run output, they can't rot silently across harness pin bumps, and the stale-entry rule turns the baseline into a forced burn-down list as features land.
How Has This Been Tested?
Replicated all six CI legs locally against the new pinned tarball (sha256-verified), exactly as CI invokes them: the four server legs and both client legs exit 0 with "Baseline check passed" —
Also verified the generated tools/list schema carries "x-mcp-header": "Region" through pydantic's json_schema_extra, and ran the discovery pass at the new pin before updating baselines to confirm server-stateless was the only newly-unexpected failure (no stale entries, no warnings anywhere).
Breaking Changes
None — CI configuration and a conformance fixture tool only.
Types of changes
Checklist
Additional context
AI Disclaimer