ci(conformance): bump harness to 0.2.0-alpha.5 preview - #2927
Conversation
Sorry, something went wrong.
…fcd39) CONFORMANCE_VERSION -> CONFORMANCE_PKG (full npx spec) so the same line accepts either a registry version or a tarball URL. Pinned to the immutable commit-SHA pkg.pr.new build of conformance#357 until alpha.5 publishes; the workflow comment carries an explicit TODO + a do-not-merge guard. Reconciliation: conformance#337 gated the SEP-2350 scope-union check to 2026-07-28+, so auth/scope-step-up now passes on the 2025 client leg and is removed from expected-failures.yml (it stays in the 2026 baseline).
… guard Adds a fetch-and-verify step to both jobs: when CONFORMANCE_PKG is a URL, download the tarball, check it against CONFORMANCE_PKG_SHA256, and re-point CONFORMANCE_PKG at the verified local file. No-op for registry specs (npm's own integrity applies).
There was a problem hiding this comment.
I didn't find any bugs — the fetch-and-verify step, sha256 pin, and expected-failures reconciliation all look consistent — but pinning main's CI to a pkg.pr.new preview tarball (vs. waiting for the alpha.5 npm release) is a maintainer policy call, so leaving this for a human.
Extended reasoning...Overview
This PR is CI-only: it switches the conformance harness pin in .github/workflows/conformance.yml from a registry version (CONFORMANCE_VERSION = 0.2.0-alpha.4) to a full package spec (CONFORMANCE_PKG) pointing at a pkg.pr.new preview tarball at commit 65fcd39, adds a fetch-and-verify step (sha256 check, then re-points to the local tarball) in both the server and client jobs, updates run-server.sh and the client steps to use $CONFORMANCE_PKG, and removes the now-passing auth/scope-step-up entry from the 2025 client baseline.
Security risks
The main consideration is supply-chain: CI now executes a tarball fetched from pkg.pr.new instead of the npm registry. The PR mitigates this reasonably — the URL uses the immutable commit-SHA form, the tarball is verified against a recorded sha256 before any npx invocation, the workflow keeps permissions: contents: read, and checkout uses persist-credentials: false. The fetch step is duplicated identically in both jobs and correctly no-ops for registry specs. I found no flaws in the implementation, but whether main's CI should depend on an unreleased preview build at all (rather than waiting for the 0.2.0-alpha.5 npm publish, per the in-file TODO) is a project policy decision.
Level of scrutiny
Moderate. The change doesn't touch SDK code, only the conformance CI harness, so worst case is CI noise rather than a shipped defect. Still, changes to what code gets downloaded and executed in CI deserve a human maintainer's sign-off rather than an automated approval.
Other factors
The author reports all five conformance legs passing locally against the pinned build, and the expected-failures edits are consistent with the stated behavior change in conformance#337 (scope-step-up removed only from the 2025 client baseline, retained for the 2026 leg). The header-comment updates in both expected-failures files match the new variable name. No prior reviews or unresolved comments are on the PR.
Sorry, something went wrong.
|
This pull request is included in pre-release v2.0.0a3 |
Sorry, something went wrong.
Bumps the conformance harness pin to the 0.2.0-alpha.5 preview build so the staging branch can assert the post-#2907 error codes (-32020..-32022) once the server-stateless / header-validation work starts emitting them.
Part of #2891. Originally targeted the v2-2026-07-28 staging branch (#2917), retargeted to main after #2917 merged.
Motivation and Context
#2912 noted that the published @modelcontextprotocol/conformance@0.2.0-alpha.4 predates conformance#353 (the renumber). conformance#357 is the alpha.5 release PR that carries it; this PR pins to its pkg.pr.new preview build at commit 65fcd39 so the staging branch runs against the renumbered harness without waiting for the npm release.
What changed
How Has This Been Tested?
All 5 conformance legs run locally against the 65fcd39 build:
No new scenarios appeared (the SEP-2663 tasks scenarios from conformance#262 and the server/discover mocks from conformance#347 are not in active/draft/all at this build).
Breaking Changes
None.
Types of changes
Checklist
AI Disclaimer