Repository navigation
SEP-2484: Require Conformance Tests for Standards Track SEPs to Reach Final Status - #2484
mcp-commander[bot] merged 7 commits into
Conversation
Sorry, something went wrong.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
Sorry, something went wrong.
| **What's required:** | ||
|
|
||
| - A conformance scenario tagged with the SEP number, targeting the `draft` spec version | ||
| - A structured traceability file (`sep-NNNN.yaml`) mapping each MUST/MUST NOT in the SEP's Specification section to either a check ID or a documented exclusion with a linked tracking issue |
There was a problem hiding this comment.
Should SHOULD/SHOULD NOT be mapped to a "warning" (not "failure") status?(Maybe a bigger question for the conformance suite at large)
Sorry, something went wrong.
There was a problem hiding this comment.
Yea, that is how we've been representing them in conformance. I left it out mapping them as a requirement here initially to keep the burden small, but reflecting more now, it makes sense to require mapping them too. They have a burden on implementors, so having it be a burden on SEP writers to test makes sense.
Sorry, something went wrong.
|
|
||
| Exclusions come in two flavors. **Framework gaps** (the behavior is observable but the framework can't express it yet) should link a tracking `issue`. **Not protocol-observable** (the requirement governs client rendering, implementation internals, or similar) needs only the `excluded` reason. A SEP whose requirements are all the second kind is exempt and doesn't need a scenario at all. | ||
|
|
||
| The sponsor verifies the traceability file is complete: every MUST and MUST NOT (and RFC 2119 equivalents: SHALL, REQUIRED) in the SEP's Specification section has a row. SHOULD and MAY requirements do not need rows, although it is recommended to include SHOULD's a WARNING check. The sponsor does not review test code; that is the conformance repository's normal PR review. What counts as a normative requirement is the sponsor's call. |
There was a problem hiding this comment.
| The sponsor verifies the traceability file is complete: every MUST and MUST NOT (and RFC 2119 equivalents: SHALL, REQUIRED) in the SEP's Specification section has a row. SHOULD and MAY requirements do not need rows, although it is recommended to include SHOULD's a WARNING check. The sponsor does not review test code; that is the conformance repository's normal PR review. What counts as a normative requirement is the sponsor's call. | |
| The sponsor verifies the traceability file is complete: every MUST and MUST NOT (and RFC 2119 equivalents: SHALL, REQUIRED) in the SEP's Specification section has a row. SHOULD and MAY requirements do not need rows, although it is recommended to include SHOULDs as a WARNING check. The sponsor does not review test code; that is the conformance repository's normal PR review. What counts as a normative requirement is the sponsor's call. |
Sorry, something went wrong.
There was a problem hiding this comment.
I'd be in favor of upgrading recommended to include SHOULDs -> required, since many protocol behaviors end up being SHOULDs. But I am OK leaving this up to the discretion of the author or sponsor.
Sorry, something went wrong.
|
I'm supportive of this! Will be a great way to systematize the spec and introducing it into the SEP process is the right place to start doing it. |
Sorry, something went wrong.
State Transition: proposal → draftThis SEP has been transitioned from proposal to draft. @pcarleton has been assigned as the sponsor for this SEP. This is an automated message from the SEP lifecycle bot. |
Sorry, something went wrong.
|
I am in favor of this. My main worry, similar with reference implementations, is tracking that SEPs have it. |
Sorry, something went wrong.
Adds a conformance test requirement to the Accepted → Final transition for Standards Track SEPs.
Summary
Before a Standards Track SEP that changes observable protocol behavior can be marked Final:
Process and Informational SEPs are exempt, as are Standards Track SEPs with no observable protocol behavior.
Why
SEP-1730's SDK tiering depends on conformance tests, but nothing keeps the suite synchronized with the spec. This ties test creation to the SEP lifecycle so the suite grows exactly as fast as the spec does.
Supersedes
SEP-1627 (Conformance Testing)
Changes