Repository navigation
SEP-3405: Conformance-Driven SDK Tiers - #3405
felixweinberger wants to merge 8 commits into
Conversation
Sorry, something went wrong.
Draft SEP that supersedes SEP-1730. An SDK's tier follows only from the conformance scenarios it passes, on stated spec versions, by stated dates.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Sorry, something went wrong.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
Sorry, something went wrong.
The SEP states that Tier 2 has exclusions and who can change them. The conformance repository is the source of truth for the list.
Tightens wording, removes repeated statements, uses one term per thing and names who confirms a tier change. No rule changes intended.
|
|
||
| ## Abstract | ||
|
|
||
| An SDK's tier depends only on which conformance scenarios it passes, on which spec versions, and by when. Process requirements no longer affect a tier. Nobody applies for a tier. An automated run computes every tier. An SDK Working Group maintainer confirms each tier change before it is published. |
There was a problem hiding this comment.
| An SDK's tier depends only on which conformance scenarios it passes, on which spec versions, and by when. Process requirements no longer affect a tier. Nobody applies for a tier. An automated run computes every tier. An SDK Working Group maintainer confirms each tier change before it is published. | |
| An SDK's tier depends only on which conformance scenarios it passes, on which spec versions, and by when. Process requirements no longer affect a tier. Nobody applies for a tier. An automated run computes every tier. An SDK Working Group maintainer confirms each time an SDK's tier changes before it is published. |
trying to clarify whether this sentence is about confirming an SDK's tier, or confirming a "tier" itself (i.e. any change to conformance tests)
Sorry, something went wrong.
| | **Tier 3** | No minimum | No minimum | | ||
| | **Process requirements** | Triage time, critical bug resolution, labels, documentation, roadmap, dependency policy | Not part of a tier | | ||
| | **How a tier is assigned** | An application issue, reviewed by the SDK Working Group | Computed by an automated run. An SDK Working Group maintainer confirms each tier change. | | ||
| | **Losing a tier** | After four weeks of failing tests, or two months of unaddressed issues | When the automated run shows the SDK no longer passes what the tier requires. No waiting period. | |
There was a problem hiding this comment.
I think we'll want a few more words on process here, otherwise a strict interpretation could be very aggressive (e.g. day after spec release, immediately downgrade SDK's w/o discussing with them a remediation plan, or oscillating back and forth depending if a conformance regression makes it to main by mistake)
Sorry, something went wrong.
| ### Timing | ||
|
|
||
| 1. A version of the conformance suite ships with the release candidate. The suite can still change after that. | ||
| 2. The lock date is at least 14 days before the spec release date. It is announced with the release candidate. |
There was a problem hiding this comment.
#3398 specifies some names for dates that I think we should leverage here:
Soft-Freeze: 2.5 months before
Hard Freeze: 1 month before
if we want another date, it probably makes sense to define it over there and reference it here.
I think hard-freeze is probably good enough for this.
The step (4) stuff I think can stay the same, just referencing the hard-freeze date.
Sorry, something went wrong.
| 4. A scenario published or changed after the lock date is still required. For Tier 1 it gets its own due date. The Core Maintainers set that date. It is the same for every SDK and at least 14 days after the scenario is published or changed. Before that date the scenario runs and is reported but does not count. For Tier 2 the scenario is due three months after the spec release date. | ||
| 5. After the spec release date, no scenario is added to that spec version's required set and none is made stricter. A scenario that is wrong can be relaxed or withdrawn at any time. | ||
|
|
||
| ### Computing and publishing tiers |
There was a problem hiding this comment.
I think this section goes into a messy middle level of detail where it is specific enough that we might do a thing that solves the problem that doesn't match what we say here, but is fairly unspecific because we haven't tried it yet.
A few options on this section:
- Cut it, might be fine w/o it
- Keep it looser, reference a "automated process agreed in SDK working group"
- Get into deeper detail if we think we're ready for that
Sorry, something went wrong.
| ### Listing | ||
|
|
||
| 1. To be listed on the [SDKs page](https://modelcontextprotocol.io/docs/sdk), an SDK needs a published security policy with a contact and at least one stable release. | ||
| 2. The SDKs page lists official SDKs and community SDKs in separate sections. Official SDKs are those in the `modelcontextprotocol` organization. Tiers are computed the same way for both. A community listing is not an endorsement. |
There was a problem hiding this comment.
I'd cut the idea of community SDK's from this SEP.
I do think we want some way for community SDK's and clients and servers to declare their conformance, but I don't want to commit to it in the SEP.
As soon as we declare community SDK's, then we need to develop a criteria for what a community SDK is, and risk devolving into a weird advertising board for side projects that happen to pass conformance
Sorry, something went wrong.
| ### No longer part of a tier | ||
|
|
||
| 1. Issue triage time, critical bug resolution time, required labels, documentation, roadmap, dependency policy and the per-release timeline for new protocol features no longer affect a tier. | ||
| 2. SEP-2596 makes marking deprecated features in the SDK's API a condition of Tier 1 status. This SEP removes that condition. Official SDKs are still expected to mark deprecated features. |
There was a problem hiding this comment.
I forgot about this... I am broadly aligned that if we can't easily test it with conformance, let's cut it from tiering
Sorry, something went wrong.
Draft SEP. Supersedes SEP-1730.
An SDK's tier is decided only by the conformance scenarios it passes: which scenarios, on which spec versions, and by when. Process requirements no longer affect a tier. An automated run computes every tier.
The SEP opens with a table of what changes from today. Open questions are listed at the end.
Work in progress. Not ready for review.