feat: scope evaluators to the current project on create - #274
Draft
quinn-galileo wants to merge 31 commits into
Draft
quinn-galileo wants to merge 31 commits into
quinn-galileo wants to merge 31 commits into
Conversation
Copy link
Copy Markdown
Collaborator
Sorry, something went wrong.
Add project_ids and is_global to the evaluator create paths and default the scope to the current project, so non-admins can create evaluators when metric RBAC is on. Surface is_global and scope_projects on Evaluator. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…roject fixture Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Resolve conflicts keeping both sides and thread scope through the new decision evaluator create path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
Sorry, something went wrong.
- Resolve the CHANGELOG merge conflict, keeping main's entries and ours. - Reject a bare string for project_ids instead of splitting it into characters. - DecisionEvaluator.create no longer marks the evaluator failed on a scope ValidationError, matching the LLM and code paths. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
quinn-galileo
commented
Oct 19, 2026
quinn-galileo
left a comment
Copy link
Copy Markdown
Collaborator
Author
There was a problem hiding this comment.
Sorry, something went wrong.
- Store constructor project_ids/is_global privately; public is_global and scope_projects now only reflect the API response (None/[] until populated). - An Unset is_global in the response maps to None. - Explain a missing default project in the NotFoundError message, normalise a whitespace-only context project, and document scope errors on create(). - Add regression tests for create() on a fetched evaluator and tighten tests. - Correct the CHANGELOG validation wording and note the project must exist. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
quinn-galileo
commented
Oct 19, 2026
quinn-galileo
left a comment
Copy link
Copy Markdown
Collaborator
Author
There was a problem hiding this comment.
Sorry, something went wrong.
The Client API spec now has a single ScorerName schema, so the CoreScorerName retitle no longer applies and the duplicate-value dedupe moves to ScorerName. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Re-imported openapi.yaml from the production Client API spec and regenerated src/splunk_ao/resources with the repo's generator scripts. No hand edits. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ScorerConfig is now RuntimeScorerConfig in the spec, and ProjectCreate no longer has create_example_templates. Both broke at import or call time without a type-check error, because mypy skips the generated package. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Also switch the non-gen_ai delete test to ProjectType.PROTECT, since LLM_MONITOR is no longer a project type. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… breaking Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Re-add vegas_gateway to IntegrationProvider and LLMIntegration and task types 7, 9 and 12 to TaskType, as the StepType patch does, and fail with a clear message if the ScorerName enum moves. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Generated output only. The remaining openapi.yaml churn is the server's unordered applicable_types lists in column examples. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Experiments().list parses a row with task_type 7. The project tests gain Given/When/Then comments, and the create test is renamed for what it guards. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
List every removed field, name removed members, describe the ScorerName rename and the RuntimeScorerConfig construction change, and drop the restored enum values from the removals. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A += on a renamed schema would create a stub and exit 0, silently dropping the restored values. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…h note Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
quinn-galileo
changed the base branch from
main
to
chore/sao-9342-t0-regenerate-client
October 20, 2026 15:41
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…enerate-client # Conflicts: # CHANGELOG.md
Adds GET /projects/{project_id}/stages/{stage_id}/rulesets with its
StageRulesetsResponse model, and the optional code_sha256 field on
BaseScorerVersionDB plus code_sha256, request_id and validation_task_id on
BaseScorerVersionResponse. Other changes are generated docstrings only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge the refreshed regenerated-client parent; drop the decision-evaluator entries now released in 0.6.0 and raise the minimum version to 0.7.0. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 task
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
With metric RBAC on (now the server default), a create request that carries neither is_global nor project_ids is treated as global, and only org admins may create global metrics. The SDK never sent scope, so every non-admin evaluator create returned 403. This PR adds scope to every SDK evaluator create path and defaults it to the user's current project. Ticket: SAO-9342.
What changed
New project_ids: list[str] | None and is_global: bool | None on Evaluators.create_custom_llm_evaluator, Evaluators.create_decision_evaluator, the module-level create_custom_llm_evaluator / create_decision_evaluator (trailing keyword params, positional order unchanged), and keyword-only on LlmEvaluator, CodeEvaluator and DecisionEvaluator. Evaluator.is_global and Evaluator.scope_projects report only what the API returned: None / [] until the evaluator is populated by create(), get(), list() or refresh(). The scope you pass to a constructor is kept privately and read only by create(), so calling create() on an evaluator fetched with get() resolves scope from the current project, not from the fetched server values.
The scope sent on create, in order:
The current project must already exist. Unlike logging, creating an evaluator does not create it, and the NotFoundError says so and points at project_ids=[...] or is_global=True.
Behaviour change for admins: an admin with a current project set now gets a project-scoped evaluator. Pass is_global=True for a global one. The CHANGELOG says so, and names 0.7.0 as the minimum splunk-ao version for creating project-scoped evaluators.
Where to start reading
Testing
poetry run pytest tests/test_evaluators.py tests/test_evaluator.py tests/test_decision_scorers.py -n 0 (254 passed, including tests/test_projects.py) and poetry run invoke type-check (clean). All network is mocked. Tests cover the default, each precedence step, explicit override, global, nothing-set, every validation error, not-found, constructor forwarding, and population from the response, for the LLM, code and decision paths.
Related PRs
🤖 Generated with Claude Code