ci: Add weekly flaky test detector workflow by sl0thentr0py · Pull Request #6484 · getsentry/sentry-python · GitHub

Summary

Adds a weekly scheduled workflow (.github/workflows/flaky-test-detector.yml) that runs every Wednesday at 08:00 UTC (also manually via workflow_dispatch). It:

(shell step) Inspects the last ~30 test.yml / ci.yml runs on master via gh run list / gh run view --log-failed and writes the failed logs to ./ci-logs/.
(Claude step) Classifies genuinely flaky tests (intermittent failures, timing/ordering/network/global-state signals) vs. real regressions and infra noise — capped at the 5 clearest, ranked by impact — and writes a summary to flaky-issue-body.md.
(shell step) Opens one summary issue from that file listing each flaky test with its node ID, evidence (run IDs / failure frequency), a one-line root cause, and a short suggested fix.

It does not edit code or open a PR; the output is purely an informational issue for humans to triage.

Security

CI failure logs contain tracebacks and stdout controlled by whoever landed the commit, so they're untrusted input. The "treat logs as data" prompt is treated as defeatable; the real protections are mechanical and keep the log-reading agent away from any credentialed write channel:

The three steps are kept separate on purpose. Log fetching (step 1) and issue creation (step 3) are plain non-LLM shell steps; only they touch a token.
The Claude step has no shell and no write token. Its --allowedTools is Read,Glob,Grep,Write,TodoWrite — no Bash. With no subprocess and no network tool it cannot run gh/curl/printenv, so a prompt injection in the logs cannot exfiltrate ANTHROPIC_API_KEY or GITHUB_TOKEN, nor create an issue directly. It can only read the pre-fetched logs and write the issue body to a file.
issues: write lives only in the final shell step, which never ingests untrusted log text. All other permissions are read-only.

This avoids the single-step gh pitfall: giving the untrusted-data agent a gh write channel is itself an exfiltration vector (gh issue create --body "$ANTHROPIC_API_KEY"), and the subprocess env-scrub feature can't fix it because it's all-or-nothing and would break gh auth.

Required setup before this runs

One repo/org secret is needed:

ANTHROPIC_API_KEY — Anthropic API key for the model.

The workflow uses the default GITHUB_TOKEN (scoped to contents: read, actions: read, issues: write) — no PAT required.

Optionally create a flaky-test label so the issues are grouped; the workflow falls back to no label if it doesn't exist.

Codecov Results 📊

✅ 88635 passed | ⏭️ 6007 skipped | Total: 94642 | Pass Rate: 93.65% | Execution Time: 298m 51s

📊 Comparison with Base Branch

Metric	Change
Total Tests	📈 +215
Passed Tests	📈 +213
Failed Tests	—
Skipped Tests	📈 +2

All tests are passing successfully.

✅ Patch coverage is 100.00%. Project has 2332 uncovered lines.
❌ Project coverage is 89.91%. Comparing base (base) to head (head).

Coverage diff

@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
- Coverage    89.92%    89.91%    -0.01%
==========================================
  Files          191       191         —
  Lines        23115     23115         —
  Branches      7952      7952         —
==========================================
+ Hits         20785     20783        -2
- Misses        2330      2332        +2
- Partials      1307      1309        +2

Generated by Codecov Action

Add a scheduled GitHub Action that runs every Wednesday at 08:00 UTC and asks Claude Code to inspect recent master CI runs for flaky tests, then open a single combined draft PR with proposed fixes. Exits cleanly when no flakes are found. Uses a FLAKY_BOT_TOKEN PAT so the draft PR triggers CI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This overall looks good. The one question I have is whether the references to "master" outside of any backticks/quotes will lead to potential unexpected behaviour, but otherwise don't have any concerns about the prompt

think the master ref is fine

GITHUB_TOKEN with issues:write is passed to the Claude action, contradicting the stated security model

The security comment on lines 19–22 claims "The Claude step gets NO … write token," but github_token: ${{ github.token }} is explicitly passed to the action on line 112 — and that token carries issues: write (declared at the workflow level on line 43). If prompt-injection succeeds, or if a future version of claude-code-action exposes the token through any built-in GitHub tool, the LLM could open or modify issues directly, bypassing the isolation that step C is designed to enforce. Consider omitting github_token from the Claude step entirely, or binding it to a separate fine-grained token scoped to only contents: read/actions: read.

Evidence

Workflow-level permissions (line 43) grant issues: write to GITHUB_TOKEN.
Line 112 passes that same token as github_token to anthropics/claude-code-action.
The security comment at lines 19–22 explicitly states "The Claude step gets NO Bash tool and NO write token" — the token claim is factually incorrect.
allowedTools (line 115) does not include a GitHub API tool, but the token is still present in the action's environment and available to any built-in capability the action uses or adds in future releases.
Step C on line 160 is the intended sole holder of write capability; passing the write token to step B weakens that isolation boundary.

Identified by Warden code-review · RPC-6G7

Cursor Bugbot has reviewed your changes and found 1 potential issue.

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3c8224a. Configure here.

Read tool can exfiltrate secrets via issue body

High Severity

The security model claims Claude "cannot exfiltrate ANTHROPIC_API_KEY or GITHUB_TOKEN" because it lacks Bash and network tools. However, the Read tool can access /proc/self/environ (a known, documented Claude Code issue with multiple open bugs), which contains both secrets as environment variables. A prompt injection in CI logs could instruct Claude to read that file and embed the extracted ANTHROPIC_API_KEY into flaky-issue-body.md via the Write tool. Step C then publishes that file content as a public GitHub issue, completing the exfiltration without any shell or network access.

Additional Locations (1)

.github/workflows/flaky-test-detector.yml#L206-L215

Reviewed by Cursor Bugbot for commit 3c8224a. Configure here.

well we just want to try this for now, seems like a known thing

## Summary Adds a weekly scheduled workflow (`.github/workflows/flaky-test-detector.yml`) that runs **every Wednesday at 08:00 UTC** (also manually via `workflow_dispatch`). It: 1. **(shell step)** Inspects the last ~30 `test.yml` / `ci.yml` runs on `master` via `gh run list` / `gh run view --log-failed` and writes the failed logs to `./ci-logs/`. 2. **(Claude step)** Classifies genuinely flaky tests (intermittent failures, timing/ordering/network/global-state signals) vs. real regressions and infra noise — capped at the 5 clearest, ranked by impact — and writes a summary to `flaky-issue-body.md`. 3. **(shell step)** Opens **one summary issue** from that file listing each flaky test with its node ID, evidence (run IDs / failure frequency), a one-line root cause, and a short suggested fix. It does **not** edit code or open a PR; the output is purely an informational issue for humans to triage. ## Security CI failure logs contain tracebacks and stdout controlled by whoever landed the commit, so they're untrusted input. The "treat logs as data" prompt is treated as defeatable; the real protections are mechanical and keep the **log-reading agent away from any credentialed write channel**: - **The three steps are kept separate on purpose.** Log fetching (step 1) and issue creation (step 3) are plain non-LLM shell steps; only they touch a token. - **The Claude step has no shell and no write token.** Its `--allowedTools` is `Read,Glob,Grep,Write,TodoWrite` — no `Bash`. With no subprocess and no network tool it cannot run `gh`/`curl`/`printenv`, so a prompt injection in the logs cannot exfiltrate `ANTHROPIC_API_KEY` or `GITHUB_TOKEN`, nor create an issue directly. It can only read the pre-fetched logs and write the issue body to a file. - **`issues: write` lives only in the final shell step**, which never ingests untrusted log text. All other permissions are read-only. This avoids the single-step `gh` pitfall: giving the untrusted-data agent a `gh` write channel is itself an exfiltration vector (`gh issue create --body "$ANTHROPIC_API_KEY"`), and the subprocess env-scrub feature can't fix it because it's all-or-nothing and would break `gh` auth. ## Required setup before this runs One repo/org secret is needed: - **`ANTHROPIC_API_KEY`** — Anthropic API key for the model. The workflow uses the default `GITHUB_TOKEN` (scoped to `contents: read`, `actions: read`, `issues: write`) — no PAT required. Optionally create a `flaky-test` label so the issues are grouped; the workflow falls back to no label if it doesn't exist. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

sentry-warden Bot reviewed Jun 2, 2026

View reviewed changes