…2-E1) (#152)
* feat(regression): sanitize + validate published regression content (S2-E1)
Regression reports (a Markdown writeup + SVG residual plot + JSON fact-box per
fit) are dataset-authored content served to anonymous users, but the build copied
the SVG/JSON verbatim and rendered the Markdown without sanitization. S2-E1 adds
the security boundary and the dataset-validation checks for that content, without
yet moving the files to the dataset (that is S2-E2/E3) — so it is backward-
compatible and hardens the existing docs/regression path in place.
New `src/kayak/web/regression.py` (used by both build and validate-dataset):
- render_markdown_safe: filter maintainer sections → python-markdown → nh3.clean
with an explicit tag/attr allowlist, http(s)-only URL schemes, link rel, and a
local-only img@src filter (strips raw HTML, event handlers, javascript:/data:,
external image sources).
- validate_svg: defusedxml parse (no DTD/entities/external) + a strict
element/attribute allowlist, then re-serialize from the validated tree (never
serve verbatim). Rejects script/foreignObject/use/image/event-handlers/href/
style, foreign-namespace elements, backslash CSS-escapes, and any resource
function (url()/image()/element(), case-insensitive) that isn't a same-document
url(#id). SVG is treated as active content (the browser renders /static/.../
<slug>.svg as a document outside the page CSP).
- validate_json_sidecar: size cap + reject NaN/Infinity + object-with-slug shape
(accepts both the pair-linear and lead/lag schemas).
Build (web/build/deploy.py): _deploy_regression_artifacts now renders/validates
through the module (re-serialized SVG, no shutil.copy2) and is fail-closed.
validate-dataset: new _check_regression ties every non-empty
calc_expression.provenance_slug to its {md,svg,json} triple, follows md links to
require companion reports + referenced sidecars (order-independent; code-fence-
aware), runs the content through the sanitizers (reject nonconforming), and warns
on orphan reports via a non-fatal warnings channel that leaves validate_dataset()'s
list[str] error contract intact. No slugs + no regression/ dir = none configured.
Deps: add nh3 + defusedxml (regenerated uv.lock; mypy override for defusedxml).
Fixture: a declared report + a link-reachable lead/lag companion under
tests/fixtures/dataset/regression/ (authored by build_dataset_fixture.py so a
regen reproduces it), exercising the slug↔report check, the maintainer-section
drop, the reference closure, and the SVG/JSON sanitizers. All 25 real reports
pass the new checks (move-verbatim invariant). Two rounds of adversarial review
(incl. headless-browser bypass testing) drove the SVG allowlist hardening.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address #152 review — deploy-gate brick, orphan-filename XSS, SVG PI/comment
1. [HIGH] Deploy-brick: deploy.sh runs `validate-dataset` against the real
kayak_data on every deploy, and the new _check_regression made the 17 declared
provenance_slugs require a regression/ directory that won't exist until the S2
file-move (D1). Deploying E1 alone would have failed the gate before
migrate/sync. Fix: the regression/ directory is the dataset's opt-in — while it
is absent the reports still live engine-side (docs/regression), so declared
slugs yield a non-fatal WARNING, not an error. Once the dir exists the check is
fully enforced (the fixture exercises that path). Real dataset now validates OK
(1 warning); fixture still fully enforced.
2. [Low-Med] Orphan-filename XSS: the generated report page interpolated the file
stem into <title> unescaped, and the build globbed every *.md regardless of
slug — so an orphan file whose NAME carried HTML metacharacters could emit a
stored-XSS page (validate-dataset only warns on orphans). Fix: the build now
skips any regression file whose stem isn't a safe slug (new
kayak.web.regression.is_safe_slug, shared with the validator's charset), and
html.escape()s the title as defence-in-depth. (Also renamed the shadowing local
`html` template var → `page`, and added encoding= to the page write.)
3. [Low] validate_svg now rejects XML comments and processing instructions
explicitly (e.g. `<?xml-stylesheet href="javascript:…"?>`), so the
reject-nonconforming contract no longer rests on ElementTree silently dropping
them. A single leading `<?xml …?>` declaration is still allowed.
Also merged current main to pick up #151's deploy/SETUP.md (the branch predated it,
so the diff had appeared to revert the healthchecks.io SA-teardown-C note).
All 25 real reports still pass; full gate green (1536 tests, wheel-smoke).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: reject a leading <?xml-stylesheet?> PI in validate_svg (#152 re-review)
The PI/comment guard treated any leading `<?xml…` as the XML declaration via
`startswith("<?xml")`, which also matched a leading `<?xml-stylesheet …?>` PI —
so that one construct (the very example the guard cites) was stripped, not
flagged. Distinguish the real declaration by the whitespace the XML spec requires
after the `xml` target (`re.match(r"<\?xml\s", body)`); the PI target
`xml-stylesheet` has a `-`, so it falls through to the reject. Not exploitable
(ElementTree drops the PI and the build serves the re-serialized SVG), but it
closes the reject-nonconforming contract over its own example.
Tests: leading xml-stylesheet PI now rejected; a genuine `<?xml version?>`
declaration still accepted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #150 (SA-teardown-B). The retired kayak-metadata-snapshot.service pinged ${HC_METADATA_SNAPSHOT} on success, so once the unit is removed its healthchecks.io check stops receiving pings and fires a false "down" alert after its grace period. The deploy/SETUP.md §8 upgrade note covered the systemd timer prune and the .env line removal but omitted pausing the HC.io check — caught during the SA-teardown-C operator pass.
This adds the "pause or remove the metadata-snapshot check on healthchecks.io" step (flagged operator-only — it lives in the HC.io account, not on the host) and rewords the trailing line so branch protection reads as the final runbook action rather than "remaining" (it's since been enabled).
Docs-only. Full gate green locally (ruff/format/mypy, pytest -m "not slow" 1406 passed, wheel-smoke, diff-check).
🤖 Generated with Claude Code