ci: validate with --skip-helm-render to dodge ksail#5362 render flake - #2270
ci: validate with --skip-helm-render to dodge ksail#5362 render flake#2270devantler merged 1 commit into
Conversation
Sorry, something went wrong.
ksail's in-process Helm render (added v7.66.0, #5344) is non-deterministic: concurrent renders share process-global Helm caches and corrupt the rendered stream, so `ksail workload validate` fails with a different random YAML parse error each run. #5362's closed fix (#5364/#5366) is incomplete — 7.77/7.78 still flake; only <=7.65 (which predates in-process render) is clean. Pass --skip-helm-render to the validate step (deterministic; validates HelmRelease CRs as-is — same coverage as the old 7.65.0 pin) rather than re-pin back 13 versions, and correct the stale Setup KSail comment that claimed the race was resolved upstream. Remove once #5362 is genuinely fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
🎉 This PR is included in version 1.80.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Sorry, something went wrong.
Root cause is now definitively identified and the real fix is up. It is a buffer-aliasing data race in kubeconform (resource.FromStream hands out resources that alias the reused bufio.Scanner buffer, parsed concurrently with the producer's next Scan()), confirmed by the Go race detector — not ksail's in-process Helm render. The render path just produces >4 MB streams that trigger the scanner refill. Upstream fix (one line + -race regression test): yannh/kubeconform#363. Verified on this tree by building ksail against the patched kubeconform: 0 data races (was 5), 10/10 deterministic validate (was ~1/5). Tracking on ksail#5362. This --skip-helm-render interim stays until ksail bumps kubeconform past the yannh/kubeconform#363 release; at that point this commit can be reverted to restore rendered-chart validation. |
Sorry, something went wrong.
Makes 🧪 Validate Manifests deterministic by passing --skip-helm-render, working around the still-open ksail#5362. Unblocks #2268 (crossview standalone) and every other platform PR.
Root cause
ksail's in-process Helm render (added v7.66.0, #5344) is non-deterministic: validate renders kustomizations concurrently (validationConcurrency=5) and concurrent helm.TemplateChart calls share process-global Helm on-disk caches → the rendered stream gets corrupted → a different kustomization fails with a different YAML parse error every run (always in the HelmRelease-dense */controllers group).
The CI pin was lifted 7.65.0 → latest in #2265 believing #5371 fixed this — but that closed the scan score-swing; the validate corruption is #5362, whose closed fix (#5364/#5366 "serialize") is incomplete.
Determinism matrix (prod overlay, identical tree each run):
Fix
Pass --skip-helm-render to both validate invocations (deterministic; validates HelmRelease CRs as-is — the same coverage as the old 7.65.0 pin, which predated in-process render anyway), instead of re-pinning back 13 versions and losing other fixes. Also corrects the stale Setup KSail comment that claimed the race was resolved upstream.
scan is left on the rendered path (it's score-gated with margin, and its threshold is calibrated against the rendered output).
Caveat (self-CI)
GitHub runs pull_request workflows from the base branch, so this PR's own Validate Manifests check still uses the old (flaky) command and may go red on the #5362 flake — merge on the diff (or re-run). Once merged, main's validate is deterministic and crossview #2268 can be re-run green.
Follow-up
Reopening #5362 upstream with this repro, and attempting the real fix (isolate each render worker's Helm caches) so --skip-helm-render can later be removed.
🤖 Generated with Claude Code