Repository navigation
Parser occasionally fails with "The specified key does not exist" due to early startup #3139
Description
Activity
Hi @cnsino
thanks for ht edetailed report.
I've never seen this behaviour and I'm running thousands of scans daily, so I'm guessing there is something with your setup which is causing this. The operator is already checking that the job are succeeded and should only start the parsers if that is the case.
So anything addional you can provide would help. operator logs, example scans, scantypes, more details on the cluster setup etc.
@J12934 error logs for operator
2025-07-20T00:02:56Z ERROR controllers.execution.Scan Failed to run Scan Finalizer {"error": "Operation cannot be fulfilled on scans.execution.securecodebox.io \"test-20250707074054-1752624000\": StorageError: invalid object, Code: 4, Key: /registry/execution.securecodebox.io/scans/default/test-20250707074054-1752624000, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4bf618d9-5160-4562-88c8-f02e78e56e85, UID in object meta: "} github.com/secureCodeBox/secureCodeBox/operator/controllers/execution/scans.(*ScanReconciler).Reconcile /workspace/controllers/execution/scans/scan_controller.go:92 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:118 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:328 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249 2025-07-20T00:02:56Z ERROR Reconciler error {"controller": "scan", "controllerGroup": "execution.securecodebox.io", "controllerKind": "Scan", "Scan": {"name":"test-20250707074054-1752624000","namespace":"default"}, "namespace": "default", "name": "test-20250707074054-1752624000", "reconcileID": "77a36019-1dca-459e-81ae-6be7ddc56d32", "error": "Operation cannot be fulfilled on scans.execution.securecodebox.io \"test-20250707074054-1752624000\": StorageError: invalid object, Code: 4, Key: /registry/execution.securecodebox.io/scans/default/test-20250707074054-1752624000, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4bf618d9-5160-4562-88c8-f02e78e56e85, UID in object meta: "} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:341 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:288 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.1/pkg/internal/controller/controller.go:249 2025-07-20T01:00:00Z INFO KubeAPIWarningLogger metadata.finalizers: "s3.storage.securecodebox.io": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writersI think i've now also observed the same behaviour, didn't se it before because the job retries made the parser succeed in the end. But it were just two cases in ~5.000 scans, is this happening regularly for you?
No idea why this happens.
If somebody knows of a combination of job fields which lets theis function consider jobs as completed even though they are still running / incomplete, would be apprichiated: https://github.com/secureCodeBox/secureCodeBox/blob/main/operator/controllers/execution/scans/job.go#L65C26-L65C47Occurs occasionally, about 2 to 5 times out of 20 scans per week
- added a commit that references this issue on Sep 3, 2026
🐞 Bug report
Describe the bug
In SecureCodeBox, we observed an intermittent issue where the parser job fails with the following error:
After investigating pod creation timestamps, we found that the parser and scanner pods were created at nearly the same time, indicating that the parser did not wait for the scanner job to complete before starting — which contradicts the expected execution order.
Steps To Reproduce
This is difficult to reproduce consistently, but occurs under the following general scenario:
Expected behavior
Parser job should only start after the scanner job has completed successfully, and after the scan result file is fully uploaded to MinIO.
System (please complete the following information):
Screenshots / Logs
Parser pod logs:
data: '<?xml version="1.0" encoding="UTF-8"?>\n' + '<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Pod creation timestamps:
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.metadata.creationTimestamp}{"\t"}{.status.containerStatuses[0].state.terminated.finishedAt}{"\n"}{end}' default parse-test-20250707074054-1752796800-vndjq-ht59g Failed 2025-07-18T00:00:39Z 2025-07-18T00:00:43Z default parse-test-20250707074054-1752796800-vndjq-lthz7 Failed 2025-07-18T00:00:00Z 2025-07-18T00:00:05Z default parse-test-20250707074054-1752796800-vndjq-mb88s Failed 2025-07-18T00:01:23Z 2025-07-18T00:01:27Z default parse-test-20250707074054-1752796800-vndjq-xl27p Failed 2025-07-18T00:00:15Z 2025-07-18T00:00:19Z default parse-test-20250707074152-1752541200-wqsl4-mbxm5 Succeeded 2025-07-15T01:03:08Z 2025-07-15T01:03:13Z default parse-test-20250707074152-1752627600-jslms-6g885 Succeeded 2025-07-16T01:03:15Z 2025-07-16T01:03:21Z default parse-test-20250707074152-1752714000-6vd8z-z42hp Succeeded 2025-07-17T01:03:08Z 2025-07-17T01:03:13Z default parse-test-20250707074152-1752800400-q6kpn-6sdch Failed 2025-07-18T01:00:14Z 2025-07-18T01:00:18Z default parse-test-20250707074152-1752800400-q6kpn-cdvpb Failed 2025-07-18T01:00:00Z 2025-07-18T01:00:04Z default parse-test-20250707074152-1752800400-q6kpn-cvgh6 Failed 2025-07-18T01:00:38Z 2025-07-18T01:00:42Z default parse-test-20250707074152-1752800400-q6kpn-kswsl Failed 2025-07-18T01:01:22Z 2025-07-18T01:01:26Z default scan-test-20250707074054-1752624000-vzvc9-rh4qc Succeeded 2025-07-16T00:00:00Z 2025-07-16T00:02:42Z default scan-test-20250707074054-1752710400-5mcnt-pd8lq Succeeded 2025-07-17T00:00:00Z 2025-07-17T00:02:52Z default scan-test-20250707074054-1752796800-dm2tm-25rw7 Succeeded 2025-07-18T00:00:00Z 2025-07-18T00:02:39Z default scan-test-20250707074152-1752627600-7x5zh-6pw7n Succeeded 2025-07-16T01:00:00Z 2025-07-16T01:03:12Z default scan-test-20250707074152-1752714000-phvgz-4bt4n Succeeded 2025-07-17T01:00:00Z 2025-07-17T01:03:05Z default scan-test-20250707074152-1752800400-cs8cp-jtjwm Succeeded 2025-07-18T01:00:00Z 2025-07-18T01:03:10ZAdditional context
This suggests that the job controller may incorrectly trigger the parser job before the scanner has completed, or before the result file has been fully uploaded to MinIO.
Possible solutions or mitigation strategies:
Let us know if a reproduction setup or further logs are needed. Thank you!