Summary
The Gerrit Required Verify workflow in this repository fails routinely, and
each failure emails the change author and the releng/collab-it lists. This is
not a recent regression — it has been steady for at least the last 35 days.
Sample: the last 200 failed runs (2026-06-30 → 2026-08-04), against 200
consecutive runs over 2026-07-27 → 2026-08-04 for the rate.
| Measure |
Value |
| Failure rate (last 200 runs) |
44 failed, 6 cancelled, 150 succeeded (22%) |
| Failure notifications |
~5.7 per day |
| Attributable to cause A below |
182 of 200 failures (91%) |
| Attributable to cause B below |
16 of 200 failures (8%) |
| Genuine INFO.yaml validation failures |
1 of 200 (working as intended) |
Neither dominant cause is a fault in this repository's own content. Both have
upstream fixes, linked below.
Cause A — verify is dispatched for private Gerrit changes (91%)
lfit/releng-gerrit_to_platform dispatches on patchset-created regardless of
whether the change is private. The Gerrit hook runs server-side with full
visibility; the fdio.required.gha account that has to clear and set the vote
does not have it. Both the prepare and vote jobs then die with:
error: fatal: Not found: 46421
fatal: one or more reviews failed; review output above
Worked example — change 46421:
| Gerrit event |
Time (UTC) |
Run |
Result |
| Uploaded patch set 1 (private) |
07:36:49 |
30888530294 |
❌ Not found: 46421 |
| Uploaded patch set 2 (private) |
07:52:40 |
30889582764 |
❌ Not found: 46421 |
| Unset private |
07:55:34 |
30889787320 |
✅ SUCCESS |
Uploading private and publishing later is a normal VPP developer workflow — 9 of
the last 60 vpp master changes did it — and every patchset pushed while
private produces a failure email.
Granting the service account visibility of private changes would not fix this:
private change refs are not replicated to GitHub, so the verify job could not
fetch the change anyway.
Importantly, unsetting private fires a fresh dispatch on its own, so suppressing
the earlier dispatches costs no verification coverage.
Upstream fix: lfit/releng-gerrit_to_platform#116
Cause B — shallow checkout when the change has already merged (8%)
gerrit-compose-required-info-yaml-verify.yaml checks isolation with
git diff --name-only HEAD~1 over a fetch-depth: 1 checkout. For CSIT
oper-* cherry-picks, which are submitted pre-approved and merge within seconds
of the dispatch, the branch tip is the change commit, so HEAD~1 lies beyond
the shallow boundary:
fatal: ambiguous argument 'HEAD~1': unknown revision or path not in the working tree.
##[error]Process completed with exit code 128.
Worked example: run 30809172607
for change 46399, merged at 11:21:02,
dispatched at 11:21:06.
Upstream fixes: lfreleng-actions/checkout-gerrit-change-action#129 and
lfit/releng-reusable-workflows#843
Contributing factors in this repository
Independent of the two causes above, both Gerrit workflows here reference
retired sources:
- lfit/gerrit-review-action — the action now lives under lfreleng-actions
- technote-space/workflow-conclusion-action — archived November 2023, targets
Node.js 20, and is the source of the deprecation warnings that appear as
annotations on every run
- lfit/releng-reusable-workflows/...@main — unpinned; behaviour can change
without a corresponding change here
Proposed actions
- Short term, here. Add a visibility gate to
gerrit-required-verify-non-voting.yaml: query Gerrit as the voting account
before doing anything, and skip the remaining jobs when the change is not
readable. This removes ~91% of the failure emails immediately and is safe to
revert once the dispatcher is fixed. A change implementing this is ready and
can be pushed for review on request.
- Short term, here. Refresh the action references above and pin the
reusable workflow call to a release SHA.
- Upstream. Land Bug: dispatch fires for changes the platform cannot read lfit/releng-gerrit_to_platform#116 so no dispatch happens
while a change is unreadable — this is the correct fix and benefits every LF
project, not only FD.io.
- Upstream. Land the checkout/isolation fixes for cause B.
- Longer term. Move the Gerrit required-verify reusable workflows to an
actively maintained home: Feat: host the Gerrit required-verify reusable workflows lfreleng-actions/generic-workflows#23
Happy to supply the full run classification or raw logs behind any of the
figures above.
Summary
The Gerrit Required Verify workflow in this repository fails routinely, and
each failure emails the change author and the releng/collab-it lists. This is
not a recent regression — it has been steady for at least the last 35 days.
Sample: the last 200 failed runs (2026-06-30 → 2026-08-04), against 200
consecutive runs over 2026-07-27 → 2026-08-04 for the rate.
Neither dominant cause is a fault in this repository's own content. Both have
upstream fixes, linked below.
Cause A — verify is dispatched for private Gerrit changes (91%)
lfit/releng-gerrit_to_platform dispatches on patchset-created regardless of
whether the change is private. The Gerrit hook runs server-side with full
visibility; the fdio.required.gha account that has to clear and set the vote
does not have it. Both the prepare and vote jobs then die with:
Worked example — change 46421:
Uploading private and publishing later is a normal VPP developer workflow — 9 of
the last 60 vpp master changes did it — and every patchset pushed while
private produces a failure email.
Granting the service account visibility of private changes would not fix this:
private change refs are not replicated to GitHub, so the verify job could not
fetch the change anyway.
Importantly, unsetting private fires a fresh dispatch on its own, so suppressing
the earlier dispatches costs no verification coverage.
Upstream fix: lfit/releng-gerrit_to_platform#116
Cause B — shallow checkout when the change has already merged (8%)
gerrit-compose-required-info-yaml-verify.yaml checks isolation with
git diff --name-only HEAD~1 over a fetch-depth: 1 checkout. For CSIT
oper-* cherry-picks, which are submitted pre-approved and merge within seconds
of the dispatch, the branch tip is the change commit, so HEAD~1 lies beyond
the shallow boundary:
Worked example: run 30809172607
for change 46399, merged at 11:21:02,
dispatched at 11:21:06.
Upstream fixes: lfreleng-actions/checkout-gerrit-change-action#129 and
lfit/releng-reusable-workflows#843
Contributing factors in this repository
Independent of the two causes above, both Gerrit workflows here reference
retired sources:
Node.js 20, and is the source of the deprecation warnings that appear as
annotations on every run
without a corresponding change here
Proposed actions
gerrit-required-verify-non-voting.yaml: query Gerrit as the voting account
before doing anything, and skip the remaining jobs when the change is not
readable. This removes ~91% of the failure emails immediately and is safe to
revert once the dispatcher is fixed. A change implementing this is ready and
can be pushed for review on request.
reusable workflow call to a release SHA.
while a change is unreadable — this is the correct fix and benefits every LF
project, not only FD.io.
actively maintained home: Feat: host the Gerrit required-verify reusable workflows lfreleng-actions/generic-workflows#23
Happy to supply the full run classification or raw logs behind any of the
figures above.