FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

tools: prefilter commit queue metadata · nodejs/node@aa3b598 · GitHub

/ node Public

Commit aa3b598

Browse files
panva
authored and
aduh95
committed
tools: prefilter commit queue metadata
Run `git node metadata --readme` in the lightweight commit queue selector before starting the checkout-heavy landing job. The selector keeps `commit-queue` on PRs that are only waiting for approvals, TSC approval count, or wait time. Other metadata failures continue to the existing commit queue script so failure labels and comments are still produced by the current landing path. Fetch age-based, fast-track, and broader queue buckets before deduplicating the list so not-yet-ready PRs do not crowd out PRs that would otherwise have been selected. Signed-off-by: Filip Skokan <panva.ip@gmail.com> PR-URL: #64343 Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
1 parent 003a913 commit aa3b598

2 files changed

Lines changed: 208 additions & 81 deletions

File tree

‎.github/workflows/commit-queue.yml‎

Lines changed: 100 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -22,51 +22,44 @@ permissions:
2222
contents: read
2323

2424
jobs:
25-
get_mergeable_prs:
25+
get_candidate_prs:
2626
permissions:
2727
pull-requests: read
2828
if: github.repository == 'nodejs/node'
2929
runs-on: ubuntu-slim
3030
outputs:
31-
numbers: ${{ steps.get_mergeable_prs.outputs.numbers }}
31+
candidates: ${{ steps.get_candidate_prs.outputs.candidates }}
3232
steps:
33-
- name: Get Pull Requests
34-
id: get_mergeable_prs
33+
- name: Get Pull Request Candidates
34+
id: get_candidate_prs
3535
run: |
36-
prs=$(gh pr list \
36+
list_prs() {
37+
gh pr list \
3738
--repo "$GITHUB_REPOSITORY" \
3839
--base "$GITHUB_REF_NAME" \
3940
--label 'commit-queue' \
41+
"$@" \
4042
--json 'number' \
41-
--search "created:<=$(date --date="2 days ago" +"%Y-%m-%dT%H:%M:%S%z") -label:blocked" \
4243
-t '{{ range . }}{{ .number }} {{ end }}' \
43-
--limit 100)
44-
fast_track_prs=$(gh pr list \
45-
--repo "$GITHUB_REPOSITORY" \
46-
--base "$GITHUB_REF_NAME" \
47-
--label 'commit-queue' \
44+
--limit 100
45+
}
46+
aged_prs=$(list_prs \
47+
--search "created:<=$(date --date="2 days ago" +"%Y-%m-%dT%H:%M:%S%z") -label:blocked")
48+
fast_track_prs=$(list_prs \
4849
--label 'fast-track' \
49-
--search "-label:blocked" \
50-
--json 'number' \
51-
-t '{{ range . }}{{ .number }} {{ end }}' \
52-
--limit 100)
53-
numbers=$(echo $prs' '$fast_track_prs | jq -r -s 'unique | join(" ")')
54-
echo "numbers=$numbers" >> "$GITHUB_OUTPUT"
50+
--search "-label:blocked")
51+
queued_prs=$(list_prs \
52+
--search "-label:blocked")
53+
candidates=$(printf '%s %s %s\n' "$aged_prs" "$fast_track_prs" "$queued_prs" |
54+
jq -r -s 'reduce .[] as $pr ([]; if index($pr) then . else . + [$pr] end) | join(" ")')
55+
echo "candidates=$candidates" >> "$GITHUB_OUTPUT"
5556
env:
5657
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
5758
commitQueue:
58-
needs: get_mergeable_prs
59-
if: needs.get_mergeable_prs.outputs.numbers != ''
59+
needs: get_candidate_prs
60+
if: needs.get_candidate_prs.outputs.candidates != ''
6061
runs-on: ubuntu-slim
6162
steps:
62-
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
63-
with:
64-
# A personal token is required because pushing with GITHUB_TOKEN will
65-
# prevent commits from running CI after they land. It needs
66-
# to be set here because `checkout` configures GitHub authentication
67-
# for push as well.
68-
token: ${{ secrets.GH_USER_TOKEN }}
69-
7063
# Install dependencies
7164
- name: Install Node.js
7265
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
@@ -93,7 +86,86 @@ jobs:
9386
GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}
9487
JENKINS_TOKEN: ${{ secrets.JENKINS_TOKEN }}
9588

89+
- name: Filter Pull Requests
90+
id: get_mergeable_prs
91+
run: |
92+
readme="${RUNNER_TEMP}/README.md"
93+
curl -fsSLo "$readme" "https://github.com/${GITHUB_REPOSITORY}/raw/${GITHUB_SHA}/README.md"
94+
95+
numbers=
96+
# shellcheck disable=SC2086
97+
for pr in $CANDIDATES; do
98+
metadata="${RUNNER_TEMP}/metadata-${pr}.json"
99+
output="${RUNNER_TEMP}/metadata-${pr}.txt"
100+
if git node metadata "$pr" \
101+
--owner "$GITHUB_REPOSITORY_OWNER" \
102+
--repo "$REPOSITORY" \
103+
--readme "$readme" \
104+
--json > "$metadata" 2> "$output"; then
105+
metadata_status=0
106+
else
107+
metadata_status=$?
108+
fi
109+
110+
if [ -s "$output" ]; then
111+
cat "$output"
112+
fi
113+
114+
case "$metadata_status" in
115+
0|2[0-9]|4[0-9]) ;;
116+
*)
117+
echo "git node metadata failed for pr ${pr} with exit code ${metadata_status}"
118+
exit 1
119+
;;
120+
esac
121+
122+
metadata_exit_code=$(jq -r '.exitCode' "$metadata") || {
123+
echo "failed to parse metadata JSON for pr ${pr}"
124+
exit 1
125+
}
126+
if [ "$metadata_exit_code" != "$metadata_status" ]; then
127+
echo "metadata JSON exitCode mismatch for pr ${pr}"
128+
exit 1
129+
fi
130+
metadata_reason_codes=$(jq -r '.reasonCodes | join(", ")' "$metadata") || {
131+
echo "failed to parse metadata reason codes for pr ${pr}"
132+
exit 1
133+
}
134+
135+
if [ "$metadata_status" -eq 0 ]; then
136+
echo "pr ${pr} is ready for the commit queue"
137+
numbers="$numbers $pr"
138+
continue
139+
fi
140+
141+
if [ "$metadata_status" -ge 20 ] && [ "$metadata_status" -le 29 ]; then
142+
echo "pr ${pr} skipped, not ready to land"
143+
echo "reason codes: ${metadata_reason_codes}"
144+
continue
145+
fi
146+
147+
echo "pr ${pr} will be handled by the commit queue"
148+
echo "reason codes: ${metadata_reason_codes}"
149+
numbers="$numbers $pr"
150+
done
151+
152+
numbers=$(echo "$numbers" | xargs)
153+
echo "numbers=$numbers" >> "$GITHUB_OUTPUT"
154+
env:
155+
CANDIDATES: ${{ needs.get_candidate_prs.outputs.candidates }}
156+
GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}
157+
158+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
159+
if: steps.get_mergeable_prs.outputs.numbers != ''
160+
with:
161+
# A personal token is required because pushing with GITHUB_TOKEN will
162+
# prevent commits from running CI after they land. It needs
163+
# to be set here because `checkout` configures GitHub authentication
164+
# for push as well.
165+
token: ${{ secrets.GH_USER_TOKEN }}
166+
96167
- name: Start the Commit Queue
97-
run: ./tools/actions/commit-queue.sh "${GITHUB_REPOSITORY_OWNER}" "${REPOSITORY}" ${{ needs.get_mergeable_prs.outputs.numbers }}
168+
if: steps.get_mergeable_prs.outputs.numbers != ''
169+
run: ./tools/actions/commit-queue.sh "${GITHUB_REPOSITORY_OWNER}" "${REPOSITORY}" ${{ steps.get_mergeable_prs.outputs.numbers }}
98170
env:
99171
GITHUB_TOKEN: ${{ secrets.GH_USER_TOKEN }}

‎doc/contributing/commit-queue.md‎

Lines changed: 108 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,48 @@
11
# Commit queue
22

3-
_tl;dr: You can land pull requests by adding the `commit-queue` label to it._
3+
_tl;dr: You can ask the queue to land pull requests by adding the
4+
`commit-queue` label to them._
45

56
Commit Queue is a feature for the project which simplifies the
67
landing process by automating it via GitHub Actions. With it, collaborators can
7-
land pull requests by adding the `commit-queue` label to a PR. All
8-
checks will run via `@node-core/utils`, and if the pull request is ready to
9-
land, the Action will rebase it and push to `main`.
8+
queue pull requests for landing by adding the `commit-queue` label to a PR. The
9+
selector checks readiness with `@node-core/utils`. If the pull request is only
10+
blocked on a deferrable condition, currently wait time, the queue leaves the
11+
label in place and retries later. Other failures continue to the existing
12+
landing and failure-reporting path.
1013

1114
This document gives an overview of how the Commit Queue works, as well as
1215
implementation details, reasoning for design choices, and current limitations.
1316

1417
## Overview
1518

16-
From a high-level, the Commit Queue works as follow:
17-
18-
1. Collaborators will add `commit-queue` label to pull requests ready to land
19-
2. Every five minutes the queue will do the following for each mergeable pull request
20-
with the label:
21-
1. Check if the PR also has a `request-ci` label (if it has, skip this PR
19+
From a high-level, the Commit Queue works as follows:
20+
21+
1. Collaborators will add `commit-queue` label to pull requests they want the
22+
queue to land. The label can be added before the pull request has completed
23+
its wait time, or before requested CI has finished. Required approvals must
24+
already be in place. The commit queue does not request CI on its own.
25+
2. On each scheduled run, the queue builds a candidate list from open pull
26+
requests with the `commit-queue` label and without the `blocked` label. The
27+
workflow uses a five-minute cron, but GitHub Actions scheduled workflows are
28+
not guaranteed to run exactly every five minutes. For each candidate, the
29+
queue will:
30+
1. In the landing job, install and configure `@node-core/utils`, then run a
31+
metadata-only readiness check without checking out the repository
32+
2. If the metadata check exits with a deferrable readiness code, meaning
33+
the PR is only blocked on wait time, keep the `commit-queue` label and
34+
skip this PR until a later queue run
35+
3. Check if the PR also has a `request-ci` label (if it has, skip this PR
2236
since it's pending a CI run)
23-
2. Check if the last Jenkins CI is finished running (if it is not, skip this
24-
PR)
25-
3. Remove the `commit-queue` label
26-
4. Run `git node land <pr> --oneCommitMax`
27-
5. If it fails:
28-
1. Abort `git node land` session
29-
2. Add `commit-queue-failed` label to the PR
30-
3. Leave a comment on the PR with the output from `git node land`
31-
4. Skip next steps, go to next PR in the queue
32-
6. If it succeeds:
33-
1. Push the changes to nodejs/node
37+
4. Check whether GitHub checks are still running (if they are, skip this PR)
38+
5. Remove the `commit-queue` label and run `git node land`
39+
6. If it fails:
40+
1. Add the `commit-queue-failed` label to the PR
41+
2. Leave a comment on the PR with the output from `git node land`
42+
3. Abort the `git node land` session. If the abort succeeds, continue to
43+
the next PR; otherwise, stop the queue in an unknown state
44+
7. If it succeeds:
45+
1. Push or merge the changes into nodejs/node
3446
2. Leave a comment on the PR with `Landed in ...`
3547
3. Close the PR
3648
4. Go to next PR in the queue
@@ -51,18 +63,23 @@ of the commit queue:
5163
guidelines or be a valid [`fixup!`](https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---fixupamendrewordltcommitgt)
5264
commit that will be correctly handled by the [`--autosquash`](https://git-scm.com/docs/git-rebase#Documentation/git-rebase.txt---autosquash)
5365
option
54-
2. A CI must've ran and succeeded since the last change on the PR
66+
2. A CI must have run and succeeded since the last change on the PR
5567
3. A collaborator must have approved the PR since the last change
5668
4. Only Jenkins CI and GitHub Actions are checked (V8 CI and CITGM are ignored)
5769
5. The PR must target the `main` branch (PRs opened against other branches, such
5870
as backport PRs, are ignored)
5971

6072
## Implementation
6173

62-
The [action](../../.github/workflows/commit-queue.yml) will run on scheduler
63-
events every five minutes. Five minutes is the smallest number accepted by
64-
the scheduler. The scheduler is not guaranteed to run every five minutes, it
65-
might take longer between runs.
74+
The [action](../../.github/workflows/commit-queue.yml) runs on scheduled events.
75+
It uses a five-minute cron because that is the smallest interval accepted by
76+
GitHub Actions. Scheduled workflows are not guaranteed to run exactly at that
77+
cadence and might take longer between runs.
78+
79+
The workflow also uses a concurrency group so only one commit queue run can be
80+
active at a time. If a scheduled run starts while a previous run is still
81+
running, GitHub Actions keeps at most one pending run for the same concurrency
82+
group. A newer pending run replaces an older pending run.
6683

6784
Using the scheduler is preferable over using pull\_request\_target for two
6885
reasons:
@@ -76,41 +93,79 @@ reasons:
7693
commit, meaning we wouldn't be able to use it for already opened PRs
7794
without rebasing them first.
7895

79-
`@node-core/utils` is configured with a personal token and
80-
a Jenkins token from
81-
[@nodejs-github-bot](https://github.com/nodejs/github-bot).
82-
`octokit/graphql-action` is used to fetch all pull requests with the
83-
`commit-queue` label. The output is a JSON payload, so `jq` is used to turn
84-
that into a list of PR ids we can pass as arguments to
85-
[`commit-queue.sh`](../../tools/actions/commit-queue.sh).
86-
87-
> The personal token only needs permission for public repositories and to read
88-
> profiles, we can use the GITHUB\_TOKEN for write operations. Jenkins token is
96+
The workflow starts with a small candidate job that uses GitHub CLI to fetch
97+
pull requests with the `commit-queue` label. It first fetches the same
98+
age-based and fast-track buckets the queue used before accepting early queue
99+
requests, then fetches the broader queue and de-duplicates the result. This
100+
keeps not-yet-ready PRs from crowding out PRs that the previous query would
101+
have selected if GitHub paginates or caps a query result.
102+
103+
If there are candidate PRs, the landing job installs and configures
104+
`@node-core/utils` once with a personal token and a Jenkins token from
105+
[@nodejs-github-bot](https://github.com/nodejs/github-bot). It then downloads
106+
the workflow commit's README without checking out the repository and runs
107+
`git node metadata --readme --json` for each candidate. This uses the same
108+
`@node-core/utils` PR readiness checks as `git node land`, but does not clone,
109+
fetch, or merge the PR. The filter consumes the structured metadata result
110+
and its exit code instead of matching human-readable output:
111+
112+
* exit code `0`: the PR is ready and is passed to
113+
[`commit-queue.sh`](../../tools/actions/commit-queue.sh)
114+
* exit codes `20`-`29`: the PR is not ready for a deferrable metadata reason,
115+
currently wait time, so it keeps the `commit-queue` label and is retried
116+
later
117+
* exit codes `40`-`49`: the PR has a hard or mixed metadata readiness failure
118+
and is passed to [`commit-queue.sh`](../../tools/actions/commit-queue.sh)
119+
120+
The `20`-`29` exit code range is reserved by `@node-core/utils` for deferrable
121+
metadata readiness states, and `40`-`49` is reserved for hard metadata failure
122+
states. Unknown filter failures fail the workflow before starting the landing
123+
script and leave PR labels unchanged so the queue can retry on a later
124+
scheduled run. PRs passed through with exit code `40`-`49` continue through
125+
`commit-queue.sh`. The workflow checks out the repository only when at least
126+
one PR remains after filtering. The script still applies its existing
127+
`request-ci` and pending-check deferrals before removing the queue label and
128+
reporting a hard failure.
129+
130+
> The personal token needs permission for public repositories and to read
131+
> profiles. It is used by `@node-core/utils` and by the landing job for
132+
> checkout, label and comment updates, merging, and pushing. Jenkins token is
89133
> required to check CI status.
90134
91135
`commit-queue.sh` receives the following positional arguments:
92136

93137
1. The repository owner
94138
2. The repository name
95-
3. The Action GITHUB\_TOKEN
96-
4. Every positional argument starting at this one will be a pull request ID of
139+
3. Every positional argument starting at this one will be a pull request ID of
97140
a pull request with commit-queue set.
98141

99-
The script will iterate over the pull requests. `ncu-ci` is used to check if
100-
the last CI is still pending, and calls to the GitHub API are used to check if
101-
the PR is waiting for CI to start (`request-ci` label). The PR is skipped if CI
102-
is pending. No other CI validation is done here since `git node land` will fail
103-
if the last CI failed.
104-
105-
The script removes the `commit-queue` label. It then runs `git node land`,
106-
forwarding stdout and stderr to a file. If any errors happen,
107-
`git node land --abort` is run, and then a `commit-queue-failed` label is added
108-
to the PR, as well as a comment with the output of `git node land`.
109-
110-
If no errors happen during `git node land`, the script will use the
111-
`GITHUB_TOKEN` to push the changes to `main`, and then will leave a
112-
`Landed in ...` comment in the PR, and then will close it. Iteration continues
113-
until all PRs have done the steps above.
142+
The script will iterate over the pull requests. GitHub CLI is used to check if
143+
the PR is waiting for CI to start (`request-ci` label) or still has pending
144+
GitHub checks. The PR is skipped if CI is pending. No other CI validation is
145+
done here since `git node land` will fail if the last CI failed.
146+
147+
The script removes the `commit-queue` label, then runs `git node land`,
148+
forwarding stdout and stderr to a file. PRs that are only blocked on wait time
149+
should have already been filtered by the metadata check. If a hard readiness
150+
failure appears between the metadata filter and `git node land`, the landing
151+
job adds a `commit-queue-failed` label to the PR, leaves a comment with the
152+
output of `git node land`, and then aborts the landing session. If the abort
153+
fails, the queue stops instead of continuing in an unknown state.
154+
155+
Fast-tracked PRs use the metadata check before checkout and the landing script.
156+
If the fast-track request has not yet received enough collaborator thumbs-up,
157+
the queue keeps the `commit-queue` label and retries until either the
158+
fast-track request is approved or the PR becomes landable through the regular
159+
wait-time rules. The commit queue does not create the fast-track request
160+
comment; that is handled when the `fast-track` label is added. If that comment
161+
is missing, the queue reports the failure instead of keeping the PR queued.
162+
163+
If no errors happen during `git node land`, the script either pushes the direct
164+
rebase landing to `main` or uses GitHub's squash merge API for single-commit and
165+
fixup landings. It then leaves a `Landed in ...` comment in the PR. GitHub
166+
closes PRs merged through the merge API automatically; for direct pushes, the
167+
script closes the PR. Iteration continues until all PRs have done the steps
168+
above.
114169

115170
## Reverting broken commits
116171

0 commit comments

Comments
 (0)
Back | FazBrowse Home | New Git URL