FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

ci: run slow CI suites in the merge queue, not on every PR by gmaclennan · Pull Request #142 · digidem/comapeo-core-react-native · GitHub

ci: run slow CI suites in the merge queue, not on every PR - #142

Merged
gmaclennan merged 5 commits into
mainfrom
claude/modest-varahamihira-067490
Jun 23, 2026
Merged

ci: run slow CI suites in the merge queue, not on every PR#142
gmaclennan merged 5 commits into
mainfrom
claude/modest-varahamihira-067490

Conversation

gmaclennan commented Jun 23, 2026
edited
Loading

Copy link
Copy Markdown
Member

Closes #139.

Problem

The slow CI ran on every PR push: BrowserStack e2e (~15–25 min paid device time), plus the >5-min native jobs — Instrumented Tests (~10m), Integration Tests (~9m), iOS Device Build (~7m). Much of it can't be affected by the change (docs/config PRs) or is redundant across the many pushes of a single PR. A naive paths: filter can't fix it: a path-skipped workflow never reports its required checks, leaving the PR stuck "pending".

Approach: run slow suites in the merge queue, defer them on PRs

The merge queue is the enforced merge path (the ruleset's merge_queue rule builds each PR individually, ALLGREEN, before it lands), so it's the right place to spend the expensive runs — once per merge instead of once per push, and they can't be bypassed.

A shared decision (event + label + path) drives every slow suite:

Trigger Slow suites run?
merge_group Yes, unless the change is docs/config-only
workflow_dispatch Yes — manual full run
pull_request No by default; Yes with the run-e2e label
pull_request_target (e2e only) Yes once safe-to-test is added, docs-only skipped

On a plain PR the slow jobs are skipped (e2e reports a cheap-green e2e / Gate; the native jobs report skipped, which satisfies their required checks), so the PR stays mergeable with no wasted device/runner time. The real run happens in the queue and ejects the PR if it fails. Devs touching device/native code add run-e2e (label created in this repo) or trigger a manual dispatch for pre-queue feedback.

What's gated

  • BrowserStack e2e (e2e-reusable.yml): build + upload + device jobs, collapsed behind a single always-reporting e2e / Gate required check (replaces 5 nested checks). Decision in its inline changes job, shared by both callers (e2e-tests.yml, e2e-trusted.yml).
  • Native >5-min jobs (android-tests.yml, ios-tests.yml): Instrumented Tests (30), Integration Tests (Example App), iOS Device Build. Decision factored into a shared detect-heavy-ci.yml reusable workflow. No ruleset change needed — the check names are unchanged and a skipped required check counts as passing.
  • Stays on every PR (fast feedback, <5 min): Lint & Unit Tests, JVM Unit Tests, Swift Package Tests.

Why actions/github-script (not native paths or dorny/paths-filter)

Native trigger paths reintroduces the stuck-pending problem and needs a fragile mirrored stub workflow. dorny/paths-filter adds a third-party dependency plus split-filter workarounds. github-script (already used in pr-title.yml) does the read-only file listing inline, no checkout, safe on pull_request_target, and is unit-testable.

PR-title lint: make it queue-compatible

Lint conventional title is a required check but only triggered on pull_request_target, so it never reported on merge_group — and an absent required check blocks the merge queue. Now the workflow also triggers on merge_group and the job passes as a no-op there (the title was validated on the PR; the queue can't change it). It stays enforced on PRs; the changelog-label job stays PR-only.

Security model unchanged

Untrusted (Dependabot/fork) PRs still leave e2e / Gate absent (not skipped-as-passing) on the internal path until a maintainer adds safe-to-test; a job-level skip was avoided for e2e because GitHub treats a skipped required check as passing. The native suites have no secrets, so skipped-as-passing is exactly what we want there (defer on PR, run in queue).

⚠️ Deployment steps (in order)

  1. Merge this PR. If the queue is currently blocked by the absent Lint conventional title on merge_group, this first merge may need an admin bypass; afterwards the queue accepts PRs normally.
  2. Swap the e2e required checks (admin-only — gh can't PATCH rulesets). On ruleset 13824010, remove the five e2e / Build (Android), e2e / Build (iOS), e2e / Run tests (Android), e2e / Run tests (iOS), e2e / Upload test suite and add e2e / Gate. Do this right after merge — until then a docs-only PR in the queue could stick on the old nested checks. (Script provided in the PR thread.)

The native required checks need no ruleset change.

Validation

  • actionlint + shellcheck clean on all seven workflows.
  • Decision logic unit-tested: e2e (13 cases) and native (10 cases) — PR-code-no-label defers; run-e2e forces a run; docs-only skips everywhere; merge_group code runs / docs skips; workflow_dispatch always runs; workflow self-tests (editing a suite's workflow runs that suite); empty file list fails safe.

To exercise the slow suites on this PR before merge: add the run-e2e label (this PR edits CI machinery, which otherwise defers to the queue), or let the queue run them.

🤖 Generated with Claude Code

The full BrowserStack e2e (build Android + iOS, upload, run on real
devices — ~15-25 min of paid device time) ran on every PR, including
ones that can't affect runtime behaviour (dependabot.yml, pr-title.yml,
docs, *.md). Path-filtering the trigger can't fix this on its own: a
workflow skipped by `paths` never reports its required checks, which
leaves the PR stuck pending.

Instead, detect changed paths inside the reusable workflow and collapse
the five nested required checks into one always-reporting gate:

- New `changes` job classifies the PR's files (via actions/github-script,
  already used in pr-title.yml — read-only file listing, no checkout, so
  it's safe on the pull_request_target path). Config/docs-only => skip;
  code, package*.json, or a change to the e2e machinery itself => run.
  merge_group and workflow_dispatch always run. Defaults to running on
  any uncertainty.
- The build/upload/test jobs are gated on that result; downstream jobs
  cascade-skip.
- A new `gate` job (needs: [everything], if: always()) passes when e2e
  was legitimately skipped or every job succeeded, and fails closed on
  any failure/cancel or a path-detection error. Its context, `e2e / Gate`,
  is meant to replace the five nested e2e checks as the required one.

Lives in the reusable workflow so both callers (e2e-tests.yml and the
label-gated e2e-trusted.yml) honour the skip from one definition. The
trust model is unchanged: untrusted PRs still leave `e2e / Gate` absent
(not skipped-as-passing) until a maintainer adds `safe-to-test`.

Requires a follow-up ruleset change (admin-only): swap the five
`e2e / Build|Run tests|Upload …` required checks for `e2e / Gate`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions Bot added the maintenance Refactor / test / chore / ci / build (changelog) label Jun 23, 2026
gmaclennan enabled auto-merge June 23, 2026 08:46
gmaclennan disabled auto-merge June 23, 2026 09:07
Now that `e2e / Gate` always reports on both `pull_request` and
`merge_group`, defer the expensive BrowserStack run to where it matters:
the merge queue, which is the enforced merge path (the ruleset's
merge_queue rule builds each PR individually before it lands).

Decision (all in the `changes` job):
- merge_group: run, unless the merge group is docs/config-only.
- workflow_dispatch: run (explicit manual request).
- pull_request: defer to the queue by default; run only if the PR
  carries the `run-e2e` label (added `labeled` to the PR triggers).
- pull_request_target: unchanged — runs once a maintainer adds
  `safe-to-test`, still skipping docs/config-only diffs.

On a plain PR the gate still passes cheaply (heavy jobs skipped), so the
PR stays mergeable with no paid device time; the real run happens in the
queue and ejects the PR if it fails. Devs who touch device-facing code
can add `run-e2e` (or dispatch) for pre-queue feedback.

merge_group has no PR file list, so its diff comes from comparing the
group's base_sha...head_sha.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gmaclennan changed the title ci: skip BrowserStack e2e on config-/docs-only PRs via an e2e gate ci: run BrowserStack e2e in the merge queue, not on every PR Jun 23, 2026
gmaclennan and others added 2 commits June 23, 2026 10:34
The three jobs that take >5 min — Instrumented Tests (10m), Integration
Tests (9m), iOS Device Build (7m) — ran on every PR push. Gate them on
the same merge-queue-first decision as the e2e suite: run in the merge
queue (the pre-merge gate), on manual dispatch, or on a PR labeled
`run-e2e`; skip on a plain PR and for docs/config-only changes.

A shared `detect-heavy-ci.yml` reusable workflow makes the decision so
the Android and iOS workflows share one copy (the e2e suite keeps its
own inline `changes` job — its self-test machinery paths differ). On a
plain PR the heavy jobs report as `skipped`, which satisfies the
required status checks without running them; the real run happens in the
queue and ejects the PR if it fails. The fast JVM-unit and Swift-package
jobs stay on every PR for quick feedback.

No ruleset change needed: the check names are unchanged and a skipped
required check counts as passing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`Lint conventional title` is a required check but only triggered on
`pull_request_target`, so it never reported on `merge_group` — an absent
required check blocks the merge queue. Trigger the workflow on
merge_group too and have the job pass as a no-op there (the title was
already validated on the PR and the queue can't change it), so the check
is present and green in the queue while staying enforced on PRs. The
label job stays PR-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gmaclennan changed the title ci: run BrowserStack e2e in the merge queue, not on every PR ci: run slow CI suites in the merge queue, not on every PR Jun 23, 2026
… reports

A matrix job skipped by a job-level `if` reports its check run under the
bare job name (`Instrumented Tests`), without the matrix suffix — so the
required `Instrumented Tests (30)` context never reported and the PR hung
on "Expected — Waiting for status to be reported".

The matrix held a single value (api-level 30), so replace it with the
literal name `Instrumented Tests (30)` and a hardcoded api-level. A plain
(non-matrix) job reports the exact context name as `skipped` when gated
out, which satisfies the required check — matching the iOS heavy jobs,
which already skip-as-passing correctly. No ruleset change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
gmaclennan added this pull request to the merge queue Jun 23, 2026
Merged via the queue into main with commit 4b19286 Jun 23, 2026
24 checks passed
gmaclennan deleted the claude/modest-varahamihira-067490 branch June 23, 2026 11:23
gmaclennan added a commit that referenced this pull request Jun 23, 2026
…146 local iOS e2e)

After merging main, square the docs with what actually shipped:

- #142 landed a shared detect-heavy-ci.yml that gates the *heavy native*
  suites too (Instrumented Tests, Integration Tests, iOS Device Build) on
  the same merge-queue / run-e2e / docs-skip logic as e2e — they skip as
  passing when gated out. Only the fast suites (JVM unit, Swift package,
  lint) stay always-on. Update the §3 workflow map, the §4.2 gate
  discussion (native suites use skipped=passing; only e2e needs the
  always-reporting gate), the §5 "always-on" claim + run-e2e label scope,
  and the CONTRIBUTING merge-queue section.
- #146 corrected the iOS-e2e-can't-run-locally claim: it runs fine on a
  simulator via `expo run:ios` (local signing → keychain works); the
  BrowserStack-only failure was the unsigned device archive. Rewrite §7.2
  to cover local iOS + Android via maestro/e2e.local.yaml.
- Drop the stale "ruleset change lands with #142" note — the swap to the
  single `e2e / Gate` required check is done.
- Add detect-heavy-ci.yml and maestro/e2e.local.yaml to the references.
gmaclennan added a commit that referenced this pull request Jun 23, 2026
…eev-ef879d

* origin/main: (47 commits)
  ci(e2e): retry BrowserStack builds when a session fails to start
  rename apps/integration for dependabot
  ci(deps): split apps/ dependabot updates into a separate grouped PR
  ci(android): pre-install NDK with retry to fix corrupt-zip flake
  Release v1.0.0-pre.4
  fix(backend): apply defaultOnlineStyleUrl to the standalone map server
  docs(testing): clarify the iOS device-build test, fix cost framing, tighten prose
  docs: reconcile testing docs with landed main (#142 heavy-CI gating, #146 local iOS e2e)
  test(ios): de-flake shutdown-IPC assertions via deterministic wait
  refactor: drop vestigial integration Maestro suite, move runner to scripts/
  docs: document running iOS e2e on a local simulator
  docs: refresh agents.md and e2e/README.md for the current layout
  test(android): cover wiped-wrapper-key recovery and legacy-entry preservation
  feat(android): harden rootkey migration and add recovery telemetry
  refactor: rename apps/example → apps/integration
  Release v1.0.0-pre.3
  docs(testing): cover all e2e jobs, link section refs, contrast the two apps
  ci: drop single-value matrix on instrumented tests so the gated check reports
  ci: report PR-title lint in the merge queue so it can be required
  ci: defer slow native test jobs to the merge queue
  ...
gmaclennan added a commit that referenced this pull request Jun 24, 2026
…-9873de

* origin/main: (46 commits)
  docs: rewrite README for Expo-first usage, API, and plugin options
  test(notification-permission): fix Android test for API < 33 emulators
  test(notification-permission): add iOS + Android coverage for the FGS gate
  ci(e2e): retry BrowserStack builds when a session fails to start
  rename apps/integration for dependabot
  ci(deps): split apps/ dependabot updates into a separate grouped PR
  ci(android): pre-install NDK with retry to fix corrupt-zip flake
  Release v1.0.0-pre.4
  fix(backend): apply defaultOnlineStyleUrl to the standalone map server
  docs(testing): clarify the iOS device-build test, fix cost framing, tighten prose
  docs: reconcile testing docs with landed main (#142 heavy-CI gating, #146 local iOS e2e)
  test(ios): de-flake shutdown-IPC assertions via deterministic wait
  refactor: drop vestigial integration Maestro suite, move runner to scripts/
  docs: document running iOS e2e on a local simulator
  docs: refresh agents.md and e2e/README.md for the current layout
  test(android): cover wiped-wrapper-key recovery and legacy-entry preservation
  feat(android): harden rootkey migration and add recovery telemetry
  refactor: rename apps/example → apps/integration
  Release v1.0.0-pre.3
  docs(testing): cover all e2e jobs, link section refs, contrast the two apps
  ...

# Conflicts:
#	.github/workflows/ios-tests.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenance Refactor / test / chore / ci / build (changelog)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: skip the BrowserStack e2e on config-/docs-only PRs (without breaking required checks)

1 participant


Back | FazBrowse Home | New Git URL