FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

Tags · uber/submitqueue · GitHub

Tags: uber/submitqueue

Tags

v0.3.0-20260824200326-c93310ddce2f

Toggle v0.3.0-20260824200326-c93310ddce2f's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(stovepipe): Consume hook events (#632)

## Summary
**What**:
- Register the hook stage and its dead-letter stage in Stovepipe's
consumer wiring, on a topic named for the domain, with a resolver that
hands every event to the no-op hook.

**Why**:
- Wire the stage in a second domain, so attaching a real integration to
Stovepipe is a resolver swap rather than new plumbing, and two domains
on one queue backend keep their own hook topics.

## Test Plan
- [x] Add unit tests.

## Revert Plan
- Revert this PR. Nothing publishes hook events yet and the resolver
returns only a no-op hook.

## Issues
-
[CODEM-416](https://linear.app/uber/issue/CODEM-416/hooks-integration-downstream-notificaiton)

## Stack
1. #607
1. #608
1. @ #632

v0.3.0-20260824195328-ac94e2437ed0

Toggle v0.3.0-20260824195328-ac94e2437ed0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(orch): Consume hook events (#608)

## Summary
**What**:
- Register the orchestrator's hook stage, so lifecycle events are
consumed from the queue and handed to the integrations each event
resolves to. No integration is wired yet, so events are accepted and
discarded.
- Register the matching dead-letter stage, so an event that fails every
retry is recorded and alerted on instead of accumulating unread.

**Why**:
- Prove the delivery path end to end before anything produces an event,
so a later change adds producers to a route that already works.
- Keep a deployment with no integrations configured distinguishable from
one that is silently losing events.

## Test Plan
- [x] Add unit tests.

## Revert Plan
- Revert this PR. Since nothing publishes hook events yet and the
resolver returns only a no-op hook, no side effect is lost.

## Issues
-
[CODEM-416](https://linear.app/uber/issue/CODEM-416/hooks-integration-downstream-notificaiton)

## Stack
1. #607
1. @ #608
1. #632

v0.3.0-20260824193758-fee08f1c5fa7

Toggle v0.3.0-20260824193758-fee08f1c5fa7's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(hook): Deliver events to integrations (#607)

## Summary
**What**:
- Consume lifecycle events from a durable queue and hand each one to the
integrations it resolves to, running them concurrently and retrying then
dead-lettering the ones that keep failing.
- Record every dead-lettered event with its full payload and failure
reason, and count it on a metric meant to page.

**Why**:
- Keep a slow or broken integration from stalling or failing the work
that triggered it.
- Make a lost notification recoverable and impossible to miss, since
nothing else in the system notices when one goes missing.

## Test Plan
- [x] Add unit tests.

## Revert Plan
- Revert this PR. No service constructs the controller or subscribes to
the hook topic.

## Issues
-
[CODEM-416](https://linear.app/uber/issue/CODEM-416/hooks-integration-downstream-notificaiton)

v0.3.0-20260824173604-9a1825ec27e7

Toggle v0.3.0-20260824173604-9a1825ec27e7's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
docs: align READMEs and RFCs with current code and AGENTS.md conventi…

…ons (#629)

Updates the documentation to match the actual repository layout and
code:

- Replaces all stale CLAUDE.md references with AGENTS.md, the
authoritative conventions file.
- Fixes the project-layout description in README.md and
submitqueue/README.md: shared code lives under platform/, not the
now-removed top-level entity/extension/core/ directories.
- Expands the empty submitqueue/orchestrator/README.md with the pipeline
stages it runs.
- Updates Stovepipe READMEs to reflect Ping + Ingest RPCs and the full
ingest → process → build → buildsignal → record pipeline, plus DLQ
reconcilers.
- Corrects fully-qualified gRPC service names in service/README.md and
TESTING.md to use the uber.* proto packages.
- Corrects Runway topic names in doc/rfc/runway/workflow.md to match the
actual TopicKey constants (merge-conflict-check, runway-merge,
merge-conflict-check-signal, merge-signal).
- Clarifies integration-test Makefile targets in TESTING.md and removes
the non-existent integration-test-{service} placeholder.
- Fixes the stale TODO line-number reference in modular-queue-wiring.md.

v0.3.0-20260824154649-d74bc8c298a9

Toggle v0.3.0-20260824154649-d74bc8c298a9's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(errs): Retry transient grouped failures (#621)

## Summary
**What**:
- Retry work when any one of several failures reported together is
transient instead of giving up on the first attempt.
- Limit a classification attached to one grouped failure to that failure
alone, instead of letting it decide the outcome for the others reported
alongside it.

**Why**:
- Stop discarding failures that would have cleared on their own by
keeping a step that fans work out to several handlers from being
abandoned because one handler hit a momentary error.

## Test Plan
- [x] Add unit tests.

## Revert Plan
- Revert this PR. The changes in this PR are confined to the error
classification framework.

## Issues
-
[CODEM-462](https://linear.app/uber/issue/CODEM-462/err-classification-framework-should-handle-bundled-errors)

v0.3.0-20260824153343-039e48e07537

Toggle v0.3.0-20260824153343-039e48e07537's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(messagequeue): run queue garbage collection on busy partitions (#622

)

## Why?

Garbage collection of acknowledged `queue_messages` rows currently runs
only on idle poll ticks.

A continuously busy partition resets the GC counter whenever it delivers
a message, so it can never reach the GC threshold. Under sustained
traffic, acknowledged message rows can therefore accumulate
indefinitely.

This also means the message deduplication horizon can remain
unnecessarily large on busy partitions.

## What?

Run the existing garbage collection cadence on every poll tick rather
than only idle ticks.

The existing 100-tick throttle is preserved, so the per-partition GC
frequency is not increased. The GC safety mechanism is also unchanged:
garbage collection still uses the minimum acknowledged offset across
consumer groups.

Added:
- unit regression coverage proving GC runs on busy ticks
- MySQL integration coverage proving acknowledged rows are reclaimed
under continuous traffic

## Testing

- `go test ./platform/extension/messagequeue/...` — PASS
- `go test ./platform/base/...` — PASS
- `go vet` on the messagequeue and integration packages — PASS
- `gofmt` and `git diff --check` — clean

Not run locally due to environment limitations:
- MySQL integration test `TestGCReclaimsAckedRowsUnderContinuousTraffic`
— Docker daemon unavailable
- `make gazelle`, `make fmt`, `make lint`, `make check-tidy`, `make
check-gazelle` and Bazel tests — `make` unavailable and Bazel's Windows
C++ toolchain is not configured

v0.3.0-20260820220230-cfeea0d92308

Toggle v0.3.0-20260820220230-cfeea0d92308's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(orchestrator): carry a failed batch's reason to the request error…

… log (#616)

## Summary

### Why?

When a batch failed, every request in it terminated as `error` with an
empty message: mergesignal and speculate recorded only the terminal
batch state, and conclude terminated the request with an empty reason —
so a "did not land: error" carried no diagnostic at all. The reason is
known at the failure site, but the requests are terminated later, in
conclude.

### What?

Carry the failure reason to conclude on the fan-out message's metadata
rather than persisting it as batch state. mergesignal stamps the merge
result's reason (or "merge failed" when it is empty) and speculate
stamps "no speculation path could pass"; conclude reads it from the
message and passes it to TerminateRequest, which already writes it to
the request's terminal log. The landed and cancelled paths carry no
reason, exactly as before.

This mirrors the dead-letter path, which already sources its reason from
the delivery context. Message metadata is durably persisted and
redelivered by the queue backend, so the reason survives at-least-once
retries — and the batch entity stays purely structural, with no new
field and no schema migration. A `publish.MessageWithMetadata` helper
and a shared `topickey.MetadataKeyFailureReason` constant carry the
wiring.

The demo client's run summary also surfaces the reason, so a failed
request reads `id=error: <reason>` instead of a bare `id=error`.

## Test Plan

✅ `bazel test //submitqueue/orchestrator/...` (14/14). Updated tests
assert the reason rides the conclude message metadata (mergesignal,
speculate finalize) and reaches the request's terminal log (conclude).
Broad build and CI gates (gazelle / tidy / mocks / fmt) clean.

## Stack
1. #615
1. @ #616

v0.3.0-20260820132521-96a8ce6118bc

Toggle v0.3.0-20260820132521-96a8ce6118bc's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(stovepipe): recover record DLQ work (#623)

## Summary
### Intent

- Complete Stovepipe DLQ coverage for failures that occur while
recording a completed request.
- Reuse the existing idempotent record reconciliation path instead of
introducing a separate controller or request lifecycle state.
- This PR builds on #619, which aligns the Stovepipe DLQ controllers
with repository conventions.

### Changes

- Register the existing record controller for the `record_dlq` topic
with a distinct consumer group.
- Derive record controller identity from its topic key so primary and
DLQ instances have separate logging and metrics.
- Cover the record-DLQ controller identity configuration.

## Test Plan
- `./tool/bazel test //stovepipe/controller/record:go_default_test
--test_output=errors`
- `./tool/bazel build //service/stovepipe/server:stovepipe`
- `make fmt`
- `make gazelle`

## Revert Plan
- Revert this PR to remove record-DLQ registration and restore the
single record-controller identity.

v0.3.0-20260820034428-9f8a3ef02bf0

Toggle v0.3.0-20260820034428-9f8a3ef02bf0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(demo): add BURST mode for a simultaneous enqueue of independent …

…changes (#615)

## Summary
Add -burst (Makefile BURST=false) which, for independent changes,
creates every change first and only then fires all the Land calls
together, so the requests reach the queue in one burst instead of being
enqueued as each is created. It works for every provider: burst runs
each source through the same two phases, so fake creates instantly,
github creates concurrently, git creates serially on its single work
tree — and in all three the enqueues arrive together once creation is
done.

Burst does not make creation faster; with the git source it is still
serialized on one work tree. It only separates creation from enqueuing
so the publishes are not spread across the creation phase. The default
is unchanged — each change lands the moment it exists, so the queue
starts working during creation.

A small lander seam lets the two-phase ordering be tested without a live
gateway.

## Test Plan

✅ New unit test asserts nothing is enqueued until every change is
created, provider-agnostically (a `lander` seam + a recording source
stand in for any provider); shape banner covered; existing
`//service/submitqueue/demo/requests` tests green.

## Stack
1. @ #615
1. #616

v0.3.0-20260820034215-0892fcf10b32

Toggle v0.3.0-20260820034215-0892fcf10b32's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat(stovepipe): dlq controller for build step (#618)

## Summary
### Intent

- Prevent build-stage failures from leaving requests processing and
consuming queue capacity indefinitely.

### Changes

- Add a build-stage DLQ controller that resolves the request from the
dead-lettered build payload and drives it to a conservative failed
outcome.
- Follow the established SubmitQueue DLQ controller naming and
construction conventions.
- Register the build DLQ topic, subscription, and controller with the
always-retryable reconciliation consumer.
- Cover successful reconciliation, malformed payloads, and empty request
identifiers.

## Test Plan
- `aifx verify`
- `./tool/bazel test //stovepipe/controller/dlq:go_default_test
--test_output=errors`
- `./tool/bazel build //service/stovepipe/server:stovepipe`
- `make lint`
- `make check-gazelle`

## Revert Plan
- Revert this PR to remove build-stage DLQ reconciliation and its topic
registration.

## Issues

Back | FazBrowse Home | New Git URL