| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
feat(stovepipe): Consume hook events (#632) ## Summary **What**: - Register the hook stage and its dead-letter stage in Stovepipe's consumer wiring, on a topic named for the domain, with a resolver that hands every event to the no-op hook. **Why**: - Wire the stage in a second domain, so attaching a real integration to Stovepipe is a resolver swap rather than new plumbing, and two domains on one queue backend keep their own hook topics. ## Test Plan - [x] Add unit tests. ## Revert Plan - Revert this PR. Nothing publishes hook events yet and the resolver returns only a no-op hook. ## Issues - [CODEM-416](https://linear.app/uber/issue/CODEM-416/hooks-integration-downstream-notificaiton) ## Stack 1. #607 1. #608 1. @ #632
feat(orch): Consume hook events (#608) ## Summary **What**: - Register the orchestrator's hook stage, so lifecycle events are consumed from the queue and handed to the integrations each event resolves to. No integration is wired yet, so events are accepted and discarded. - Register the matching dead-letter stage, so an event that fails every retry is recorded and alerted on instead of accumulating unread. **Why**: - Prove the delivery path end to end before anything produces an event, so a later change adds producers to a route that already works. - Keep a deployment with no integrations configured distinguishable from one that is silently losing events. ## Test Plan - [x] Add unit tests. ## Revert Plan - Revert this PR. Since nothing publishes hook events yet and the resolver returns only a no-op hook, no side effect is lost. ## Issues - [CODEM-416](https://linear.app/uber/issue/CODEM-416/hooks-integration-downstream-notificaiton) ## Stack 1. #607 1. @ #608 1. #632
feat(hook): Deliver events to integrations (#607) ## Summary **What**: - Consume lifecycle events from a durable queue and hand each one to the integrations it resolves to, running them concurrently and retrying then dead-lettering the ones that keep failing. - Record every dead-lettered event with its full payload and failure reason, and count it on a metric meant to page. **Why**: - Keep a slow or broken integration from stalling or failing the work that triggered it. - Make a lost notification recoverable and impossible to miss, since nothing else in the system notices when one goes missing. ## Test Plan - [x] Add unit tests. ## Revert Plan - Revert this PR. No service constructs the controller or subscribes to the hook topic. ## Issues - [CODEM-416](https://linear.app/uber/issue/CODEM-416/hooks-integration-downstream-notificaiton)
docs: align READMEs and RFCs with current code and AGENTS.md conventi… …ons (#629) Updates the documentation to match the actual repository layout and code: - Replaces all stale CLAUDE.md references with AGENTS.md, the authoritative conventions file. - Fixes the project-layout description in README.md and submitqueue/README.md: shared code lives under platform/, not the now-removed top-level entity/extension/core/ directories. - Expands the empty submitqueue/orchestrator/README.md with the pipeline stages it runs. - Updates Stovepipe READMEs to reflect Ping + Ingest RPCs and the full ingest → process → build → buildsignal → record pipeline, plus DLQ reconcilers. - Corrects fully-qualified gRPC service names in service/README.md and TESTING.md to use the uber.* proto packages. - Corrects Runway topic names in doc/rfc/runway/workflow.md to match the actual TopicKey constants (merge-conflict-check, runway-merge, merge-conflict-check-signal, merge-signal). - Clarifies integration-test Makefile targets in TESTING.md and removes the non-existent integration-test-{service} placeholder. - Fixes the stale TODO line-number reference in modular-queue-wiring.md.
fix(errs): Retry transient grouped failures (#621) ## Summary **What**: - Retry work when any one of several failures reported together is transient instead of giving up on the first attempt. - Limit a classification attached to one grouped failure to that failure alone, instead of letting it decide the outcome for the others reported alongside it. **Why**: - Stop discarding failures that would have cleared on their own by keeping a step that fans work out to several handlers from being abandoned because one handler hit a momentary error. ## Test Plan - [x] Add unit tests. ## Revert Plan - Revert this PR. The changes in this PR are confined to the error classification framework. ## Issues - [CODEM-462](https://linear.app/uber/issue/CODEM-462/err-classification-framework-should-handle-bundled-errors)
fix(messagequeue): run queue garbage collection on busy partitions (#622 ) ## Why? Garbage collection of acknowledged `queue_messages` rows currently runs only on idle poll ticks. A continuously busy partition resets the GC counter whenever it delivers a message, so it can never reach the GC threshold. Under sustained traffic, acknowledged message rows can therefore accumulate indefinitely. This also means the message deduplication horizon can remain unnecessarily large on busy partitions. ## What? Run the existing garbage collection cadence on every poll tick rather than only idle ticks. The existing 100-tick throttle is preserved, so the per-partition GC frequency is not increased. The GC safety mechanism is also unchanged: garbage collection still uses the minimum acknowledged offset across consumer groups. Added: - unit regression coverage proving GC runs on busy ticks - MySQL integration coverage proving acknowledged rows are reclaimed under continuous traffic ## Testing - `go test ./platform/extension/messagequeue/...` — PASS - `go test ./platform/base/...` — PASS - `go vet` on the messagequeue and integration packages — PASS - `gofmt` and `git diff --check` — clean Not run locally due to environment limitations: - MySQL integration test `TestGCReclaimsAckedRowsUnderContinuousTraffic` — Docker daemon unavailable - `make gazelle`, `make fmt`, `make lint`, `make check-tidy`, `make check-gazelle` and Bazel tests — `make` unavailable and Bazel's Windows C++ toolchain is not configured
fix(orchestrator): carry a failed batch's reason to the request error… … log (#616) ## Summary ### Why? When a batch failed, every request in it terminated as `error` with an empty message: mergesignal and speculate recorded only the terminal batch state, and conclude terminated the request with an empty reason — so a "did not land: error" carried no diagnostic at all. The reason is known at the failure site, but the requests are terminated later, in conclude. ### What? Carry the failure reason to conclude on the fan-out message's metadata rather than persisting it as batch state. mergesignal stamps the merge result's reason (or "merge failed" when it is empty) and speculate stamps "no speculation path could pass"; conclude reads it from the message and passes it to TerminateRequest, which already writes it to the request's terminal log. The landed and cancelled paths carry no reason, exactly as before. This mirrors the dead-letter path, which already sources its reason from the delivery context. Message metadata is durably persisted and redelivered by the queue backend, so the reason survives at-least-once retries — and the batch entity stays purely structural, with no new field and no schema migration. A `publish.MessageWithMetadata` helper and a shared `topickey.MetadataKeyFailureReason` constant carry the wiring. The demo client's run summary also surfaces the reason, so a failed request reads `id=error: <reason>` instead of a bare `id=error`. ## Test Plan ✅ `bazel test //submitqueue/orchestrator/...` (14/14). Updated tests assert the reason rides the conclude message metadata (mergesignal, speculate finalize) and reaches the request's terminal log (conclude). Broad build and CI gates (gazelle / tidy / mocks / fmt) clean. ## Stack 1. #615 1. @ #616
feat(stovepipe): recover record DLQ work (#623) ## Summary ### Intent - Complete Stovepipe DLQ coverage for failures that occur while recording a completed request. - Reuse the existing idempotent record reconciliation path instead of introducing a separate controller or request lifecycle state. - This PR builds on #619, which aligns the Stovepipe DLQ controllers with repository conventions. ### Changes - Register the existing record controller for the `record_dlq` topic with a distinct consumer group. - Derive record controller identity from its topic key so primary and DLQ instances have separate logging and metrics. - Cover the record-DLQ controller identity configuration. ## Test Plan - `./tool/bazel test //stovepipe/controller/record:go_default_test --test_output=errors` - `./tool/bazel build //service/stovepipe/server:stovepipe` - `make fmt` - `make gazelle` ## Revert Plan - Revert this PR to remove record-DLQ registration and restore the single record-controller identity.
feat(demo): add BURST mode for a simultaneous enqueue of independent … …changes (#615) ## Summary Add -burst (Makefile BURST=false) which, for independent changes, creates every change first and only then fires all the Land calls together, so the requests reach the queue in one burst instead of being enqueued as each is created. It works for every provider: burst runs each source through the same two phases, so fake creates instantly, github creates concurrently, git creates serially on its single work tree — and in all three the enqueues arrive together once creation is done. Burst does not make creation faster; with the git source it is still serialized on one work tree. It only separates creation from enqueuing so the publishes are not spread across the creation phase. The default is unchanged — each change lands the moment it exists, so the queue starts working during creation. A small lander seam lets the two-phase ordering be tested without a live gateway. ## Test Plan ✅ New unit test asserts nothing is enqueued until every change is created, provider-agnostically (a `lander` seam + a recording source stand in for any provider); shape banner covered; existing `//service/submitqueue/demo/requests` tests green. ## Stack 1. @ #615 1. #616
feat(stovepipe): dlq controller for build step (#618) ## Summary ### Intent - Prevent build-stage failures from leaving requests processing and consuming queue capacity indefinitely. ### Changes - Add a build-stage DLQ controller that resolves the request from the dead-lettered build payload and drives it to a conservative failed outcome. - Follow the established SubmitQueue DLQ controller naming and construction conventions. - Register the build DLQ topic, subscription, and controller with the always-retryable reconciliation consumer. - Cover successful reconciliation, malformed payloads, and empty request identifiers. ## Test Plan - `aifx verify` - `./tool/bazel test //stovepipe/controller/dlq:go_default_test --test_output=errors` - `./tool/bazel build //service/stovepipe/server:stovepipe` - `make lint` - `make check-gazelle` ## Revert Plan - Revert this PR to remove build-stage DLQ reconciliation and its topic registration. ## Issues
| Back | FazBrowse Home | New Git URL |