Summary

Makes SGP span export concurrent (bounded), removing the per-pod egress ceiling that caused trace backlogs under load. Follow-up to #362 (span queue linger + per-loop keepalive), which fixed ingest latency but left egress serial.

Findings — why egress couldn't keep up

A load test showed span export collapsing under 2× traffic: the agent's PUT rate dropped while the EGP backend sat idle, and the backlog only drained after the run ended. We chased this with the agent logs + Datadog metrics/APM:

• Not data loss. All spans arrive eventually (aside from a small % lost to backend 401s, which are a separate EGP auth issue, not this code). The DB got every row once the pods drained the backlog post-run.

• Not CPU. Mock-agent CPU sat at ~8–11% of limit during both the 1× and 2× runs, with zero CFS throttling. The event loop was not saturated.

• Not TLS / the HTTP client. Export goes over plain http:// to an in-cluster service (egp-api-backend.egp.svc.cluster.local) — sub-ms connection setup, no handshake. (The keepalive pool from perf(tracing): span queue linger + per-loop httpx keepalive #362 was correct but unused.)

• It's serial issuance vs. per-request latency. APM shows each PUT /public/v5/spans/batch takes ~150 ms p50 / ~330 ms p95 (server-side batch processing — matches the ~0.5 s batch p99 from the run), in both runs. The drain loop awaits each upsert_batch to completion before sending the next, so one request is in flight per pod at a time:

  per-pod egress ≈ 1 / (2 × 150 ms) × 2 PUTs ≈ ~6–7 PUT/s per pod, independent of offered load.

  EGP absorbs ~169 PUT/s in aggregate because it serves requests concurrently; the agent never issued them concurrently, so it couldn't use that capacity. At 2× the per-pod ceiling is exceeded → backlog grows → drains once production stops. This is a Little's Law cap: throughput = concurrency ÷ latency, with concurrency pinned at 1.

Change

The drain loop now dispatches each batch's export as its own task, bounded by AGENTEX_SPAN_QUEUE_CONCURRENCY (default 8), so multiple upsert_batch requests run at once. With ~150 ms latency this lifts the per-pod ceiling from ~6–7 PUT/s to ~50–70 PUT/s, reusing the keepalive pool #362 added. Setting concurrency=1 restores the old strictly-serial behavior.

Ordering is preserved

to_request_params() always sends output (null on a START), so a START landing after its END could null out the written output. The change keeps the per-span START-before-END invariant: END send-tasks snapshot the in-flight START tasks and await them before issuing. Because a span's START is always enqueued — and therefore dispatched (FIFO) — before its END, that span's START send is either still in flight (and waited on) or already complete. Independent spans export fully concurrently.

shutdown() still drains via queue.join() (task_done moved into the send-task) and cancels stragglers on timeout.

Test plan

• rye run pytest tests/lib/core/tracing/ — 52 passed, 2 skipped
• rye run ruff check / rye run pyright / ruff format --check — clean
• New tests cover: bounded concurrency (overlap capped at the limit), concurrency=1 serializes, concurrent drain is >2× faster than serial, and the per-span END-waits-for-START ordering invariant under concurrency.
• Re-run the 2× load test and confirm egress keeps up with production (no post-run backlog).

Config

Known limitation

When queue-level retry is enabled (AGENTEX_SPAN_QUEUE_MAX_RETRIES > 1, default 1 = off), a re-enqueued START goes to the back of the queue and may not be in a concurrently-dispatched END's barrier snapshot — a narrow window where a retried START could reorder against its END. Default config is unaffected. Worth tightening if we turn retries on by default.

Greptile Summary

This PR replaces the serial drain loop with bounded-concurrency span export: each batch's HTTP call is now dispatched as its own asyncio task (capped by AGENTEX_SPAN_QUEUE_CONCURRENCY, default 3), removing the per-pod egress ceiling that caused trace backlogs under 2× load.

• Concurrent dispatch with ordering guarantee: _dispatch spawns _run_send tasks and END tasks snapshot the live _inflight_starts set as a barrier, ensuring a span's START HTTP call completes before its END is issued — even across concurrent batches.
• Shutdown hardening: shutdown() now checks not self._inflight before the early exit, drives queue.join() to drain all in-flight (and retried) items, then cancels straggler tasks on timeout.
• Known limitation documented: retried STARTs re-queue to the back and may miss the END's barrier snapshot; a code-level note has been added to _reenqueue flagging this for if retries are ever enabled by default.

Confidence Score: 5/5

Safe to merge — the concurrent dispatch logic is correct, the START-before-END ordering invariant holds under all tested scenarios, and shutdown correctly awaits in-flight tasks.

The barrier mechanism correctly handles both same-batch and cross-batch START/END ordering: a completed START is already gone from _inflight_starts (no barrier needed), an in-flight START is captured by the snapshot and waited on. The _send_sema is the true HTTP concurrency control and prevents overrun regardless of processor count. The only active gap is the known re-enqueue ordering hole (documented in _reenqueue) which is inactive at the default max_retries=1.

No files require special attention; span_queue.py carries the complexity but is well-tested.

Important Files Changed

Comments Outside Diff (1)

1. src/agentex/lib/core/tracing/span_queue.py, line 343-345 (link)

   The PR description calls out a known ordering hole when max_retries > 1: a retried START goes to the back of the queue and won't be in a concurrently-dispatched END's barrier snapshot. Since this latent bug activates the moment someone enables retries, a code-level note in _reenqueue would make this visible without relying on the PR description.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/agentex/lib/core/tracing/span_queue.py
   Line: 343-345
   
   Comment:
   The PR description calls out a known ordering hole when `max_retries > 1`: a retried START goes to the back of the queue and won't be in a concurrently-dispatched END's barrier snapshot. Since this latent bug activates the moment someone enables retries, a code-level note in `_reenqueue` would make this visible without relying on the PR description.
   
   
   
   How can I resolve this? If you propose a fix, please make it concise.

   

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
src/agentex/lib/core/tracing/span_queue.py:28
**Default concurrency mismatch with PR description**

The PR description's config table documents the default as `8`, but the code sets `_DEFAULT_CONCURRENCY = 3`. The performance projections in the PR ("~50–70 PUT/s") are also based on `concurrency=8`; at the actual default of `3` the ceiling is closer to `~20 PUT/s`. If the choice was intentional (conservative ramp), a note here or in the PR description would prevent confusion for anyone tuning this later.

Reviews (4): Last reviewed commit: "perf(tracing): lower default span-export..." | Re-trigger Greptile