Stack (2/3)

1. feat(tracing): correlate business spans with active observability trace (feat(tracing): correlate business spans with active observability trace #465)
2. feat(tracing): establish W3C trace context at FastACP ingress ← this PR
3. feat(tracing): propagate trace context into Temporal via OTel interceptor

Based on obs/01-span-obs-correlation. Review/merge after #465.

What this PR does

Extends RequestIDMiddleware in base_acp_server.py to extract the inbound W3C trace context (traceparent/tracestate) and attach it as the current OpenTelemetry context for the lifetime of the request.

This gives obs_correlation() (PR 1) a real emit-time context: business spans created while handling an RPC — and any Temporal workflow started from that request (PR 3) — now adopt the caller's observability trace_id.

Why it's needed

PR 1 only reads the active obs context; without this, there is no context to read on the ACP request path (the SGP /v5/spans/batch transport doesn't carry it). This is the client-side propagation half.

Safety

Best-effort and lazy-imported: no traceparent header, or OTel unavailable, is a safe no-op and never breaks a request. Context is detached in a finally.

Verification

py_compile + functional smoke: a traceparent header yields a valid current span context with the expected trace_id; no header → invalid context (no-op).

Greptile Summary

This PR extends RequestIDMiddleware in base_acp_server.py to extract the inbound W3C trace context (traceparent/tracestate) and attach it as the active OpenTelemetry context for the lifetime of each HTTP request, enabling downstream business spans and Temporal workflows to share the caller's observability trace ID.

• Two small helper functions (_attach_incoming_trace_context, _detach_trace_context) are added; both are best-effort with lazy OTel imports so a missing opentelemetry package or a missing header is always a safe no-op.
• RequestIDMiddleware.__call__ is refactored from a guard-and-fall-through pattern to an early-return for non-HTTP scope types, and the trace context attach/detach is wrapped in a try/finally block alongside the existing request-ID logic.

Confidence Score: 4/5

Safe to merge — the change is additive, never raises, and the existing request-handling path is functionally unchanged in the absence of OTel.

The attach/detach lifecycle is correct and properly guarded with try/finally; asyncio.create_task inherits the contextvars copy at creation time so background tasks correctly retain the trace context. The one imprecision is the plain-dict carrier that silently drops all but the last value when a caller forwards multiple tracestate headers, which is spec-legal but uncommon in practice.

base_acp_server.py — the multi-value tracestate edge case in _attach_incoming_trace_context is the only area that warrants a second look.

Important Files Changed

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller as Caller (SGP/upstream)
    participant MW as RequestIDMiddleware
    participant OTel as OpenTelemetry
    participant App as FastAPI App
    participant BG as Background Task

    Caller->>MW: HTTP Request (traceparent header)
    MW->>MW: Extract x-request-id → ctx_var_request_id.set()
    MW->>OTel: extract(carrier) → W3C context
    OTel-->>MW: ctx (with trace_id from traceparent)
    MW->>OTel: context.attach(ctx) → token
    MW->>App: await self.app(scope, receive, send)
    App->>App: asyncio.create_task(_process_request) copies current context
    App-->>MW: response sent
    BG->>BG: runs with copied context (trace_id still active)
    MW->>OTel: context.detach(token) [finally]
    Note over MW,OTel: original context restored

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller as Caller (SGP/upstream)
    participant MW as RequestIDMiddleware
    participant OTel as OpenTelemetry
    participant App as FastAPI App
    participant BG as Background Task

    Caller->>MW: HTTP Request (traceparent header)
    MW->>MW: Extract x-request-id → ctx_var_request_id.set()
    MW->>OTel: extract(carrier) → W3C context
    OTel-->>MW: ctx (with trace_id from traceparent)
    MW->>OTel: context.attach(ctx) → token
    MW->>App: await self.app(scope, receive, send)
    App->>App: asyncio.create_task(_process_request) copies current context
    App-->>MW: response sent
    BG->>BG: runs with copied context (trace_id still active)
    MW->>OTel: context.detach(token) [finally]
    Note over MW,OTel: original context restored

Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
src/agentex/lib/sdk/fastacp/base/base_acp_server.py:62-67
**Duplicate `tracestate` headers silently dropped**

The plain-dict carrier overwrites duplicate keys, so if an intermediary forwards multiple `tracestate` headers (which the W3C Trace Context spec explicitly permits), all but the last entry are silently discarded. The OTel `DefaultGetter` calls `carrier.get(key)` on a plain dict, which can only return a single string value, so combined trace-state from several systems in the propagation chain would be partially lost.

To handle multi-value headers correctly, the carrier should concatenate repeated values with a comma separator before inserting them into the dict, or use a multi-value mapping with a custom `Getter`.

Reviews (1): Last reviewed commit: "feat(tracing): establish W3C trace conte..." | Re-trigger Greptile

Greptile also left 1 inline comment on this PR.