| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
Temporal serializes start_workflow / execute_activity across (potentially cross-process) boundaries and does not carry the active W3C traceparent, so spans created inside a workflow or activity become detached roots -- the trace shatters at every Temporal hop. This bites agentex directly: adk.tracing.span creates the business span as a Temporal activity when in_temporal_workflow(), so without propagation those spans detach from the turn's obs trace. Wire temporalio's first-party TracingInterceptor onto both Temporal client factories (worker client + the ACP's workflow-starting TemporalClient) and the AgentexWorker, so client -> workflow -> activity is one trace. The interceptor injects context on outbound calls and extracts + roots execution spans under it, using the global OpenTelemetry propagator. - ENABLED BY DEFAULT. Opt out with AGENTEX_TEMPORAL_TRACE_INTERCEPTOR_ENABLED=false (also 0/no/off). Safe no-op (never raises) if temporalio's OTel contrib isn't importable, so default-on can't break a worker. - Tracing interceptor is placed OUTERMOST on the worker so existing business interceptors (and their spans) nest under the propagated span. Tests: tests/lib/core/tracing/test_temporal_interceptor.py -- default-on returns a TracingInterceptor, env opt-out returns [], contrib-missing returns []. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| interceptors=self.interceptors, # Pass interceptors to Worker | ||
| # Tracing interceptor OUTERMOST so business interceptors (and the spans | ||
| # they create) nest under the propagated workflow/activity span. | ||
| interceptors=[*temporal_tracing_interceptors(), *self.interceptors], |
There was a problem hiding this comment.
This double installs the tracing interceptor on the worker. Worker.init prepends client interceptors that also implement temporalio.worker.Interceptor, and TracingInterceptor implements both, so the instance already on this worker's client (from get_temporal_client above) applies to the worker automatically. With this line the worker runs two instances. Verified with a local repro on temporalio 1.26.0: every worker side span (RunWorkflow, CompleteWorkflow, StartActivity, RunActivity) is emitted twice, while client only wiring emits each span once with propagation intact.
Suggest interceptors=self.interceptors. The inherited instance is prepended, so tracing still sits outermost ahead of the business interceptors.
Sorry, something went wrong.
start_span/end_span run as SEPARATE Temporal activities (START_SPAN/END_SPAN) that Temporal can route to different worker processes. The obs wrapper handle is stored in a process-local module dict, so on a multi-replica fleet the END lands on a different worker than the START: the handle is never popped (leak / OOM risk) and the wrapper span is never ended (dangling obs_span_id in Tempo). Inside a Temporal activity, skip opening our own wrapper and instead stamp the reverse tag onto the interceptor-propagated ambient span (tag_ambient_obs_span) and read forward ids via obs_correlation(). Trace-level correlation is preserved via the Temporal OTel TracingInterceptor (#485); the per-step named wrapper and TurnTrace RETRY/ASYNC roll-up are deferred (see TODO(obs-followup)). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
start_span/end_span run as SEPARATE Temporal activities (START_SPAN/END_SPAN) that Temporal can route to different worker processes. The obs wrapper handle is stored in a process-local module dict, so on a multi-replica fleet the END lands on a different worker than the START: the handle is never popped (leak / OOM risk) and the wrapper span is never ended (dangling obs_span_id in Tempo). Inside a Temporal activity, skip opening our own wrapper and instead stamp the reverse tag onto the interceptor-propagated ambient span (tag_ambient_obs_span) and read forward ids via obs_correlation(). Trace-level correlation is preserved via the Temporal OTel TracingInterceptor (#485); the per-step named wrapper and TurnTrace RETRY/ASYNC roll-up are deferred (see TODO(obs-followup)). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
| Back | FazBrowse Home | New Git URL |
Problem
Temporal breaks OpenTelemetry context propagation. A start_workflow / execute_activity call is serialized and dispatched to a (potentially different) worker process, and the active W3C traceparent is not carried across that boundary. So any span created inside a workflow or activity becomes a new detached root — the trace shatters at every Temporal hop.
This bites agentex directly: adk.tracing.span branches on in_temporal_workflow() and, when true, creates the business span as a Temporal activity (TracingActivityName.START_SPAN). Without propagation, those business spans (and any downstream spans) detach from the turn's obs trace.
The existing ContextInterceptor threads business context (task_id) across the boundary — but nothing threads the obs-trace (W3C) context.
Change
Wire temporalio's first-party temporalio.contrib.opentelemetry.TracingInterceptor onto our Temporal client factories and worker, so client → workflow → activity is one trace. It injects the active span context into Temporal headers on outbound calls, and extracts + roots the workflow/activity execution spans under it, using the global OpenTelemetry propagator.
Wired at three points (both client factories + the worker):
Default-on with an opt-out
Enabled by default. Opt out with AGENTEX_TEMPORAL_TRACE_INTERCEPTOR_ENABLED=false (also accepts 0 / no / off). It also degrades to a no-op and never raises if temporalio's OpenTelemetry contrib isn't importable — so default-on can't break a worker. (adk already depends on opentelemetry-api/sdk, so the contrib is available in practice.)
Ordering (important)
On the worker the tracing interceptor is outermost:
TracingInterceptor ← continues/roots the workflow/activity span first └─ ContextInterceptor ← threads task_id, creates business spans as children └─ handlerIf reversed, business spans would be created before the trace context is active → back to detached roots.
Tests
tests/lib/core/tracing/test_temporal_interceptor.py — default-on returns a TracingInterceptor, env opt-out returns [], and the contrib-missing fallback returns [] (verified against the real temporalio contrib).
Risk
Low — additive, config-gated, no-op when disabled or when the contrib is absent; no workflow-determinism concern (TracingInterceptor is designed for the workflow sandbox). Independent of PR #484 (correlation edge) — branched off main.
Relation to platform tracing
This is the Temporal-boundary piece of "W3C propagation everywhere": once it lands, an agent turn (or any workflow) that fans out through Temporal activities stays one trace end-to-end, and the business/anchor spans created inside activities attach to the turn.
🤖 Generated with Claude Code
Greptile Summary
The PR enables OpenTelemetry trace-context propagation across Temporal boundaries.
Confidence Score: 5/5
The PR appears safe to merge.
No blocking failure remains; the constructor now executes inside the guarded try block, so ordinary constructor failures follow the documented no-op fallback instead of aborting Temporal startup.
Important Files Changed
Sequence Diagram
sequenceDiagram participant C as Temporal Client participant TW as Temporal Workflow Worker participant A as Temporal Activity C->>TW: start_workflow + injected OTel context activate TW TW->>A: execute_activity + propagated OTel context activate A A-->>TW: activity result deactivate A TW-->>C: workflow result deactivate TWReviews (2): Last reviewed commit: "feat(tracing): propagate OTel trace cont..." | Re-trigger Greptile