FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix(cli): stop unbounded debug trace retention on the api server by a2105z · Pull Request #6916 · google/adk-python · GitHub

fix(cli): stop unbounded debug trace retention on the api server - #6916

Open
a2105z wants to merge 1 commit into
google:mainfrom
a2105z:fix/api-server-unbounded-trace-retention
Open

fix(cli): stop unbounded debug trace retention on the api server#6916
a2105z wants to merge 1 commit into
google:mainfrom
a2105z:fix/api-server-unbounded-trace-retention

Conversation

a2105z commented Aug 26, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
get_fast_api_app(web=False) unconditionally installed ApiServerSpanExporter and InMemoryExporter on the process tracer. Those buffers store full llm_request / llm_response attributes and every span, with no eviction. The debug endpoints that read them (/dev/apps/.../debug/trace) are only mounted with web=True, so a production Cloud Run / GKE process pays unbounded memory for buffers nobody reads. Measured ~200 KB/LLM turn and OOM about every 6 days.

Solution:

  • Do not register the internal exporters on ApiServer (web=False). DevServer (web=True / adk web) still registers them.
  • FIFO-cap the event dict (512), span deque (4096), and session index (512) so a long-lived adk web session cannot retain every span either.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
uv run python -m pytest tests/unittests/cli/test_fast_api.py::test_api_server_span_exporter_records_only_llm_and_tool_spans tests/unittests/cli/test_fast_api.py::test_api_server_span_exporter_skips_span_without_event_id tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_returns_only_spans_of_requested_session tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_falls_back_to_conversation_id tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_clear_drops_spans_but_keeps_session_index tests/unittests/cli/test_fast_api.py::test_api_server_span_exporter_evicts_oldest_events tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_evicts_oldest_spans tests/unittests/cli/test_fast_api.py::test_in_memory_exporter_evicts_oldest_sessions tests/unittests/cli/test_fast_api.py::test_debug_trace_exporters_only_registered_for_dev_server tests/unittests/cli/test_fast_api.py::test_dev_only_endpoints_absent_when_web_disabled tests/unittests/cli/test_fast_api.py::test_debug_trace -q
11 passed

Manual End-to-End (E2E) Tests:

from google.adk.cli.fast_api import get_fast_api_app
from opentelemetry import trace

app = get_fast_api_app(agents_dir="agents", web=False)
tp = trace.get_tracer_provider()
procs = tp._active_span_processor._span_processors
print([type(p.span_exporter).__name__ for p in procs])
# no ApiServerSpanExporter / InMemoryExporter

adk web still serves /dev/apps/{app}/debug/trace/{event_id} (covered by test_debug_trace).

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

get_fast_api_app(web=False) still installed ApiServerSpanExporter and
InMemoryExporter even though the debug endpoints that read them are only
mounted with web=True. Those buffers retain every span (including full
llm_request / llm_response) for the life of the process and OOM
long-lived Cloud Run / GKE instances.

Register the exporters only on DevServer, and FIFO-cap the event dict,
span deque, and session index so adk web cannot grow without bound either.

Closes google#6915
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant


Back | FazBrowse Home | New Git URL