fix(memory): wrap boto3 clients in context-isolating proxy to silence OTel detach errors - #555
Open
citizen204 wants to merge 1 commit into
Open
fix(memory): wrap boto3 clients in context-isolating proxy to silence OTel detach errors#555citizen204 wants to merge 1 commit into
citizen204 wants to merge 1 commit into
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
… OTel detach errors When MemoryClient is used from an asyncio task that already has an OTel context attached (e.g. via aws-opentelemetry-distro), the boto3 auto-instrumentation calls opentelemetry.context.attach() then detach() around every API call. If the call crosses an asyncio task or thread-pool boundary, Python raises: ValueError: <Token> was created in a different Context because ContextVar.reset() only accepts tokens created in the current execution context. The error is logged at ERROR level by the OTel runtime on every boto3 call, polluting CloudWatch and obscuring real failures. _ContextIsolatingProxy wraps each method call in contextvars.copy_context().run() so the OTel attach/detach pair is fully contained within the copied context snapshot -- reset() always matches the token created in the same context. Applies to both gmdp_client and gmcp_client at construction time, covering every call site in MemoryClient and AgentCoreMemorySessionManager without any per-call changes. Fixes aws#456
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with 
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
citizen204
had a problem deploying
to
manual-approval
June 27, 2026 08:59 — with
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When MemoryClient is used inside an asyncio application with aws-opentelemetry-distro (or any OTel boto3 auto-instrumentation), every data-plane or control-plane call emits an ERROR-level log:
This happens because the OTel boto3 instrumentation calls context.attach() before the API call and context.detach() after. Python's ContextVar.reset() raises ValueError when the token was created in a different execution context -- which occurs when the SDK is used from an asyncio task that inherited OTel context from a parent task, or when a ThreadPoolExecutor worker crosses await boundaries.
At scale (many concurrent agents, AgentCoreMemorySessionManager with asyncio.to_thread()), this fills CloudWatch with spurious ERROR entries that obscure real failures.
Fix
Added _ContextIsolatingProxy -- a thin wrapper around a boto3 client that runs each method call inside contextvars.copy_context().run(). This creates an isolated context snapshot so the OTel attach/detach pair is always contained within the same context, making ContextVar.reset() succeed.
Both gmdp_client (data plane) and gmcp_client (control plane) are wrapped at construction time in MemoryClient.__init__. This fixes every call site -- including direct self.gmdp_client.create_event(...) calls in session_manager.py -- without per-call changes.
The proxy exposes meta directly (for meta.region_name logging) and wraps all callable attributes; non-callable attributes are returned as-is.
Changes
Fixes #456