FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix: deep eval adapter to extract multi-turn conversations from service-normalized SESSION format by ybdarrenwang · Pull Request #622 · aws/bedrock-agentcore-sdk-python · GitHub

fix: deep eval adapter to extract multi-turn conversations from service-normalized SESSION format - #622

Open
ybdarrenwang wants to merge 2 commits into
aws:mainfrom
ybdarrenwang:fix/deepeval-adapter
Open

fix: deep eval adapter to extract multi-turn conversations from service-normalized SESSION format#622
ybdarrenwang wants to merge 2 commits into
aws:mainfrom
ybdarrenwang:fix/deepeval-adapter

Conversation

ybdarrenwang commented Aug 5, 2026
edited
Loading

Copy link
Copy Markdown

Background: PR #568

PR #568 (feat: third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers) introduced the DeepEvalAdapter — a generic wrapper that takes any DeepEval metric and runs it inside an AgentCore Lambda evaluator. The adapter uses strands-evals mappers to auto-detect span formats, extract fields, and construct the appropriate DeepEval test case (LLMTestCase for single-turn, ConversationalTestCase for multi-turn).

The Bug

  1. At SESSION level, the AgentCore eval service collapses all spans sharing a session.id into one span with multiple span_events (one per turn):
Session
 └─ Trace 1 → ONE span
                └─ span_events: [ {body: turn1}, {body: turn2}, {body: turn3} ]
  1. strands-evals CloudWatchSessionMapper folds that one span into a single AgentInvocationSpan (first user prompt, last assistant response),
  2. so _session_to_span_map_result() yields only 2 turn entries
  3. the len(turns) > 2 guard nulls it
  4. conversational metrics raise "only a single turn was found."

PR #568's _extract_from_service_format() fallback couldn't help: it only parsed single-turn gen_ai.* events, and map_spans() only invokes it when the mapper yields no usable input/output — which never happens here, so any multi-turn logic added to it was unreachable.

Impact: Every multi-turn metric failed at SESSION level because no turns were extracted at all, forcing users to bypass the adapter and write custom conversational handlers.

The Fix

Two parts, in registry.py:

  1. Reachability: map_spans() now detects the collapsed multi-event shape via a new _has_multi_event_span() helper (any span with >1 span_events) and runs the service-format extraction to populate turns even when the mapper already produced a valid single-turn input/actual_output.
  2. Correct decoding: _extract_from_service_format() gained a multi-turn branch that iterates span_events[*].body, and its _extract_message_text() helper now reuses CloudWatchSessionMapper's text parsing (_extract_text_from_content). This decodes the real double-encoded content (a JSON string like '[{"text": ...}]') into plain text instead of returning it literally, and selects the latest message per role, consistent with the CloudWatch single-turn path (PR fix: preserve multi-turn history in Strands ConversationTurn #454).

Backward-compatible: the single-turn gen_ai.* events path is unchanged; the multi-turn path activates only when >1 span_event is present.

Scope

This PR restores turn extraction (role + content) from the service-normalized SESSION format. That is sufficient for the multi-turn metrics that only need the conversation turns:

  • GoalAccuracy, RoleAdherence, ConversationCompleteness, KnowledgeRetention, TopicAdherence, TurnRelevancy, ConversationalGEval

Out of Scope / Known Limitation (follow-up)

Metrics that need per-turn side data are NOT fixed by this PR, because the service-normalized span_events[*].body format has no slot for that data — only input.messages and output.messages:

  • retrieval_context → TurnContextualPrecision, TurnContextualRecall, TurnFaithfulness
  • tools_called → ToolUse

_build_conversational_test_case() constructs Turn(role, content) only, so even if the body carried these fields, they would be dropped. Fixing this requires a format convention for where per-turn retrieval_context / tools_called live in span_events[*].body (e.g. body.retrieval_context, body.tools), then threading them through SpanMapResult.turns[i] into the Turn(...) constructor. Tracked as a follow-up.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

ybdarrenwang requested a review from a team August 5, 2026 17:16
github-actions Bot added the size/m PR size: M label Aug 5, 2026
ybdarrenwang changed the title Fix DeepEvalAdapter to extract multi-turn conversations from service-normalized SESSION format fix: DeepEvalAdapter to extract multi-turn conversations from service-normalized SESSION format Aug 5, 2026
github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 5, 2026

stone-coding left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Reviewed PR. The fix correctly handles the SESSION-level service-normalized format where multiple turns are collapsed into span_events[*].body. My original _extract_from_service_format() only covered single-turn gen_ai events.

ybdarrenwang force-pushed the fix/deepeval-adapter branch from 699048e to 338de68 Compare August 6, 2026 19:50
github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 14, 2026
github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 24, 2026
github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 24, 2026
github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 24, 2026
ybdarrenwang changed the title fix: DeepEvalAdapter to extract multi-turn conversations from service-normalized SESSION format fix: deep eval adapter to extract multi-turn conversations from service-normalized SESSION format Aug 24, 2026
github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants


Back | FazBrowse Home | New Git URL