| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
There was a problem hiding this comment.
Reviewed PR. The fix correctly handles the SESSION-level service-normalized format where multiple turns are collapsed into span_events[*].body. My original _extract_from_service_format() only covered single-turn gen_ai events.
Sorry, something went wrong.
…-normalized SESSION format
| Back | FazBrowse Home | New Git URL |
Background: PR #568
PR #568 (feat: third-party eval metrics adapter (DeepEval + Autoevals) with strands-evals mappers) introduced the DeepEvalAdapter — a generic wrapper that takes any DeepEval metric and runs it inside an AgentCore Lambda evaluator. The adapter uses strands-evals mappers to auto-detect span formats, extract fields, and construct the appropriate DeepEval test case (LLMTestCase for single-turn, ConversationalTestCase for multi-turn).
The Bug
Session └─ Trace 1 → ONE span └─ span_events: [ {body: turn1}, {body: turn2}, {body: turn3} ]PR #568's _extract_from_service_format() fallback couldn't help: it only parsed single-turn gen_ai.* events, and map_spans() only invokes it when the mapper yields no usable input/output — which never happens here, so any multi-turn logic added to it was unreachable.
Impact: Every multi-turn metric failed at SESSION level because no turns were extracted at all, forcing users to bypass the adapter and write custom conversational handlers.
The Fix
Two parts, in registry.py:
Backward-compatible: the single-turn gen_ai.* events path is unchanged; the multi-turn path activates only when >1 span_event is present.
Scope
This PR restores turn extraction (role + content) from the service-normalized SESSION format. That is sufficient for the multi-turn metrics that only need the conversation turns:
Out of Scope / Known Limitation (follow-up)
Metrics that need per-turn side data are NOT fixed by this PR, because the service-normalized span_events[*].body format has no slot for that data — only input.messages and output.messages:
_build_conversational_test_case() constructs Turn(role, content) only, so even if the body carried these fields, they would be dropped. Fixing this requires a format convention for where per-turn retrieval_context / tools_called live in span_events[*].body (e.g. body.retrieval_context, body.tools), then threading them through SpanMapResult.turns[i] into the Turn(...) constructor. Tracked as a follow-up.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.