fix(ci): add pinned deepeval and autoevals to dev group - #609
Merged
jariy17 merged 1 commit intoJul 31, 2026
Merged
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
The deepeval adapter imports deepeval at module scope (src/.../third_party/deepeval/adapter.py:6), but deepeval was absent from the dev dependency group. ci.yml runs 'uv sync --dev', so the deepeval unit tests could not be collected, exiting pytest with code 2 and cancelling the whole matrix. Both are pinned with an upper bound so major versions cannot drift in. The deepeval extra's >=2.0.0 floor was also unsatisfiable: the adapter imports deepeval.test_case.conversational_test_case.Turn, which does not exist before 3.0.8, and deepeval <3.5.0 pins rich <14, conflicting with strands-agents-evals (rich >=14). Corrected to >=3.5.0.
Copy link
Copy Markdown
Contributor
|
Claude Security Review: no high-confidence findings. (run) |
Sorry, something went wrong.
tejaskash
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ci.yml unit tests have been failing on main since 9b7d38e (#568), on all four Python versions:
src/bedrock_agentcore/evaluation/custom_code_based_evaluators/third_party/deepeval/adapter.py:6 imports deepeval at module scope:
But deepeval was not in the dev dependency group, and ci.yml installs via uv sync --dev. So the tests could not even be collected — pytest exits 2, and fail-fast cancels the whole matrix.
The autoevals sibling adapter imports nothing at module scope (only inside functions), which is why only the deepeval tests error. That asymmetry is the underlying issue.
Change
Add both to the dev group, pinned with an upper bound so a new major version cannot drift in:
Also corrects an unsatisfiable floor in the deepeval extra since ConversationalTurn was introduced in v3
Testing
Follow-up worth considering
deepeval is declared optional in pyproject.toml, yet importing the adapter module hard-requires it. Making the deepeval adapter lazy-import like the autoevals one would mean an optional extra can't break collection again. Happy to file separately.
Separate from #608, which fixes the integration-test workflow.