Snapshot record then reply testing - #740
Merged
mateusz834 merged 1 commit intoApr 27, 2026
Merged
mateusz834 merged 1 commit into
mateusz834 merged 1 commit into
Conversation
Copy link
Copy Markdown
Member
Sorry, something went wrong.
mateusz834
force-pushed
the
dev/MP/snapshots
branch
16 times, most recently
from
1198d47 to
d029c1d
Compare
April 24, 2026 06:45
mateusz834
marked this pull request as ready for review
April 24, 2026 06:45
mateusz834
force-pushed
the
dev/MP/snapshots
branch
from
d029c1d to
7f769bb
Compare
April 24, 2026 06:46
mateusz834
requested review from
Ickerday and
szykol
and removed request for
szykol
April 24, 2026 07:16
ghost
reviewed
Apr 24, 2026
Comment thread
tests/ai_testlib.py
Resolved
ghost
reviewed
Apr 24, 2026
Comment thread
tests/integration/ai/test_agent.py
Resolved
ghost
reviewed
Apr 24, 2026
Comment thread
tests/ai_testlib.py
Resolved
ghost
reviewed
Apr 24, 2026
ghost
left a comment
Copy link
Copy Markdown
There was a problem hiding this comment.
I really like the solution.
Sorry, something went wrong.
szykol
reviewed
Apr 24, 2026
Comment thread
tests/integration/ai/test_agent.py
Resolved
szykol
reviewed
Apr 24, 2026
Comment thread
tests/integration/ai/test_agent_mcp_tools.py
Resolved
szykol
reviewed
Apr 24, 2026
Comment thread
tests/ai_testlib.py
Resolved
szykol
reviewed
Apr 24, 2026
Comment thread
tests/ai_testlib.py
Resolved
szykol
reviewed
Apr 24, 2026
Comment thread
tests/ai_testlib.py
Resolved
szykol
reviewed
Apr 24, 2026
Comment thread
tests/ai_testlib.py
Resolved
mateusz834
force-pushed
the
dev/MP/snapshots
branch
2 times, most recently
from
32aadd8 to
6aa74cd
Compare
April 27, 2026 08:15
This change implements snapshot, record then reply testing for all integration tests. Re-enables recently skipped (falky) tests: - test_agent_understands_other_agents (snapshot was edited manually) - test_supervisor_resumes_subagent_thread_across_invocations - test_supervisor_resumes_subagent_thread_across_invocations_structured Introduces a deterministic thread_id mock generator, such that snapshots are deterministic, for: - test_supervisor_resumes_subagent_thread_across_invocations - test_supervisor_resumes_subagent_thread_across_invocations_structured Modified test_tool_execution_service_access using tool middleware, to make the test deterministic.
mateusz834
force-pushed
the
dev/MP/snapshots
branch
from
6aa74cd to
cd1a2ee
Compare
April 27, 2026 11:37
szykol
approved these changes
Apr 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change implements snapshot, record then reply testing for all integration tests.
Re-enables recently skipped (flaky) tests:
Introduces a deterministic thread_id mock generator, such that snapshots are deterministic, for:
Modified test_tool_execution_service_access using tool middleware, to make the test deterministic.
E2E tests still call the LLMs directly.