1. Home
2. Run Agents
3. Evaluation

Environment simulation for evaluations¶

When evaluating agents that rely on external dependencies such as APIs, databases, or third-party services running those tools live during testing can be slow, costly, or unreliable. The Environment Simulator lets you safely intercept these tool calls during agent execution and replace them with controlled, deterministic responses, without modifying the agent itself. This approach can fill a critical gap in the agent improvement loop, allowing you to create hermetic, offline test runs that isolate your agent logic for reliable scoring.

Overall, this feature lets you:

• Test how an agent handles API errors or edge-case responses.
• Run evaluations offline, without access to live backends.
• Generate realistic mock responses automatically using an LLM.
• Produce reproducible test runs by seeding probabilistic injections.

The Environment Simulation integrates with ADK's tool execution pipeline via the before_tool_callback hook or the plugin system, so no changes to your agent code are required.

The Environment Simulation is an experimental feature. Its API may change in future
releases.

How it works¶

While User Simulation drives the conversation forward, Environment Simulation provides the stable backend. At a high level, the Environment Simulator sits between your agent and its tools. When the agent calls a tool, the simulator intercepts the call and decides whether to return a synthetic response either a predefined injection or an LLM-generated mock or to let the real tool execute.

The decision logic follows this order for each configured tool:

1. Injection configs are checked first, in order. If a matching injection is found (based on argument matching and probability), its error or response is returned immediately.
2. Mock strategy is used as a fallback if no injection config applies. The simulator calls an LLM to generate a realistic response based on the tool's schema and any stateful context.
3. No-op is returned (None) if the tool is not in the simulator config, allowing the real tool to execute normally.

Integration¶

The EnvironmentSimulationFactory class provides two integration points:

• create_callback() Returns an async callable suitable for use as a before_tool_callback on any LlmAgent.
• create_plugin() Returns an EnvironmentSimulationPlugin instance that integrates with the ADK plugin system.

Using as a callback¶

The following example shows how to create an environment simulation as one of the adk agent callbacks.

from google.adk.agents import LlmAgent
from google.adk.tools.environment_simulation import EnvironmentSimulationFactory
from google.adk.tools.environment_simulation.environment_simulation_config import (
    EnvironmentSimulationConfig,
    InjectedError,
    InjectionConfig,
    ToolSimulationConfig,
)

config = EnvironmentSimulationConfig(
    tool_simulation_configs=[
        ToolSimulationConfig(
            tool_name="get_user_profile",
            injection_configs=[
                InjectionConfig(
                    injected_error=InjectedError(
                        injected_http_error_code=503,
                        error_message="Service temporarily unavailable.",
                    )
                )
            ],
        )
    ]
)

agent = LlmAgent(
    name="my_agent",
    model="gemini-flash-latest",
    tools=[get_user_profile],
    before_tool_callback=EnvironmentSimulationFactory.create_callback(config),
)

Using as a plugin¶

The following example shows how to create environment simulation as an ADK agent plugin.

from google.adk.apps import App
from google.adk.tools.environment_simulation import EnvironmentSimulationFactory
from google.adk.tools.environment_simulation.environment_simulation_config import (
    EnvironmentSimulationConfig,
    MockStrategy,
    ToolSimulationConfig,
)

config = EnvironmentSimulationConfig(
    tool_simulation_configs=[
        ToolSimulationConfig(
            tool_name="search_products",
            mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
        )
    ]
)

app = App(
    name="my_app",
    root_agent=my_agent,
    plugins=[EnvironmentSimulationFactory.create_plugin(config)],
)

Configuration reference¶

You can configure the Environment Simulator with a set of dataclasses. The following sections provide a detailed reference for each configuration object.

EnvironmentSimulationConfig¶

The top-level configuration object.

ToolSimulationConfig¶

Defines how a single named tool should be simulated.

InjectionConfig¶

Controls a single synthetic response that can be injected into a tool call. Exactly one of injected_error or injected_response must be set.

InjectedError¶

Defines an HTTP-style error response.

MockStrategy¶

Enum controlling how the simulator generates responses when no injection fires.

Injection mode¶

Use injection configs to test specific failure or edge-case scenarios. Injections are evaluated in list order; the first one whose match_args criteria are met (and whose probability check passes) is applied.

Injecting errors¶

The following example shows how to inject errors with specific error code and error message to the agent.

from google.adk.tools.environment_simulation.environment_simulation_config import (
    InjectedError,
    InjectionConfig,
    ToolSimulationConfig,
)

ToolSimulationConfig(
    tool_name="charge_payment",
    injection_configs=[
        InjectionConfig(
            injected_error=InjectedError(
                injected_http_error_code=402,
                error_message="Payment declined.",
            )
        )
    ],
)

The agent will receive {"error_code": 402, "error_message": "Payment declined."} instead of a real tool result, allowing you to evaluate how the agent handles payment failures.

Injecting fixed responses¶

Use the following InjectionConfig to specify a success response with fixed response payload.

InjectionConfig(
    injected_response={"status": "ok", "order_id": "ORD-9999"}
)

Conditional injection with argument matching¶

Use match_args to inject only when specific arguments are passed.

InjectionConfig(
    match_args={"item_id": "ITEM-404"},
    injected_error=InjectedError(
        injected_http_error_code=404,
        error_message="Item not found.",
    ),
)

Here, the error is injected only when the tool is called with item_id="ITEM-404". All other calls pass through to the next injection config or to the mock strategy.

Probabilistic injection¶

Set injection_probability to a value between 0.0 and 1.0 to simulate flaky behavior. For reproducible test runs, pin the random outcome with random_seed.

InjectionConfig(
    injection_probability=0.3,
    random_seed=42,
    injected_error=InjectedError(
        injected_http_error_code=500,
        error_message="Internal server error.",
    ),
)

Injecting latency¶

Use injected_latency_seconds to simulate slow backend responses, useful for testing timeout handling or user experience under degraded conditions.

InjectionConfig(
    injected_latency_seconds=5.0,
    injected_response={"result": "slow but successful"},
)

Combining multiple injection configs¶

Multiple injection configs on a single tool are checked in order. You can combine them to test multiple scenarios:

ToolSimulationConfig(
    tool_name="get_inventory",
    injection_configs=[
        # Always fail for a specific out-of-stock item
        InjectionConfig(
            match_args={"sku": "OOS-001"},
            injected_response={"quantity": 0, "available": False},
        ),
        # Randomly fail 20% of the time for all other items
        InjectionConfig(
            injection_probability=0.2,
            random_seed=7,
            injected_error=InjectedError(
                injected_http_error_code=503,
                error_message="Inventory service unavailable.",
            ),
        ),
    ],
)

Mock strategy mode¶

When you want the simulator to generate plausible responses automatically rather than returning hand-crafted values use MOCK_STRATEGY_TOOL_SPEC.

The simulator uses an LLM to:

1. Analyze the schemas of all tools the agent has access to, and identify stateful dependencies between them (e.g., a create_order tool produces an order_id that get_order consumes).
2. Track a state store of IDs and resources created during the session.
3. Generate a response that is consistent with the tool's schema and the current state returning a 404-style error if a consuming tool requests a resource that was never created.

from google.adk.tools.environment_simulation.environment_simulation_config import (
    EnvironmentSimulationConfig,
    MockStrategy,
    ToolSimulationConfig,
)

config = EnvironmentSimulationConfig(
    tool_simulation_configs=[
        ToolSimulationConfig(
            tool_name="create_order",
            mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
        ),
        ToolSimulationConfig(
            tool_name="get_order",
            mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
        ),
        ToolSimulationConfig(
            tool_name="cancel_order",
            mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
        ),
    ]
)

With this config, the simulator will automatically generate an order_id when create_order is mocked, and use it to return consistent results (or a not-found error) when get_order or cancel_order are subsequently called.

Providing environment data¶

Pass domain-specific context through environment_data to make mock responses more realistic. This can be a JSON string representing a snapshot of your database or any structured context the LLM should use when generating responses.

import json

db_snapshot = {
    "products": [
        {"id": "P-001", "name": "Wireless Headphones", "price": 79.99, "stock": 12},
        {"id": "P-002", "name": "USB-C Hub", "price": 34.99, "stock": 0},
    ],
    "warehouse_location": "US-WEST-2",
}

config = EnvironmentSimulationConfig(
    tool_simulation_configs=[
        ToolSimulationConfig(
            tool_name="search_products",
            mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
        ),
    ],
    environment_data=json.dumps(db_snapshot),
)

The LLM will use this data to return product names, prices, and stock levels that match your domain, rather than generating arbitrary placeholder values.

Providing tracing data¶

Feed traces generated in the agent to be mocked through tracing to make mock responses more realistic.

import json

agent_traces = [
    {
        "invocation_id": "inv-001",
        "user_content": {"role": "user", "parts": [{"text": "Search for high-end headphones"}]},
        "intermediate_data": {
            "tool_uses": [
                {
                    "name": "search_products",
                    "args": {"query": "high-end headphones"},
                    "response": {"products": [{"id": "P-123", "name": "Premium Wireless ANC Headphones"}]}
                }
            ]
        }
    }
]

config = EnvironmentSimulationConfig(
    tool_simulation_configs=[
        ToolSimulationConfig(
            tool_name="search_products",
            mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
        ),
    ],
    tracing=json.dumps(agent_traces),
)

The LLM will use this data to return product names, prices, and stock levels that match your domain, rather than generating arbitrary placeholder values.

Mixing injections and mock strategy¶

Injection configs and a mock strategy can be combined on the same tool. Injections are always checked first; the mock strategy fires only when no injection applies.

ToolSimulationConfig(
    tool_name="send_notification",
    injection_configs=[
        # Always fail for a known-bad recipient
        InjectionConfig(
            match_args={"recipient_id": "INVALID"},
            injected_error=InjectedError(
                injected_http_error_code=400,
                error_message="Invalid recipient.",
            ),
        ),
    ],
    # For all other recipients, generate a plausible success response
    mock_strategy_type=MockStrategy.MOCK_STRATEGY_TOOL_SPEC,
)