# Practical and Useful Patterns with ADK 🧪 You can't hit production APIs during agent testing. ADK's Environment Simulation intercepts tool calls and replaces them with controlled responses, letting you reproduce edge cases without affecting production systems. 📌 Title: Environment Simulation 🔗 URL: 🧩 Overview ADK's Environment Simulator intercepts tool calls during agent execution and replaces them with controlled responses — all without modifying agent code. It supports error injection, fixed responses, conditional injection, probabilistic failures, and latency simulation. It integrates as a Callback or Plugin, with a decision priority of: injection configs, then mock strategy, then real tool execution. 🛠 Usage Integration as a Callback: Import `LlmAgent` from `google.adk.agents`, `EnvironmentSimulationFactory` from ` and `EnvironmentSimulationConfig`, `InjectedError`, `InjectionConfig`, `ToolSimulationConfig` from the `environment_simulation_config` submodule. Create an `EnvironmentSimulationConfig` with a list of `ToolSimulationConfig` entries in `tool_simulation_configs`. Each entry specifies a `tool_name` (e.g., `"get_user_profile"`) and `injection_configs` containing `InjectionConfig` instances -- for example, an `InjectedError` with `injected_http_error_code=503` and an `error_message`. Convert the config to a callback via `EnvironmentSimulationFactory.create_callback(config)` and pass it as the `before_tool_callback` parameter of `LlmAgent`. Fixed response injection: Set the `injected_response` parameter of `InjectionConfig` to a dictionary (e.g., `{"status": "ok", "order_id": "ORD-9999"}`) to return a fixed response for tool calls. Conditional injection based on arguments: Use `match_args` in `InjectionConfig` to specify an argument pattern (e.g., `{"item_id": "ITEM-404"}`). The injection triggers only when tool call arguments match. Pair it with `injected_error` set to `InjectedError(injected_http_error_code=404, error_message="Item not found.")` to return a 404 error for that specific argument. Probabilistic failure simulation: Set `injection_probability` on `InjectionConfig` to a value like `0.3` for a 30% failure rate, and `random_seed` (e.g., `42`) for reproducibility. Combined with `injected_error` set to `InjectedError(injected_http_error_code=500, error_message="Internal server error.")`, this randomly injects 500 errors at the specified probability. Latency simulation: Set `injected_latency_seconds` on `InjectionConfig` to the desired delay (e.g., `5.0` seconds) along with an `injected_response` (e.g., `{"result": "slow but successful"}`). The response is returned after the specified delay. Mock Strategy mode auto-generates responses from tool schemas: Import `MockStrategy` from `environment_simulation_config` and set `mock_strategy_type` to `MockStrategy.MOCK_STRATEGY_TOOL_SPEC` on a `ToolSimulationConfig` with the target `tool_name` (e.g., `"create_order"`). This auto-generates responses based on the tool's schema definition. 🏗 Practical Patterns **Testing Agent Resilience to API Failures**: Inject 503 or 500 errors to verify that agents handle retries and fallbacks correctly. Reproduce failure scenarios without touching production systems. **Edge Case Testing with Conditional Injection**: Use `match_args` to trigger errors only for specific argument patterns — "non-existent product ID," "out-of-stock item." Pinpoint-test agent behavior under specific conditions. **Realistic Testing with Probabilistic Failures**: Use `injection_probability` to simulate flaky APIs with, say, 30% failure rate. Fix `random_seed` for reproducible tests. **Realistic Mock Generation with Environment Data**: Provide DB snapshots or other context via `environment_data` so the Mock Strategy generates more realistic responses: Convert context data such as a DB snapshot (e.g., a dictionary of product records) to a JSON string via `json.dumps()` and pass it as the `environment_data` parameter of `EnvironmentSimulationConfig`. The Mock Strategy references this environment data to generate more realistic responses. **Combining Injections with Mock Strategy**: Inject errors for specific conditions while using Mock Strategy for everything else. Injection configs are evaluated first; unmatched calls fall through to the mock strategy. 💡 Use Cases 🔧 Verifying agent resilience to external API failures 🎯 Reproducing edge cases under specific conditions 📉 Probabilistic simulation of flaky API interactions ⏱ Verifying timeout handling with high-latency APIs ⚠️ Caveats - `injected_latency_seconds` has a maximum of 120 seconds. - `InjectionConfig` accepts either an error or a response, not both simultaneously. - `tool_name` must exactly match the registered tool name. Watch for typos. - Mock Strategy depends on tool schemas — insufficient schemas lead to lower-quality generated responses. ✨ Environment Simulation is a powerful testing foundation that reproduces edge cases without contaminating production. Combine error injection with conditional mocks to verify agent resilience against every failure scenario! #ADK# #AIAgent#

# Useful but Little-Known Features of ADK 2.0 🌍 What if your agent's prompts could improve themselves automatically, instead of requiring manual tuning? ADK 2.0's optimization feature combines samplers and agent optimizers to automatically refine your agent's instructions based on evaluation results. 📌 Title: Optimization 🔗 URL: 🧩 Overview The optimization feature consists of two components: Samplers (which evaluate candidates) and Agent Optimizers (which improve instructions). `LocalEvalSampler` runs candidate agents against evaluation datasets and computes scores. Two optimizers are available: `GEPARootAgentPromptOptimizer`, which uses the GEPA algorithm to optimize the root agent's prompt, and `SimplePromptOptimizer`, which iterates through an execute-evaluate-critique-rewrite cycle. Results are returned as an `OptimizerResult` containing optimized agents with their scores. 🛠 How to use it Run optimization from the CLI with `adk optimize`. ```python import asyncio from google.adk.evaluation.eval_config import EvalConfig from google.adk.evaluation.local_eval_sets_manager import LocalEvalSetsManager from google.adk.optimization.local_eval_sampler import LocalEvalSampler from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig from google.adk.optimization.simple_prompt_optimizer import SimplePromptOptimizer from google.adk.optimization.simple_prompt_optimizer import ( SimplePromptOptimizerConfig, ) # Set up the sampler sampler_config = LocalEvalSamplerConfig( eval_config=EvalConfig(criteria={"response_match_score": 0.75}), app_name="my_agent", train_eval_set="train_eval_set", ) sampler = LocalEvalSampler(sampler_config, LocalEvalSetsManager(agents_dir=".")) # Run optimization optimizer = SimplePromptOptimizer(config=SimplePromptOptimizerConfig()) result = sampler)) # Get the optimized agent and score best = result.optimized_agents[0] best_agent = best.optimized_agent print(f"Score: {best.overall_score}") ``` 🏗 Building it into production ・Run optimization periodically as evaluation datasets evolve, keeping prompts continuously refined ・Use `SimplePromptOptimizer` for quick iterations and `GEPARootAgentPromptOptimizer` for thorough exploration ・Compare pre- and post-optimization scores and only update prompts when improvement is confirmed ・Build evaluation datasets from real user interactions for practical optimization 💡 Use cases 📈 Automatically improve agent prompts based on evaluation scores 🔬 Systematically compare multiple prompt candidates and select the best one 🔄 Re-optimize prompts after model changes or API specification updates 🎯 Fine-tune prompts for specific tasks like code generation, summarization, or classification ⚠️ Watch out `GEPARootAgentPromptOptimizer` only optimizes the root agent's prompt. It does not cover sub-agent prompts, so optimize those separately if needed. Also, optimization requires multiple evaluation runs, so keep an eye on LLM API call costs. ✨ With optimization, you can move from trial-and-error prompt tuning to data-driven automation. Set up regular optimization cycles to continuously raise your agent's quality bar. #ADK# #AIAgent#

# Practical and Useful Patterns with ADK 🎭 Fixed-prompt tests can't replicate the unpredictable behavior of real users. ADK's User Simulation feature enables persona-driven conversational testing — "impatient customer," "vague requester" — to automatically verify agent robustness. 📌 Title: User Simulation 🔗 URL: 🧩 Overview ADK's User Simulation dynamically evaluates conversational agents using AI-generated user prompts. Instead of fixed test cases, a simulated user adapts to agent responses, making it ideal for testing unpredictable conversation flows. A ConversationScenario consists of three elements: `starting_prompt` (initial message), `conversation_plan` (high-level goals the user must achieve), and `user_persona` (user traits and interaction style). 🛠 Usage Basic user simulator configuration: ```json { "user_simulator_config": { "model": "gemini-flash-latest", "model_configuration": { "thinking_config": { "include_thoughts": true, "thinking_budget": 10240 } }, "max_allowed_invocations": 20 } } ``` ADK provides three pre-built personas: | Behavior | EXPERT | NOVICE | EVALUATOR | |----------|--------|--------|-----------| | Advance | Detail-oriented | Goal-oriented | Detail-oriented | | Answer | Relevant questions | All questions | Relevant questions | | Correct inaccuracies | Yes | No | No | | Troubleshoot errors | Once | Never | Never | | Tone | Professional | Conversational | Conversational | Define custom personas directly in ConversationScenario JSON: ```json { "user_persona": { "id": "IMPATIENT_USER", "description": "User in a rush, easily frustrated", "behaviors": [ { "name": "Short responses", "description": "Provides brief, incomplete replies", "behavior_instructions": [ "Keep responses under 10 words", "Omit polite phrases" ], "violation_rubrics": [ "Response exceeds 10 words", "Response is overly polite" ] } ] } } ``` Auto-generate evaluation scenarios: ```bash adk eval_set generate_eval_cases \ --user_simulation_config_file= ``` 🏗 Practical Patterns **Robustness Testing with "Impatient Customer" Persona**: Use an IMPATIENT_USER persona that gives sub-10-word responses and skips pleasantries. This tests whether your agent can gather sufficient information through proper follow-up questions despite receiving minimal input. Especially valuable for customer support agents. **"Vague Requester" Persona**: Simulate users who give abstract, unspecific requirements. Test whether your agent asks clarifying questions to pin down needs, rather than making assumptions or going off-track with ambiguous input. **Leveraging Pre-Built Personas**: EXPERT validates technical accuracy, NOVICE verifies that explanations are beginner-friendly, and EVALUATOR provides neutral performance assessment. Each reveals different aspects of agent quality. **Auto-Generated Scenarios for Coverage**: Use `generate_eval_cases` to automatically create diverse conversation scenarios, discovering edge cases you wouldn't think of manually. Specify `generation_instruction` for scenario direction and `environment_context` for backend state to produce realistic scenarios. 💡 Use Cases 🎭 Verifying customer support agent adaptability across diverse user types 🧪 Discovering edge-case conversation paths beyond fixed tests 🤖 Domain-specific robustness testing with custom personas 📊 Comprehensive quality evaluation through mass scenario generation ⚠️ Caveats - Without proper `max_allowed_invocations` settings, simulations may run indefinitely. Adjust from the default of 20 based on your domain. - Auto-generating scenarios requires the Vertex Gen AI Evaluation Service API. - Vague `violation_rubrics` in custom personas can lead to inconsistent persona behavior. Define specific violation conditions. - Enabling thinking in the simulator model produces more natural dialogue but increases costs. ✨ User Simulation breaks through the limitations of fixed tests. Use personas to simulate "how real users would behave" and verify agent robustness from multiple angles! #ADK# #AIAgent#

# Practical and Useful Patterns with ADK 📏 What you measure determines the quality of your evaluation. ADK's Evaluation Criteria provide built-in metrics covering tool trajectory accuracy, hallucination detection, safety assessment, and multi-turn task success. 📌 Title: Evaluation Criteria 🔗 URL: 🧩 Overview ADK provides multiple built-in evaluation criteria, all returning scores normalized to 0.0-1.0. **tool_trajectory_avg_score** verifies agent tool call sequences. **response_match_score** measures response similarity via ROUGE-1. **hallucinations_v1** validates response grounding against context. **safety_v1** assesses harmful content. **multi_turn_task_success_v1** measures task completion across conversation turns. 🛠 Usage Each criterion is configured in a JSON configuration file. **tool_trajectory_avg_score** — choose from three match types: ```json { "criteria": { "tool_trajectory_avg_score": { "threshold": 1.0, "match_type": "EXACT" } } } ``` - `EXACT`: Perfect match required — no extra or missing tool calls - `IN_ORDER`: Expected tools must appear in sequence, but other calls between them are allowed - `ANY_ORDER`: All expected tools must be present, regardless of order **response_match_score** — ROUGE-1 unigram overlap: ```json { "criteria": { "response_match_score": 0.8 } } ``` **hallucinations_v1** — validates each sentence against provided context: ```json { "criteria": { "hallucinations_v1": { "threshold": 0.8, "judge_model_options": { "judge_model": "gemini-flash-latest" }, "evaluate_intermediate_nl_responses": true } } } ``` Each sentence receives a label: supported, unsupported, contradictory, disputed, or not_applicable. The score is the percentage of sentences labeled supported or not_applicable. **safety_v1** — evaluates for hate speech, harassment, and dangerous information: ```json { "criteria": { "safety_v1": 0.8 } } ``` **multi_turn_task_success_v1** — measures task achievement across a full conversation: ```json { "criteria": { "multi_turn_task_success_v1": 0.8 } } ``` 🏗 Practical Patterns **Strict Tool Trajectory Verification**: In domains like finance or healthcare where tool call order matters — such as "authenticate first, check balance, then transfer" — use `EXACT` match to ensure the agent follows correct procedures. **Flexible Tool Verification**: For exploratory tasks, use `ANY_ORDER` to verify all necessary tools were called. `IN_ORDER` fits cases where main step order must be preserved but auxiliary tool calls between them are acceptable. **Hallucination Detection in Practice**: Use `hallucinations_v1` with RAG-based agents to automatically detect when agents generate information not grounded in retrieved results. Enable `evaluate_intermediate_nl_responses` to validate not just final answers but intermediate responses too. **Balancing Safety and Task Success**: Combine `safety_v1` with `multi_turn_task_success_v1` to catch agents that are "safe but ineffective" or "effective but inappropriate." 💡 Use Cases 🎯 Strict tool call order verification for finance/healthcare agents 🔍 Automatic hallucination detection in RAG agents 🛡 Safety evaluation for customer support agents 📈 Quantitative task completion rate measurement in multi-turn dialogues ⚠️ Caveats - `safety_v1`, `multi_turn_task_success_v1`, `multi_turn_trajectory_quality_v1`, and `multi_turn_tool_use_quality_v1` require Google Cloud Project setup (`GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` environment variables). - LLM-based criteria can vary between judge model runs. Set `num_samples` for multiple sampling with majority voting to stabilize results. - `response_match_score` uses ROUGE-1 (word overlap only) and does not evaluate semantic equivalence. Consider `final_response_match_v2` for semantic evaluation. ✨ Choosing the right evaluation criteria is the foundation of agent quality management. Match EXACT/IN_ORDER/ANY_ORDER to your domain requirements, and combine hallucination detection with safety assessment for comprehensive quality assurance! #ADK# #AIAgent#

# Useful but Little-Known Features of ADK 2.0 🌍 What if you could grade your agent on custom criteria that go beyond standard metrics — like compliance, tone, or domain-specific accuracy? ADK 2.0's custom evaluation metrics let you define your own evaluation logic as Python functions, measuring agent quality against any criteria you choose. 📌 Title: Custom Evaluation Metrics 🔗 URL: 🧩 Overview Custom evaluation metrics are Python functions that return an `EvaluationResult`. Each function receives four parameters: `eval_metric` (the metric definition), `actual_invocations` (the agent's actual responses), `expected_invocations` (the expected responses), and `conversation_scenario` (the conversation context). Async functions are fully supported, enabling integration with external services. 🛠 How to use it Create a custom metric function and register it in the eval config JSON. ```python from google.adk.evaluation.eval_metrics import EvalStatus from google.adk.evaluation.evaluator import EvaluationResult async def response_length_metric( eval_metric, actual_invocations, expected_invocations, conversation_scenario ) -> EvaluationResult: # Evaluate whether the response length is appropriate last_response = "".join( part.text for part in actual_invocations[-1].final_response.parts ) score = 1.0 if len(last_response) < 500 else 0.5 return EvaluationResult( overall_score=score, overall_eval_status=( EvalStatus.PASSED if score >= eval_metric.criterion.threshold else EvalStatus.FAILED ), ) ``` Register the metric in eval config JSON with a `criteria` threshold and a `custom_metrics` entry whose `code_config.name` is the function's import path. ```json { "criteria": { "response_length": {"threshold": 0.8} }, "custom_metrics": { "response_length": { "code_config": {"name": "my_metrics.response_length_metric"} } } } ``` 🏗 Building it into production ・Implement domain-specific quality criteria (accuracy, tone, compliance) as metrics ・Run custom metrics automatically in CI/CD pipelines and block deployments when quality drops below thresholds ・Use async metrics to integrate with external evaluation APIs or LLM-as-a-Judge patterns ・Combine multiple metrics to compute a comprehensive quality score for your agent 💡 Use cases 📏 Evaluate response conciseness and information density against custom standards 🏢 Automatically check compliance with industry regulations and requirements 🎯 Verify that specific tool call patterns match expected behavior 🗣 Score response tone and politeness automatically using LLM-as-a-Judge ⚠️ Watch out Heavy processing inside custom metric functions can slow down the entire evaluation run. Use async functions and set timeouts as needed. Also, the `code_config` module path must be importable from the evaluation execution environment. ✨ Custom evaluation metrics let you systematically enforce quality standards that go beyond what built-in metrics can cover. Build evaluations tailored to your specific use case. #ADK# #AIAgent#

# Practical and Useful Patterns with ADK 🧪 How do you verify that a prompt change hasn't degraded your agent's quality? ADK's Evaluation framework provides two approaches — test file method and dataset method — to systematically validate agent performance. Integrate it into CI to automate regression testing on every prompt change. 📌 Title: Evaluation Overview 🔗 URL: 🧩 Overview ADK provides two primary evaluation methods. The **test file method** uses `.test.json` files containing single sessions (with multiple turns) for unit-test-like fast iteration. The **evalset method** supports multiple lengthy sessions across many turns, suitable for integration testing. Some criteria (such as response quality, safety, and multi-turn quality) require the Vertex Gen AI Evaluation Service API. Built-in criteria include `tool_trajectory_avg_score`, `response_match_score`, `hallucinations_v1`, `safety_v1`, and more. 🛠 Usage Test files describe user queries, expected tool use trajectories, and expected responses. Three execution methods are available: **Web UI** for interactive evaluation: ```bash adk web path/to/agents_dir ``` **Pytest integration**: Call `await AgentEvaluator.evaluate()` with `agent_module` set to the target agent module name (e.g., `"home_automation_agent"`) and `eval_dataset_file_path_or_dir` pointing to the test file path (e.g., `"tests/integration/fixture/test.test.json"`). **CLI execution**: ```bash adk eval --config_file_path= ``` Configure evaluation thresholds in `test_config.json`: ```json { "criteria": { "tool_trajectory_avg_score": 1.0, "response_match_score": 0.8 } } ``` By default, tool trajectory requires 100% match while response matching accepts 80% similarity. 🏗 Practical Patterns **Regression Testing for Agent Updates**: Run existing test files after every prompt or tool definition change. `tool_trajectory_avg_score` quantitatively verifies tool selection accuracy, while `response_match_score` checks response quality. Early detection of quality degradation lets you iterate on prompts with confidence. **Fast Iteration with Test File Method**: During development, the `.test.json` approach works like unit testing. Define expected outcomes for single sessions, run them via pytest, and get quality feedback in seconds. Gradually expand test coverage by adding test files for each edge case. **Batch Evaluation with Dataset Method**: For complex, production-like scenarios, use the evalset method to evaluate many sessions at once. Multi-turn conversation flows and diverse user patterns can be comprehensively tested as integration tests. **CI Pipeline Integration**: Embed `adk eval` in your CI pipeline to automatically run quality checks on every pull request that modifies prompts. Fail the CI when scores drop below thresholds, preventing quality-degraded prompts from being merged. 💡 Use Cases 🔄 Automated regression testing on prompt changes ⚡ Fast development iteration with the test file method 📊 Comprehensive batch evaluation with the dataset method 🔗 Quality gate integration in CI/CD pipelines ⚠️ Caveats - Some criteria (such as response quality, safety, and multi-turn quality) require the Vertex Gen AI Evaluation Service API, which is a paid Google service. - Setting thresholds too high increases false negatives; too low risks missing quality degradation. Adjust incrementally from defaults. - LLM-based criteria (e.g., `hallucinations_v1`) can vary between judge model runs. Use `num_samples` for multiple sampling to improve reliability. ✨ ADK's Evaluation framework transforms agent quality management from intuition to quantitative benchmarks. Start with the test file method and progress to CI integration! #ADK# #AIAgent#