Register and share your invite link to earn from video plays and referrals.

Search results for AgentEvaluation
AgentEvaluation community
One keyword maps to one global community path.
Create community
People
Not Found
Tweets including AgentEvaluation
The OCI Agent Evaluation Framework goes beyond final-answer scoring to evaluate prompts, RAG, tools, trajectories, state changes, and production behavior across the agent lifecycle. Learn more:
Show more
Is your enterprise SQL analytics agent silently returning zero rows from perfectly valid SQL, and nobody notices? Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents Amazon compared a trace-based LLM judge (Strands Evals), a text-only LLM judge (PromptFoo), and deterministic checks (Agenta) on two retail supply chain forecasting agents, across about 11,700 evaluations. Highlights 🔍 What the evaluator can see decides detection On 72 known-failure cases, Strands, which sees the full execution trace, reached F1 0.896 with a 10.3% false positive rate. Text-only PromptFoo has similar recall (87.9%) but its false positive rate jumps to 48.7%. Tool-correctness detection was 89% vs 11%. 🧪 Unknown failures depend on the kind of test Of 24 novel failures found by auto-generated tests, zero overlapped between functional and security testing. Strands found things like a negative-index SUBSTR bug, while PromptFoo found schema leakage and system prompt disclosure. 🎲 LLM judges wobble 35 of 36 configurations were stable, but decision consistency was only 82 to 92%, and one Qwen3 setup fell to 20%. Too early for per-case CI/CD gating. The takeaway: trace-based judging for function, red-teaming for security. #AIAgents# #LLMEvaluation#
Show more
# Useful but Little-Known Features of OpenAI Agent SDK 🌍 Ever wished you could go back in a conversation and try "what if I'd said something different?" AdvancedSQLiteSession's conversation branching lets you fork from any point in the history and explore alternative paths. 📌 Title: AdvancedSQLiteSession Conversation Branching 🔗 URL: 🧩 Overview `AdvancedSQLiteSession` supports branching conversations from specific turns or content matches. `create_branch_from_turn(turn_number)` forks from a specific turn, while `create_branch_from_content(search_term)` forks from a turn containing the search term. You can manage branches with `list_branches()`, `switch_to_branch()`, and `delete_branch()`. 🛠 How to use it ```python from agents.extensions.memory import AdvancedSQLiteSession session = AdvancedSQLiteSession( session_id="chat-001", db_path="sessions.db", create_tables=True, # Defaults to False; required to create the branch tables ) # Create a branch from turn 3 branch_id = await session.create_branch_from_turn(turn_number=3) # Create a branch from a turn containing specific content branch_id = await session.create_branch_from_content("API design discussion") # Manage branches branches = await session.list_branches() await session.switch_to_branch(branch_id) await session.delete_branch(branch_id, force=True) # force=True to delete the current branch ``` 🏗 Building it into production ・When users want to retry, create a branch from the previous turn for a fresh attempt ・In agent evaluation, compare multiple response paths for the same input ・In customer support, explore different solutions in parallel and pick the best one ・Use branches as conversation checkpoints to record important decision points 💡 Use cases 🔀 "What-if" analysis on conversations 🧪 A/B testing agent response quality 🎯 Parallel exploration of multiple solution paths 📝 Checkpoint management for conversation history ⚠️ Watch out Storage usage grows as branches accumulate. Regularly clean up unused branches with `delete_branch()` (pass `force=True` to delete the current branch; `main` cannot be deleted). Branch IDs are unique for the lifetime of a session ID and are not released by deleting a branch or clearing the session, so use a new name for each new branch. Also, manage branch labels carefully to avoid context confusion when switching between branches. ✨ With conversation branching, add time-travel capabilities to your agent interactions. #OpenAIAgentSDK# #AIAgent#
Show more
Don't sleep on using Jev-as-a-Judge for agent evaluation. This is one of the most impressive Jev use cases I have found so far. Jev is a natural fit as a Judge, but it doesn't mean you use it everywhere. Similarly, you shouldn't use frontier models for evals everywhere. I'm running lots of tests on this atm, but early results point to an optimized flow (balancing accuracy and cost) that combines Jev and frontier models. Concretely, use Jev in high-confidence situations, and escalate to a frontier model (GPT-6 or Opus 5.5) in low-confidence verdicts. Entire write-up coming soon. Let me know if you have questions as I build the full guide.
Show more
“BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure” Reward hacking in agent benchmarks is often an infrastructure problem, not just a model-behavior problem. So this paper formalizes the full reward path and instruments runs to distinguish vulnerable tasks from actual exploit use, reaching 96% runtime detection accuracy and much higher exploit-chain recall than prior scanning baselines.
Show more
Great paper from Amazon. In discusses when not to trust LLM judges for agent evaluation. (bookmark it) A common way to compare task agents is to have an LLM user simulator talk to each one and an LLM judge score the transcript. This paper from Amazon shows that gate fails in two specific ways. 1. Satisfaction does not track success. 57.5% of conversations the raters marked satisfied had failed the customer's task. 2. Close calls go wrong. The ranking holds across agents of very different ability, but among near-equal agents the gate picks the lower-reward one on 31% of pairs, compared with under 1% for pairs far apart. The study covers 25 agents from six providers on tau2-bench and SimulatorArena. Judges also favored agents from their own model family. The fix is cheap. A judge-free completion bit catches truncation regressions, and the judge is trusted only after calibration against a verifiable reward. Paper:
Show more
# Useful but Little-Known Features of ADK 2.0 🌍 What if you could replay realistic user conversations to evaluate your agent — without needing actual users? ADK 2.0's user simulation feature dynamically generates conversations with virtual users of varying personas, enabling systematic evaluation of your agent's response quality. 📌 Title: User Simulation 🔗 URL: 🧩 Overview User simulation automatically generates dynamic conversational prompts for agent evaluation. You define a `ConversationScenario` with a `starting_prompt` (the opening message), a `conversation_plan` (how the conversation should progress), and a `user_persona` (the simulated user's behavior). Three built-in personas are provided: EXPERT (detail-oriented, corrects the agent's mistakes), NOVICE (passive, waits for prompts and guidance), and EVALUATOR (detail-oriented but does not correct errors). You can also create custom personas with `UserPersona`. 🛠 How to use it Use the CLI command `adk eval_set generate_eval_cases` to generate evaluation cases. For custom personas, use the `UserPersona` class. ```python from google.adk.evaluation.conversation_scenarios import ConversationScenario from google.adk.evaluation.simulation.user_simulator_personas import ( UserBehavior, UserPersona, ) # Define a custom persona impatient_user = UserPersona( id="IMPATIENT", description="Prefers short answers and shows frustration with lengthy explanations", behaviors=[ UserBehavior( name="Short responses", description="The user provides very short, sometimes incomplete responses", behavior_instructions=["Keep your responses under 10 words.", "Omit polite phrases."], violation_rubrics=["The user response is over 10 words."], ) ], ) # Define a scenario scenario = ConversationScenario( starting_prompt="Help me book a trip", conversation_plan="Confirm destination → dates → budget, then complete the booking", user_persona=impatient_user ) ``` 🏗 Building it into production ・Integrate evaluation scenarios into CI/CD pipelines to run regression tests automatically on agent updates ・Use all three built-in personas (EXPERT, NOVICE, EVALUATOR) to cover different user segments ・Design `conversation_plan` around real business workflows to match actual usage patterns ・Create custom personas that mirror your product's specific user behavior patterns 💡 Use cases 🧪 Automatically detect conversation quality regressions after prompt or model changes 👥 Verify that different response strategies work for beginners vs. power users 📊 Run batch evaluations across multiple scenarios to systematically identify agent weaknesses 🔄 Test how well the agent handles topic shifts mid-conversation or ambiguous questions ⚠️ Watch out Since simulations are LLM-generated, the same scenario may produce slightly different conversation flows on each run. For reproducible evaluations, run multiple iterations and assess results statistically. Also, each built-in persona behaves quite differently — make sure you select the one that matches your evaluation goals. ✨ User simulation lets you continuously improve your agent's conversational quality without involving real users. Make multi-persona evaluation a regular part of your development workflow. #ADK# #AIAgent#
Show more
We tested Jev against LLM judges on accuracy, repeatability, latency, and cost to see whether System One models could offer a new approach to agent evaluation.
0
159
2.5K
313
Forward to community
We're hosting a meetup! Come meet the community behind the benchmark, celebrate recent releases, and join our discussions about the future of agent evaluation.
Today, most agents are built with the help of other agents like Sierra's Ghostwriter. Yesterday, Sierra open-sourced hyper-𝜏-bench (published as 𝜏^𝜏-bench), a new long horizon agent evaluation that measures how well models can not only act as an agent, but construct one.
Show more