Testing AI agents isn't like testing regular software. The same input can produce a different output every run, there's rarely just one correct answer, and even a correct result can come from an "inappropriate" path. Title: The Most Common Questions About Agent Evals, Answered URL: ❓ Where should you even start? 💡 Pick one behavior to evaluate (say, whether a travel assistant books the requested date and price range), build a small dataset of 10-20 examples from your traces, run a baseline on your current version, and then keep adding failures you find in production or during development. ❓ What's the difference between offline and online evals? 💡 Offline evals run against a dataset before shipping and can compare against reference outputs, making them great for testing new prompts, models, or tools. Online evals score real production traces in real time, usually without reference outputs, relying on code checks and LLM judges to catch quality drift and surprises. ❓ How do you make an LLM judge actually trustworthy? 💡 Validate it against human scores on a small sample, phrase questions narrowly (like "did the reply mention every connecting flight?"), and score with a binary pass/fail or a few categories instead of a 1-10 scale. Picking a cheap judge model can also cut costs dramatically: one example dropped total judging cost from $28.17 with Claude Sonnet to just $0.34. ❓ How do you evaluate multi-turn conversations? 💡 A conversation can look fine turn by turn while still breaking down overall. The fix is to group turns by thread ID, display every message and tool call in order, and evaluate the whole thread with questions like "does the final booking reflect every change the user made?" #AIAgents# #Evals#