The OCI Agent Evaluation Framework goes beyond final-answer scoring to evaluate prompts, RAG, tools, trajectories, state changes, and production behavior across the agent lifecycle. Learn more:

Don't sleep on using Jev-as-a-Judge for agent evaluation. This is one of the most impressive Jev use cases I have found so far. Jev is a natural fit as a Judge, but it doesn't mean you use it everywhere. Similarly, you shouldn't use frontier models for evals everywhere. I'm running lots of tests on this atm, but early results point to an optimized flow (balancing accuracy and cost) that combines Jev and frontier models. Concretely, use Jev in high-confidence situations, and escalate to a frontier model (GPT-6 or Opus 5.5) in low-confidence verdicts. Entire write-up coming soon. Let me know if you have questions as I build the full guide.

“BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure” Reward hacking in agent benchmarks is often an infrastructure problem, not just a model-behavior problem. So this paper formalizes the full reward path and instruments runs to distinguish vulnerable tasks from actual exploit use, reaching 96% runtime detection accuracy and much higher exploit-chain recall than prior scanning baselines.

Great paper from Amazon. In discusses when not to trust LLM judges for agent evaluation. (bookmark it) A common way to compare task agents is to have an LLM user simulator talk to each one and an LLM judge score the transcript. This paper from Amazon shows that gate fails in two specific ways. 1. Satisfaction does not track success. 57.5% of conversations the raters marked satisfied had failed the customer's task. 2. Close calls go wrong. The ranking holds across agents of very different ability, but among near-equal agents the gate picks the lower-reward one on 31% of pairs, compared with under 1% for pairs far apart. The study covers 25 agents from six providers on tau2-bench and SimulatorArena. Judges also favored agents from their own model family. The fix is cheap. A judge-free completion bit catches truncation regressions, and the judge is trusted only after calibration against a verifiable reward. Paper:

Today, most agents are built with the help of other agents like Sierra's Ghostwriter. Yesterday, Sierra open-sourced hyper-𝜏-bench (published as 𝜏^𝜏-bench), a new long horizon agent evaluation that measures how well models can not only act as an agent, but construct one.

Finally a good paper testing whether memory-based self-improving agents actually improve. The re-evaluation adds two things prior work skipped, multiple runs to measure variance and randomly shuffled task orders. Both hurt. Agent evaluation is already noisy on multi-step tasks, and stacking a self-improvement loop amplifies that noise. Default task orderings impose an implicit curriculum that much of the reported gain was riding on. Adding detailed rubrics and environment feedback to memory construction recovers part of the drop, and a significant gap remains. Paper: Track more trending AI papers in our academy: