Is your enterprise SQL analytics agent silently returning zero rows from perfectly valid SQL, and nobody notices?
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
Amazon compared a trace-based LLM judge (Strands Evals), a text-only LLM judge (PromptFoo), and deterministic checks (Agenta) on two retail supply chain forecasting agents, across about 11,700 evaluations.
Highlights
๐ What the evaluator can see decides detection
On 72 known-failure cases, Strands, which sees the full execution trace, reached F1 0.896 with a 10.3% false positive rate. Text-only PromptFoo has similar recall (87.9%) but its false positive rate jumps to 48.7%. Tool-correctness detection was 89% vs 11%.
๐งช Unknown failures depend on the kind of test
Of 24 novel failures found by auto-generated tests, zero overlapped between functional and security testing. Strands found things like a negative-index SUBSTR bug, while PromptFoo found schema leakage and system prompt disclosure.
๐ฒ LLM judges wobble
35 of 36 configurations were stable, but decision consistency was only 82 to 92%, and one Qwen3 setup fell to 20%. Too early for per-case CI/CD gating.
The takeaway: trace-based judging for function, red-teaming for security.
#
AIAgents# #
LLMEvaluation#