注册并分享邀请链接,可获得视频播放与邀请奖励。

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture 投稿は個人の意見です。
加入 May 2026
286 正在关注    471 粉丝
Is your enterprise SQL analytics agent silently returning zero rows from perfectly valid SQL, and nobody notices? Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents Amazon compared a trace-based LLM judge (Strands Evals), a text-only LLM judge (PromptFoo), and deterministic checks (Agenta) on two retail supply chain forecasting agents, across about 11,700 evaluations. Highlights 🔍 What the evaluator can see decides detection On 72 known-failure cases, Strands, which sees the full execution trace, reached F1 0.896 with a 10.3% false positive rate. Text-only PromptFoo has similar recall (87.9%) but its false positive rate jumps to 48.7%. Tool-correctness detection was 89% vs 11%. 🧪 Unknown failures depend on the kind of test Of 24 novel failures found by auto-generated tests, zero overlapped between functional and security testing. Strands found things like a negative-index SUBSTR bug, while PromptFoo found schema leakage and system prompt disclosure. 🎲 LLM judges wobble 35 of 36 configurations were stable, but decision consistency was only 82 to 92%, and one Qwen3 setup fell to 20%. Too early for per-case CI/CD gating. The takeaway: trace-based judging for function, red-teaming for security. #AIAgents# #LLMEvaluation#
显示更多