Register and share your invite link to earn from video plays and referrals.

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture ๆŠ•็จฟใฏๅ€‹ไบบใฎๆ„่ฆ‹ใงใ™ใ€‚
Joined May 2026
286 Following    471 Followers
Is your enterprise SQL analytics agent silently returning zero rows from perfectly valid SQL, and nobody notices? Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents Amazon compared a trace-based LLM judge (Strands Evals), a text-only LLM judge (PromptFoo), and deterministic checks (Agenta) on two retail supply chain forecasting agents, across about 11,700 evaluations. Highlights ๐Ÿ” What the evaluator can see decides detection On 72 known-failure cases, Strands, which sees the full execution trace, reached F1 0.896 with a 10.3% false positive rate. Text-only PromptFoo has similar recall (87.9%) but its false positive rate jumps to 48.7%. Tool-correctness detection was 89% vs 11%. ๐Ÿงช Unknown failures depend on the kind of test Of 24 novel failures found by auto-generated tests, zero overlapped between functional and security testing. Strands found things like a negative-index SUBSTR bug, while PromptFoo found schema leakage and system prompt disclosure. ๐ŸŽฒ LLM judges wobble 35 of 36 configurations were stable, but decision consistency was only 82 to 92%, and one Qwen3 setup fell to 20%. Too early for per-case CI/CD gating. The takeaway: trace-based judging for function, red-teaming for security. #AIAgents# #LLMEvaluation#
Show more