🕵️ The cause of an agent's failure may be findable from the "flow of success," not from failures.
Title: Tracing Agentic Failure from the Flow of Success
URL:
🧩 Overview
Oat is trained only on successful trajectories and pinpoints which steps of a failed trajectory caused the failure, with no labels. Accepted to NeurIPS 2026.
🎯 The problem
Existing failure attribution needs either expensive LLM prompting pipelines or failure data with step-level error labels. Both are costly.
🔧 Method
・Represent each step with LLM hidden states
・Learn the dynamics of successful runs with a Neural CDE
・A gated control path improves robustness to distribution shift
・At inference, steps that deviate from the learned flow get high anomaly scores (top-k or conformal prediction)
Only about 100 successful trajectories are used for training.
📊 Results
・In-domain F1: Oat 0.435 vs. GPT-4o 0.212 and GPT-5 0.181
・About +7 F1 points out of distribution too
・About 7 ms per trajectory with zero tokens, vs. about 39.6 s for GPT-5
・200-5000x faster on scoring alone
#
AIAgents# #
LLM#