🛠 Reviewing production traces, finding failures, writing evaluators, opening fix PRs, that tedious improvement loop is now fully automated by LangSmith Engine. Title: Introducing LangSmith Engine URL: 📝 Overview LangSmith Engine is an autonomous system that automates the entire agent improvement cycle. It continuously monitors production traces to identify issues and propose fixes. It launched in public beta on May 13, 2026. ❓ Challenges Solved Traditionally, reviewing traces for patterns, identifying failures, writing evaluators, and creating fixes were all manual. Spotting failure patterns from individual traces was hard, tracking recurrence and building ground-truth data was tedious, and targeted regression evaluators were lacking. 💡 Methodology & Proposed Approach ・It acts as a deep agent with access to three sources: trace data, existing evaluator feedback, and source code ・Instead of listing individual problems, it clusters multiple failures into one named issue, with severity and correlation to deployments ・It offers three resolution actions: generating a fix PR, creating custom evaluators that resurface recurring patterns, and building offline eval datasets from failing traces ・Each resolved issue strengthens eval coverage, so production failures become future test cases in a self-reinforcing loop 🎯 Use Cases It automates detecting, diagnosing, and proposing fixes for production failure modes, like a support agent mishandling cancellations. It builds on existing LangSmith infrastructure, requiring no new system. 📊 Results ・In one example it detected the failure pattern in 12% of support sessions, correlated it to a recent deployment, and pinpointed ambiguous tool descriptions as the root cause ・Companies including Cogent, Harmonic, and Campfire have used it across thousands of traces ・An early adopter says it saves their team hours of digging #AIAgents# #Observability#