Even with automated evals, only a person can tell why an agent failed. LangChain argues expert judgment is a strategic advantage 🧑‍⚖️ Why human insight is a strategic advantage for building better agents Automated evaluation can flag problems, but it can't explain the cause or the fix. The article lays out three principles for building human review into the evaluation system. 🎯 Principle 1: humans define what "good" looks like Reviewers supply context a trace doesn't contain, such as the standard of quality for the job and what the user was after. Scores with reasoning also expose gaps in the evaluators themselves. 🧵 Principle 2: context makes the judgment hold up Looking at one step misses contradictions with something said several turns earlier. For multi-turn agents, the whole conversation (a thread) is the unit of review. 🔁 Principle 3: turn judgment into evaluation data Labels become ground truth, used to align LLM judges with human scores, update online evaluators, and build golden datasets. In the case study, Fetch Rewards manually checks even product-card images and offer availability. Automation handles scale; the advantage comes from turning expert judgment into a reusable asset. #AIAgents# #LLMEvaluation#

How does an AI coach that can't touch medicine prove it's safe? Numan, a UK digital health company, built Nu, an AI coach that is a wellness product: no diagnosis, no prescriptions. It had to hold that line while still coaching well. The team first scored replies one by one on static datasets, but averages hid a few dangerous replies, and scripted conversations couldn't follow Nu's unexpected turns. Worse, some low scores came from a rubric demanding empathy and advice within two turns. The turning point was treating evaluation criteria, not prompts, as the scarce asset. They test the exact same LangGraph graph patients use, score by theme, let safety probes block regressions at release, and have a cheaper model play patients so a judge can score whole conversations. A provider migration that usually takes weeks took about a day. How Numan built an eval-first AI health coach with LangGraph and LangSmith #LangGraph# #LLMEvaluation#

💻 Even top-tier coding AIs see their scores collapse by more than half when the user isn't a programmer. A new benchmark puts that uncomfortable fact on the table. Title: SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction URL: ❓ What was the problem? Existing coding benchmarks tend to use short tasks with the full spec handed over up front, which doesn't match how real users gradually reveal requirements and react to what the assistant does. ❓ What does SWE-Journey do differently? It auto-synthesizes long-horizon tasks (averaging 10.2 changed files and 768 lines) and has AI play four personas drawn from real Claude conversation logs - non-coder, product manager, new developer, and software architect - each chatting with the assistant over dozens of turns. ❓ What did they find? Even the best model only reaches 52.22% Final F2P (final feature-implementation success). It hits 78.50% with the software-architect persona but drops to just 23.00% with non-coders, a 55.50-point gap. ❓ Why such a big gap? Weaker models rarely ask clarifying questions when requirements are vague, stay overconfident about the wrong part of the code, and sometimes report a fix as "done" with no real evidence. How well a model adapts its questioning and verification style to the user turns out to matter as much as raw coding skill. #AICoding# #LLMEvaluation#

How do you actually build evaluation benchmarks for AI agents at scale? LangChain just shared their full approach. Title: How We Build Agent Environments & Tasks URL: ❓ What exactly makes up an agent "task"? 💡 A task has three components: an input, an environment, and a test script. The environment hosts agent execution, and a rubric defines scoring criteria. A "world spec" consolidates shared domain knowledge across related tasks — API schemas, data generation methods, trace parsing scripts — in one place. ❓ How do you create tasks efficiently at scale? 💡 LangChain uses a two-step pipeline. First, a "spec generation" phase where a coding agent scans repositories, groups traces, maps credentials, and auto-generates an initial world spec. Then a "Spec2Task" phase converts that spec into runnable evaluation tasks. The spec from the very first task becomes the foundation, iteratively refined through subsequent task creation cycles. ❓ What are the most common mistakes when building evaluation tasks? 💡 Three pitfalls stand out: ・Don't skip running tasks with real agents — paper evaluation won't surface environment flaws ・Calibrate difficulty across model tiers (e.g. gpt-5.6-Luna vs Sol) — what's hard for one may be trivial for another ・Match the data generation method to the data type: LLM-based approaches for free-text, SQL scripts for tabular data ❓ Is a benchmark "done" once you've built it? 💡 Not at all. Continuous improvement from production data is the core idea. Real production traces feed back into cost modeling, prompt simplification validation, and tool configuration testing — the benchmark evolves alongside the system it measures. Treating evaluation environment engineering as ongoing rather than a one-time project is the key practical insight here. #AIAgents# #LLMEvaluation#

Judging billions of trace tokens a day with a frontier LLM was financially untenable 💸 Here's how fine-tuning a small open model matched the accuracy at 10-100x lower cost. Title: Building a 100x Cheaper Trace Judge with Fireworks URL: 💸 Overview LangChain Labs partnered with Fireworks to build a "Perceived Error" detector for agent traces. It detects cases where users felt the assistant made a mistake or needed correction — using a small open model. ❓ Challenges Solved LangSmith processes billions of tokens daily across production traces. ・Evaluating them with frontier LLMs is prohibitively expensive at scale ・The question: how to cost-effectively mine important signals from every trace while maintaining frontier performance? 💡 Methodology & Proposed Approach ・Supervised fine-tuning (SFT) with LoRA on the open-source Qwen-3.5-35B, on Fireworks infrastructure ・Training data from two production datasets: chat-langchain (technical Q&A, 707 examples) and Fleet (no-code agent, 727 examples) ・It learns "Perceived Error," doing the judging without relying on huge frontier models 📊 Experimental Results ・Accuracy: the fine-tuned Qwen matched or exceeded frontier models (96.1% on chat-langchain, 90.8% cross-domain on Fleet) ・Cost: 10-100x cheaper than frontier models depending on trace volume ・Transferability: the model trained on chat-langchain beat all frontier models on Fleet without retraining #LLMEvaluation# #FineTuning#

Can an AI actually mediate a conflict between people? 🤝 A benchmark that tries to measure that, reliably, under realistic conditions. Title: SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations URL: 🤝 Overview This work proposes SoCRATES, a comprehensive benchmark for evaluating LLMs as mediators. An agentic pipeline builds realistic conflict scenarios across eight domains from actual public disputes, enabling automated and reliable evaluation of proactive LLM mediation. ❓ Challenges Solved Using LLMs to guide disputing parties toward agreement is gaining attention, but evaluating it is hard. ・Real conflicts shift constantly as disputants' emotions, intentions, and context change mid-mediation ・Existing benchmarks rely on a limited set of expert-authored scenarios ・They also score every turn against every topic, injecting noise that muddies the evaluation signal 💡 Methodology & Proposed Approach SoCRATES integrates three approaches. ・Agentic scenario curation: agents find genuine public disputes, restructure them into mediation scenarios, and filter for cases that truly need intervention ・Socio-cognitive probing: vary each scenario across five independent dimensions (strategic posture, party composition, conversation-history length, emotional reactivity, cultural identity) to pinpoint capability gaps ・Topic-localized evaluation: instead of scoring every topic at every turn, rate only the turns where a topic is actively discussed, reducing noise It spans eight domains: transactional, health, environmental, B2B, policy, international, legal, and intra-organizational. 🌍 Use Cases / Experimental Results The results were sober and revealing. ・The evaluator reached r=0.82 alignment with human experts (trajectory level), more than doubling baseline performance ・Among eight frontier LLMs, even the best, GPT-5.4-mini, closed only about 34.4% of the consensus gap (all-mediator average 25.9%) ・Big domain spread: 41.3% improvement in transactional disputes versus just 16.6% in intra-organizational ones The key takeaway: meaningful progress needs better social adaptation to diverse conditions, not just general capability gains. #LLMEvaluation# #AIMediation#

Congratulations to seven research projects that were selected for the Spring 2026 cohort. - Emmanuel Candès, Stanford University: Statistically Efficient LLM Evaluation and Ranking with Pairwise Human Preference Data - Haifeng Xu @haifengxu0, University of Chicago: Towards a Principled Evaluation of LLMs’ Predictive Intelligence - Lihua Lei @lihua_lei_stat, Stanford University: Is Bradley–Terry Enough? Heterogeneous Preference Modeling for RLHF - Negin Golrezaei @NeginGolrezaei, Massachusetts Institute of Technology: Causal inference framework for multi-turn AI evaluation. - Ramesh Raskar @raskarmit, Massachusetts Institute of Technology: Toward Adaptive, Task-Conditional Evaluation for AI Agents - Steven Wu @zstevenwu and Andrew Ilyas @andrew_ilyas, Carnegie Mellon University: Goodhart’s Law on the Leaderboard: Can Cheap Preference Optimization Game Arena? -Tal Linzen @tallinzen, New York University: A Rigorous Evaluation of Meta-Cognitive Monitoring in LLMs Read more about the program details and submit your proposal for Fall 2026 here: