🧭 TL;DR: Instead of estimating an LLM's confidence from the current inference alone, this method calibrates it against how often similar past attempts actually succeeded. It beats 10-sample self-consistency in accuracy while costing far less. Title: Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents URL: Points 🗂 Stores past task, reasoning trace, stated confidence, actual outcome, and a lesson learned in an experience bank 🔍 Recall stage retrieves k=50 similar episodes and computes their real hit rate statistically 💭 Reflect stage has the model restate confidence in words after seeing that track record 🏆 Matches or beats SC@10 on 23 of 24 model-dataset combinations by AUROC 🤖 Biggest gains on agent tasks where failures are silent — even beats a trained verifier on AppWorld 📉 2-20x lower calibration error across domains, at roughly 1/10th the compute cost A great example of the shift from "judge the current attempt" to "judge from accumulated track record." #LLMEval# #AIAgents#

Even with automated evals, only a person can tell why an agent failed. LangChain argues expert judgment is a strategic advantage 🧑‍⚖️ Why human insight is a strategic advantage for building better agents Automated evaluation can flag problems, but it can't explain the cause or the fix. The article lays out three principles for building human review into the evaluation system. 🎯 Principle 1: humans define what "good" looks like Reviewers supply context a trace doesn't contain, such as the standard of quality for the job and what the user was after. Scores with reasoning also expose gaps in the evaluators themselves. 🧵 Principle 2: context makes the judgment hold up Looking at one step misses contradictions with something said several turns earlier. For multi-turn agents, the whole conversation (a thread) is the unit of review. 🔁 Principle 3: turn judgment into evaluation data Labels become ground truth, used to align LLM judges with human scores, update online evaluators, and build golden datasets. In the case study, Fetch Rewards manually checks even product-card images and offer availability. Automation handles scale; the advantage comes from turning expert judgment into a reusable asset. #AIAgents# #LLMEvaluation#

How does an AI coach that can't touch medicine prove it's safe? Numan, a UK digital health company, built Nu, an AI coach that is a wellness product: no diagnosis, no prescriptions. It had to hold that line while still coaching well. The team first scored replies one by one on static datasets, but averages hid a few dangerous replies, and scripted conversations couldn't follow Nu's unexpected turns. Worse, some low scores came from a rubric demanding empathy and advice within two turns. The turning point was treating evaluation criteria, not prompts, as the scarce asset. They test the exact same LangGraph graph patients use, score by theme, let safety probes block regressions at release, and have a cheaper model play patients so a judge can score whole conversations. A provider migration that usually takes weeks took about a day. How Numan built an eval-first AI health coach with LangGraph and LangSmith #LangGraph# #LLMEvaluation#

💻 Even top-tier coding AIs see their scores collapse by more than half when the user isn't a programmer. A new benchmark puts that uncomfortable fact on the table. Title: SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction URL: ❓ What was the problem? Existing coding benchmarks tend to use short tasks with the full spec handed over up front, which doesn't match how real users gradually reveal requirements and react to what the assistant does. ❓ What does SWE-Journey do differently? It auto-synthesizes long-horizon tasks (averaging 10.2 changed files and 768 lines) and has AI play four personas drawn from real Claude conversation logs - non-coder, product manager, new developer, and software architect - each chatting with the assistant over dozens of turns. ❓ What did they find? Even the best model only reaches 52.22% Final F2P (final feature-implementation success). It hits 78.50% with the software-architect persona but drops to just 23.00% with non-coders, a 55.50-point gap. ❓ Why such a big gap? Weaker models rarely ask clarifying questions when requirements are vague, stay overconfident about the wrong part of the code, and sometimes report a fix as "done" with no real evidence. How well a model adapts its questioning and verification style to the user turns out to matter as much as raw coding skill. #AICoding# #LLMEvaluation#

Ever spend way too long hunting across scattered sources just to find "which benchmark actually fits this task"? Title: Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation URL: ❓ What is Benchmark Radar? 💡 It's a search engine that crawls 37 sources daily, arXiv, Hugging Face, GitHub and more, and pulls each benchmark's paper, dataset, repository, and score history into one place, with a CLI for offline queries. ❓ How much data are we talking about? 💡 1,283 source records from four catalogs, covering 12,916 numeric scores across 790 benchmarks. Daily collection alone logs 11,068 observations across 6,546 distinct artifacts. ❓ Why can't scores just be compared directly? 💡 Only 82 of the scored records use a verified 0-100 percentage scale, the rest use different or unverified scales, and roughly half lack a known release date, so naive comparisons would be misleading. ❓ So what's the fix? 💡 Instead of forcing comparability, it preserves source identity and citations so readers can inspect the evaluation conditions themselves. #LLMEval# #Benchmarks#

LLMs now beat humans on math accuracy. But does that mean they're actually building understanding on top of the right foundations? Title: Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory URL: ❓ Do LLMs actually build on prerequisite knowledge to get answers right? 💡 Accuracy favors LLMs (92.5% for the best model, Qwen3-80B, vs. 79.6% for humans), but "perfect prerequisite satisfaction" tells a different story: 72.7% for humans vs. only 48.16% for the best LLM. Lots of correct answers rest on shaky foundations. ❓ Does giving them prerequisite hints help? 💡 Surprisingly, prerequisite-grounded context barely outperformed unrelated examples. That points to surface-level pattern matching rather than genuine structured reasoning over prerequisites. ❓ Do strong models at least share a consistent knowledge structure with each other? 💡 Human learner groups overlap at 0.9+ in their knowledge structure, but LLM pairs only overlap 0.38-0.6 — and stronger models diverge even further from humans. ❓ So what's the takeaway? 💡 Accuracy alone hides how differently LLMs "understand" math. Knowledge Space Theory offers a lens that exposes the fragmented structure lurking behind impressive scores. #LLMEval# #MathReasoning#

A pipeline with 100% delivery rate, 100% schema validity, and zero retries or errors... yet re-running the exact same request flips the verdict. This paper reports that shocking negative result, fully preregistered with a complete audit trail. Title: Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints URL: 🔧 Highlight 1: Clean engineering does not mean reliable measurement Across 3,312 planned calls, response rate and schema validity both hit 100% with zero retries or errors, yet the repeat-ranking agreement (median Spearman) was only 0.400 — clearly failing the preregistered stability gate (threshold 0.90). 🎲 Highlight 2: Byte-identical inputs still drift Replaying the exact same request 24 hours later gave only a 0.780 exact-ranking agreement rate. Interestingly, comparisons across windows on the same day scored a nearly identical 0.805 median, revealing this isn't "next-day drift" but immediate platform-level nondeterminism. 📉 Highlight 3: More samples don't fix it Scaling observer calls from 8 to 500 (748,000 calls total) still left the stability gate pass rate at 0/500. The root cause: score gaps between candidates were 7-10 orders of magnitude smaller than the noise floor, making most tasks fundamentally unrankable. I think this is a great reminder that before using an LLM as a measurement instrument, you need to measure whether the instrument itself is stable. #LLMEval# #Reproducibility#

How do you actually build evaluation benchmarks for AI agents at scale? LangChain just shared their full approach. Title: How We Build Agent Environments & Tasks URL: ❓ What exactly makes up an agent "task"? 💡 A task has three components: an input, an environment, and a test script. The environment hosts agent execution, and a rubric defines scoring criteria. A "world spec" consolidates shared domain knowledge across related tasks — API schemas, data generation methods, trace parsing scripts — in one place. ❓ How do you create tasks efficiently at scale? 💡 LangChain uses a two-step pipeline. First, a "spec generation" phase where a coding agent scans repositories, groups traces, maps credentials, and auto-generates an initial world spec. Then a "Spec2Task" phase converts that spec into runnable evaluation tasks. The spec from the very first task becomes the foundation, iteratively refined through subsequent task creation cycles. ❓ What are the most common mistakes when building evaluation tasks? 💡 Three pitfalls stand out: ・Don't skip running tasks with real agents — paper evaluation won't surface environment flaws ・Calibrate difficulty across model tiers (e.g. gpt-5.6-Luna vs Sol) — what's hard for one may be trivial for another ・Match the data generation method to the data type: LLM-based approaches for free-text, SQL scripts for tabular data ❓ Is a benchmark "done" once you've built it? 💡 Not at all. Continuous improvement from production data is the core idea. Real production traces feed back into cost modeling, prompt simplification validation, and tool configuration testing — the benchmark evolves alongside the system it measures. Treating evaluation environment engineering as ongoing rather than a one-time project is the key practical insight here. #AIAgents# #LLMEvaluation#

Judging billions of trace tokens a day with a frontier LLM was financially untenable 💸 Here's how fine-tuning a small open model matched the accuracy at 10-100x lower cost. Title: Building a 100x Cheaper Trace Judge with Fireworks URL: 💸 Overview LangChain Labs partnered with Fireworks to build a "Perceived Error" detector for agent traces. It detects cases where users felt the assistant made a mistake or needed correction — using a small open model. ❓ Challenges Solved LangSmith processes billions of tokens daily across production traces. ・Evaluating them with frontier LLMs is prohibitively expensive at scale ・The question: how to cost-effectively mine important signals from every trace while maintaining frontier performance? 💡 Methodology & Proposed Approach ・Supervised fine-tuning (SFT) with LoRA on the open-source Qwen-3.5-35B, on Fireworks infrastructure ・Training data from two production datasets: chat-langchain (technical Q&A, 707 examples) and Fleet (no-code agent, 727 examples) ・It learns "Perceived Error," doing the judging without relying on huge frontier models 📊 Experimental Results ・Accuracy: the fine-tuned Qwen matched or exceeded frontier models (96.1% on chat-langchain, 90.8% cross-domain on Fleet) ・Cost: 10-100x cheaper than frontier models depending on trace volume ・Transferability: the model trained on chat-langchain beat all frontier models on Fleet without retraining #LLMEvaluation# #FineTuning#

Can an AI actually mediate a conflict between people? 🤝 A benchmark that tries to measure that, reliably, under realistic conditions. Title: SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations URL: 🤝 Overview This work proposes SoCRATES, a comprehensive benchmark for evaluating LLMs as mediators. An agentic pipeline builds realistic conflict scenarios across eight domains from actual public disputes, enabling automated and reliable evaluation of proactive LLM mediation. ❓ Challenges Solved Using LLMs to guide disputing parties toward agreement is gaining attention, but evaluating it is hard. ・Real conflicts shift constantly as disputants' emotions, intentions, and context change mid-mediation ・Existing benchmarks rely on a limited set of expert-authored scenarios ・They also score every turn against every topic, injecting noise that muddies the evaluation signal 💡 Methodology & Proposed Approach SoCRATES integrates three approaches. ・Agentic scenario curation: agents find genuine public disputes, restructure them into mediation scenarios, and filter for cases that truly need intervention ・Socio-cognitive probing: vary each scenario across five independent dimensions (strategic posture, party composition, conversation-history length, emotional reactivity, cultural identity) to pinpoint capability gaps ・Topic-localized evaluation: instead of scoring every topic at every turn, rate only the turns where a topic is actively discussed, reducing noise It spans eight domains: transactional, health, environmental, B2B, policy, international, legal, and intra-organizational. 🌍 Use Cases / Experimental Results The results were sober and revealing. ・The evaluator reached r=0.82 alignment with human experts (trajectory level), more than doubling baseline performance ・Among eight frontier LLMs, even the best, GPT-5.4-mini, closed only about 34.4% of the consensus gap (all-mediator average 25.9%) ・Big domain spread: 41.3% improvement in transactional disputes versus just 16.6% in intra-organizational ones The key takeaway: meaningful progress needs better social adaptation to diverse conditions, not just general capability gains. #LLMEvaluation# #AIMediation#