🧭 TL;DR: Instead of estimating an LLM's confidence from the current inference alone, this method calibrates it against how often similar past attempts actually succeeded. It beats 10-sample self-consistency in accuracy while costing far less. Title: Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents URL: Points 🗂 Stores past task, reasoning trace, stated confidence, actual outcome, and a lesson learned in an experience bank 🔍 Recall stage retrieves k=50 similar episodes and computes their real hit rate statistically 💭 Reflect stage has the model restate confidence in words after seeing that track record 🏆 Matches or beats SC@10 on 23 of 24 model-dataset combinations by AUROC 🤖 Biggest gains on agent tasks where failures are silent — even beats a trained verifier on AppWorld 📉 2-20x lower calibration error across domains, at roughly 1/10th the compute cost A great example of the shift from "judge the current attempt" to "judge from accumulated track record." #LLMEval# #AIAgents#

Ever spend way too long hunting across scattered sources just to find "which benchmark actually fits this task"? Title: Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation URL: ❓ What is Benchmark Radar? 💡 It's a search engine that crawls 37 sources daily, arXiv, Hugging Face, GitHub and more, and pulls each benchmark's paper, dataset, repository, and score history into one place, with a CLI for offline queries. ❓ How much data are we talking about? 💡 1,283 source records from four catalogs, covering 12,916 numeric scores across 790 benchmarks. Daily collection alone logs 11,068 observations across 6,546 distinct artifacts. ❓ Why can't scores just be compared directly? 💡 Only 82 of the scored records use a verified 0-100 percentage scale, the rest use different or unverified scales, and roughly half lack a known release date, so naive comparisons would be misleading. ❓ So what's the fix? 💡 Instead of forcing comparability, it preserves source identity and citations so readers can inspect the evaluation conditions themselves. #LLMEval# #Benchmarks#

LLMs now beat humans on math accuracy. But does that mean they're actually building understanding on top of the right foundations? Title: Do LLMs Exhibit Coherent Knowledge Structures in Mathematical Reasoning? A Perspective from Knowledge Space Theory URL: ❓ Do LLMs actually build on prerequisite knowledge to get answers right? 💡 Accuracy favors LLMs (92.5% for the best model, Qwen3-80B, vs. 79.6% for humans), but "perfect prerequisite satisfaction" tells a different story: 72.7% for humans vs. only 48.16% for the best LLM. Lots of correct answers rest on shaky foundations. ❓ Does giving them prerequisite hints help? 💡 Surprisingly, prerequisite-grounded context barely outperformed unrelated examples. That points to surface-level pattern matching rather than genuine structured reasoning over prerequisites. ❓ Do strong models at least share a consistent knowledge structure with each other? 💡 Human learner groups overlap at 0.9+ in their knowledge structure, but LLM pairs only overlap 0.38-0.6 — and stronger models diverge even further from humans. ❓ So what's the takeaway? 💡 Accuracy alone hides how differently LLMs "understand" math. Knowledge Space Theory offers a lens that exposes the fragmented structure lurking behind impressive scores. #LLMEval# #MathReasoning#

A pipeline with 100% delivery rate, 100% schema validity, and zero retries or errors... yet re-running the exact same request flips the verdict. This paper reports that shocking negative result, fully preregistered with a complete audit trail. Title: Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints URL: 🔧 Highlight 1: Clean engineering does not mean reliable measurement Across 3,312 planned calls, response rate and schema validity both hit 100% with zero retries or errors, yet the repeat-ranking agreement (median Spearman) was only 0.400 — clearly failing the preregistered stability gate (threshold 0.90). 🎲 Highlight 2: Byte-identical inputs still drift Replaying the exact same request 24 hours later gave only a 0.780 exact-ranking agreement rate. Interestingly, comparisons across windows on the same day scored a nearly identical 0.805 median, revealing this isn't "next-day drift" but immediate platform-level nondeterminism. 📉 Highlight 3: More samples don't fix it Scaling observer calls from 8 to 500 (748,000 calls total) still left the stability gate pass rate at 0/500. The root cause: score gaps between candidates were 7-10 orders of magnitude smaller than the noise floor, making most tasks fundamentally unrankable. I think this is a great reminder that before using an LLM as a measurement instrument, you need to measure whether the instrument itself is stable. #LLMEval# #Reproducibility#

How do you actually build evaluation benchmarks for AI agents at scale? LangChain just shared their full approach. Title: How We Build Agent Environments & Tasks URL: ❓ What exactly makes up an agent "task"? 💡 A task has three components: an input, an environment, and a test script. The environment hosts agent execution, and a rubric defines scoring criteria. A "world spec" consolidates shared domain knowledge across related tasks — API schemas, data generation methods, trace parsing scripts — in one place. ❓ How do you create tasks efficiently at scale? 💡 LangChain uses a two-step pipeline. First, a "spec generation" phase where a coding agent scans repositories, groups traces, maps credentials, and auto-generates an initial world spec. Then a "Spec2Task" phase converts that spec into runnable evaluation tasks. The spec from the very first task becomes the foundation, iteratively refined through subsequent task creation cycles. ❓ What are the most common mistakes when building evaluation tasks? 💡 Three pitfalls stand out: ・Don't skip running tasks with real agents — paper evaluation won't surface environment flaws ・Calibrate difficulty across model tiers (e.g. gpt-5.6-Luna vs Sol) — what's hard for one may be trivial for another ・Match the data generation method to the data type: LLM-based approaches for free-text, SQL scripts for tabular data ❓ Is a benchmark "done" once you've built it? 💡 Not at all. Continuous improvement from production data is the core idea. Real production traces feed back into cost modeling, prompt simplification validation, and tool configuration testing — the benchmark evolves alongside the system it measures. Treating evaluation environment engineering as ongoing rather than a one-time project is the key practical insight here. #AIAgents# #LLMEvaluation#

Judging billions of trace tokens a day with a frontier LLM was financially untenable 💸 Here's how fine-tuning a small open model matched the accuracy at 10-100x lower cost. Title: Building a 100x Cheaper Trace Judge with Fireworks URL: 💸 Overview LangChain Labs partnered with Fireworks to build a "Perceived Error" detector for agent traces. It detects cases where users felt the assistant made a mistake or needed correction — using a small open model. ❓ Challenges Solved LangSmith processes billions of tokens daily across production traces. ・Evaluating them with frontier LLMs is prohibitively expensive at scale ・The question: how to cost-effectively mine important signals from every trace while maintaining frontier performance? 💡 Methodology & Proposed Approach ・Supervised fine-tuning (SFT) with LoRA on the open-source Qwen-3.5-35B, on Fireworks infrastructure ・Training data from two production datasets: chat-langchain (technical Q&A, 707 examples) and Fleet (no-code agent, 727 examples) ・It learns "Perceived Error," doing the judging without relying on huge frontier models 📊 Experimental Results ・Accuracy: the fine-tuned Qwen matched or exceeded frontier models (96.1% on chat-langchain, 90.8% cross-domain on Fleet) ・Cost: 10-100x cheaper than frontier models depending on trace volume ・Transferability: the model trained on chat-langchain beat all frontier models on Fleet without retraining #LLMEvaluation# #FineTuning#

Can an AI actually mediate a conflict between people? 🤝 A benchmark that tries to measure that, reliably, under realistic conditions. Title: SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations URL: 🤝 Overview This work proposes SoCRATES, a comprehensive benchmark for evaluating LLMs as mediators. An agentic pipeline builds realistic conflict scenarios across eight domains from actual public disputes, enabling automated and reliable evaluation of proactive LLM mediation. ❓ Challenges Solved Using LLMs to guide disputing parties toward agreement is gaining attention, but evaluating it is hard. ・Real conflicts shift constantly as disputants' emotions, intentions, and context change mid-mediation ・Existing benchmarks rely on a limited set of expert-authored scenarios ・They also score every turn against every topic, injecting noise that muddies the evaluation signal 💡 Methodology & Proposed Approach SoCRATES integrates three approaches. ・Agentic scenario curation: agents find genuine public disputes, restructure them into mediation scenarios, and filter for cases that truly need intervention ・Socio-cognitive probing: vary each scenario across five independent dimensions (strategic posture, party composition, conversation-history length, emotional reactivity, cultural identity) to pinpoint capability gaps ・Topic-localized evaluation: instead of scoring every topic at every turn, rate only the turns where a topic is actively discussed, reducing noise It spans eight domains: transactional, health, environmental, B2B, policy, international, legal, and intra-organizational. 🌍 Use Cases / Experimental Results The results were sober and revealing. ・The evaluator reached r=0.82 alignment with human experts (trajectory level), more than doubling baseline performance ・Among eight frontier LLMs, even the best, GPT-5.4-mini, closed only about 34.4% of the consensus gap (all-mediator average 25.9%) ・Big domain spread: 41.3% improvement in transactional disputes versus just 16.6% in intra-organizational ones The key takeaway: meaningful progress needs better social adaptation to diverse conditions, not just general capability gains. #LLMEvaluation# #AIMediation#

JSEF v1.10.0 is out 🎉 A model-agnostic LLM eval toolchain + 20+ new hard vulnerability samples. ✅ Swap any OpenAI/Anthropic-compatible model (GLM-5.3, Mythos, GPT…) ✅ One-click cross-model leaderboard (Recall/Precision/F1/accuracy) ✅ DeepSwe-style trials stability (Pass@1, N-run) ✅ Multi-state gate, JWT 4-step chain, async taint chain Star → #LLMSecurity# #AppSec# #WebSecurity# #JavaSecurity# #VulnerabilityDetection# #AISecurity#