I’m at AIE NYC this week, and I’ll be giving a talk on evaluating subjective / open-ended tasks! Talk details: - Wednesday, October 14, 3:20pm - 3:38pm, Turing Room (Riverside Ballroom) - I’ll follow up with slides, links, full details, reading material, and more resources after the talk. Motivation. Evaluating LLMs is difficult, and much of this difficulty comes from noise. How do we know if a better eval result is due to a better model or just attributable to noise / variance? We need a methodology for building evaluations that yield confident / useful information despite the non-determinism of LLMs. Subjective tasks. The problem of noise in evaluations becomes even harder when working with subjective tasks. Recent LLM research focuses heavily on verifiable tasks, where correctness is judged with a deterministic check (e.g., string matches, symbolic verification, test cases, etc.). For subjective tasks, even defining correctness is an open question. We must clearly define what constitutes a correct solution and try to capture this with an LLM judge or rubric. Then, even our verifier is non-deterministic, thus making the evaluation process noisier. The subjectivity of the evaluation process makes it really easy to build poor evals that provide no useful information about model quality. Roadmap for Highly-Subjective Evals. To address these issues, my talk will outline a roadmap for properly evaluating subjective tasks. The roadmap consists of three primary components: 1. Human evaluation: the North Star for evaluation, must be calibrated and come with clear guidelines. 2. Offline (automatic) evaluation: an efficient (but imperfect) proxy for human evaluation that allows experiments to be scaled. 3. Production monitoring: key metrics that allow us to monitor quality in the wild and create “soft” model deployments via A/B testing. Multi-stage evaluation. These components create a tiered evaluation strategy for new models and ideas: Start with offline evaluation > Human evaluate promising candidates > A/B test in production > Deploy We have multiple stages of evaluation that each are used to detect different kinds of errors. By combining them together, we can evaluate model quality holistically, detect a much larger number of errors, and (hopefully) avoid surprises in production. Handling noise. Finally, there are a variety of techniques we can use to minimize the amount of noise in this evaluation process and increase confidence in our evaluation results: - Create clear quality guidelines for human evaluation. - Calibrate the human evaluation process to ensure agreement. - Calibrate LLM judges / rubrics to ensure agreement with humans. - Invest into proper data sampling for offline evaluation (diverse, realistic, difficult, and accurate). We should also leverage statistics at every step of the evaluation process. There are a core set of statistical concepts that can be used to quantify and reduce noise in LLM evals. We can use power analysis to estimate necessary sizes and setups for benchmarks / evaluation exercises, include confidence intervals with all of our evaluation results, re-sample model / judge completions multiple times to tighten confidence intervals, or even leverage pairwise model comparisons to more confidently detect differences in model evaluation results.

Quick lookback on technical AI themes of 2026: (dataset: daily ainews title stories) Jan: - Ralph Loops - RLMs - Filesystems are all you need - X LLM recsys open sourced - X Grok Imagine #1# Video model - Anthropic blocked OpenCode / OpenClaw - MCP Apps - Moltbook - Apple picks Gemini for new Siri - Notable models: Kimi K2.5 - Fast inference: OAI x Cerebras partnership - Big raises: xAI @ $230B + SpaceX, OpenEvidence @ $12B Feb: - Codex App launch - Claude Code 1 year anniversary - Context Graphs - Fast Inference: Taalas 17k tok/s/user Llama 3 8B - Anthropic vs Deepseek/Moonshot/Minimax distillation accusation - Notable models: Opus/Sonnet 4.6, 5.3 Codex and Codex Spark, Qwen-Image 2.0, Seedance 2.0, GLM-5, Gemini 3.1 Pro and Deep Think, Nano Banana 2 - Big raises: OpenAI @ $840B, Anthropic @ $380B, ElevenLabs @ $11B, Cerebras @ $23B Mar: - T3 Code - Autoresearch - Claude Code source leak - Claude Cowork Dispatch - OpenAI buys Astral - Cloudflare-Next.js "Slop Fork" - Meta buys Dreamer - NVIDIA GTC: OpenClaw, Vera CPU, 1T sales backlog, "The Inference Inflection" - ARC-AGI 3 introduced, SOTA models are 1% - Notable models: GPT 5.4 unifies chat and codex, Gemini 3.1 Flash-Lite/Live, Nemotron 3 Super, Composer 2, MiniMax 2.7, Arcee’s Trinity-Large-Thinking - Big Raises: Yann LeCun AMI Labs $1B seed Apr: - Claude Design - OpenAI Symphony & Harness Engineering discourse - @LilianWeng writeup later - OpenAI and Microsoft break up - Tokenmaxxing fears - Codex Superapp - AIE Europe, AIEi Miami - Hermes Agent gains traction w/ Manim skill - Cyber: Anthropic Project Glasswing - Notable models: Claude Mythos "Capybara", Opus 4.7, Muse Spark, Gemma 4, Kimi K2.6, GPT-Image-2, GPT 5.5 "Spud", DeepSeek v4 Pro/Flash Base/Instruct on Huawei Ascend, Poolside Laguna XS.2, Grok 4.3 - Big Raises: not much but Anthropic $19B->$30B ARR in one month... May: - Cyber: OpenAI Daybreak, Tanstack Mini-Shai Hulud - FDE: OpenAI DeployCo, Ode by Anthropic - OpenAI solve Erdos planar unit distance problem - OpenAI deprecates finetuning - "How to land a job at a frontier lab" - Anthropic rents Colossus I from SpaceX for $5B/yr - Thinking Machines Interaction Models (still underrated!!!) - Notable models: GPT-Realtime-2/Translate/Whisper, Gemini 3.5 Flash, Omni, Antigravity 2.0, Opus 4.8, Cosmos 3, Minimax M3 - Big Raises: Fireworks @ $15B, Baseten @ $11B, Cognition @ $26B, Cerebras $60B IPO, Anthropic @ $965B Jun: - Claude Tag - OpenAI Jalapeno - "The Untrainable" - @saranormous - Loopcraft: The Art of Stacking Loops - Metaharnesses discourse? Flue v Eve - Zai Prof @jietang predicts Open Fable by Dec - "SpaceX is a $28B/yr neocloud" - @jaminball - Anthropic-Fable White House issue (resolved) - Notable models: GPT 5.6 Sol/Terra/Luna, Fable 5, Sonnet 5, MAI-Thinking-1 and MAI Family, Nemotron 3 Ultra, GLM 5.2, Reve 2/ Ideogram 4 - Big Raises: SpaceXai buys Cursor for $60B, Supabase @ $10B Jul: - AIE World's Fair 3 - OpenCode 2.0 - Cyber: OpenAI-HuggingFace incident - Chinese Model ban discourse - GPT 5.6 Luna 80% price cut - Notable models: Grok 4.5, Opus 5, Muse Image, Thinky Inkling, Kimi K3, BFL FLUX 3, DeepSeek-v4-Flash-0731 - Big Raises: Aug: - EU AI text watermarking - Microduck - Megakernels - Memory prices up 500% in 12 months - How to steal a Reasoning Trace - Grok @bot! - OpenAI shuts off Cursor - Fast Inference: AMD buys Taalas - @andrewng stars AI Engineering series - Hot Chips: Jalapeno, Cerebrase CS-5, Groq 3, Apple M6 - Demis steps down from CEO, & Dean et al leave - Notable models: Qwen 3.8 Max, Muse Spark 1.2, Muse Glimmer, Grok 4.6, Gemini 3.7 Flash, GLM 5.3, @Fal H3 Max Live - Big Raises: Stripe buys OpenRouter $7B, Poolside $12B reverse-execuhire to NVIDIA, NVIDIA buys HuggingFace for $13B Sep: - AIEi Paris - OpenAI Navier-Stokes singularity - collusion . wiki - Yegge shuts down Gas Town - AEF-1 standard for Third Party Evaluators - Meta Muse App launch. Also new glasses and Charm - OpenAI blocks Cursor - OpenAI Dots - Opus 5.5 explainer videos - Pi 1.0 & Pi Durable - Notable Models: Fable 5.1, Mythos 5.1, Opus 5.5, GPT-6 Astra, GPT Image 2.5, DeepSeek v4.1-Flash, @typesafeai Jev, MiMo-v2.6-Pro new Open SOTA, GPT 6.1 Sol & Ultrafast & Decisions API, Gemini 4 Argon - Big Raises: Cognition @ $48B, Mistral @ $24B, AMD buys World Labs for $8.2B Oct to date: - AIE NYC - @dhh stops using Rails - ArtCraft clones Adobe suite - Interpretability vs Polysemanticity - OpenAI Quasi-Riemann-Hypothesis & 722 Math - Notable Models: Reflection Beam, Claude Haiku 5.5, Muse Wonder & Wonder Full reach Opus 5.5 tier - Big Raises: TypeSafe $7.5B 3 weeks after launch, Modal @ $16B (this was done for research purposes for the keynote in Nov, just sharing notes in case helpfuil)