I’m at AIE NYC this week, and I’ll be giving a talk on evaluating subjective / open-ended tasks! Talk details: - Wednesday, October 14, 3:20pm - 3:38pm, Turing Room (Riverside Ballroom) - I’ll follow up with slides, links, full details, reading material, and more resources after the talk. Motivation. Evaluating LLMs is difficult, and much of this difficulty comes from noise. How do we know if a better eval result is due to a better model or just attributable to noise / variance? We need a methodology for building evaluations that yield confident / useful information despite the non-determinism of LLMs. Subjective tasks. The problem of noise in evaluations becomes even harder when working with subjective tasks. Recent LLM research focuses heavily on verifiable tasks, where correctness is judged with a deterministic check (e.g., string matches, symbolic verification, test cases, etc.). For subjective tasks, even defining correctness is an open question. We must clearly define what constitutes a correct solution and try to capture this with an LLM judge or rubric. Then, even our verifier is non-deterministic, thus making the evaluation process noisier. The subjectivity of the evaluation process makes it really easy to build poor evals that provide no useful information about model quality. Roadmap for Highly-Subjective Evals. To address these issues, my talk will outline a roadmap for properly evaluating subjective tasks. The roadmap consists of three primary components: 1. Human evaluation: the North Star for evaluation, must be calibrated and come with clear guidelines. 2. Offline (automatic) evaluation: an efficient (but imperfect) proxy for human evaluation that allows experiments to be scaled. 3. Production monitoring: key metrics that allow us to monitor quality in the wild and create “soft” model deployments via A/B testing. Multi-stage evaluation. These components create a tiered evaluation strategy for new models and ideas: Start with offline evaluation > Human evaluate promising candidates > A/B test in production > Deploy We have multiple stages of evaluation that each are used to detect different kinds of errors. By combining them together, we can evaluate model quality holistically, detect a much larger number of errors, and (hopefully) avoid surprises in production. Handling noise. Finally, there are a variety of techniques we can use to minimize the amount of noise in this evaluation process and increase confidence in our evaluation results: - Create clear quality guidelines for human evaluation. - Calibrate the human evaluation process to ensure agreement. - Calibrate LLM judges / rubrics to ensure agreement with humans. - Invest into proper data sampling for offline evaluation (diverse, realistic, difficult, and accurate). We should also leverage statistics at every step of the evaluation process. There are a core set of statistical concepts that can be used to quantify and reduce noise in LLM evals. We can use power analysis to estimate necessary sizes and setups for benchmarks / evaluation exercises, include confidence intervals with all of our evaluation results, re-sample model / judge completions multiple times to tighten confidence intervals, or even leverage pairwise model comparisons to more confidently detect differences in model evaluation results.