I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.