I just published my roadmap for building evals on highly subjective tasks. Here are the key steps involved and practical tricks for reducing noise / increasing confidence… Blog is here: Why is this difficult? Evaluating agents is already hard due to their broad scope, emphasis on expensive / long-horizon tasks, and non-determinism. Subjective tasks make this even harder because we cannot deterministically verify solutions and must rely on human judgments (or LLM judges). This introduces additional noise into an already difficult evaluation process. Evaluation roadmap. We can build reliable evaluations using a repeatable framework with three components: - Human evaluation: Our North Star for quality. Create quality guidelines that clearly define quality with the help of subject-matter experts, calibrate human evaluators to ensure agreement, and continually monitor evaluation quality. - Offline evaluation: Use calibrated LLM judges or task-specific rubrics as an efficient proxy for human evaluation. This allows us to scale experimentation without relying on expensive human evaluation for every new model. - Production monitoring: Track quality signals from real-world agent usage and use A/B testing to validate improvements before fully deploying a new agent. Multi-stage evaluation. These components form a tiered evaluation process: Offline evaluation → Human evaluation → Production A/B test → Full deployment We can test many ideas automatically, send promising candidates to human evaluation, and validate improvements through production A/B testing. Continual production monitoring then allows us to discover new use cases / failure cases in production that we can add into the earlier stages of evaluation to create a feedback loop. Sampling useful data. Evaluation quality depends heavily on which tasks we select. We want tasks that are diverse, realistic, difficult, discriminative, and accurate. Techniques like structured task taxonomies, model-based difficulty filtering, Item Response Theory (IRT), and LLM-based auditing help us construct more informative evaluation sets. Handling noise. We should leverage statistics throughout the evaluation process: - Power analysis: Estimate the number of evaluation samples needed to detect meaningful improvements. - Confidence intervals: Quantify uncertainty and provide clear confidence signals alongside evaluation results. - Re-sampling: Repeat agent / judge evaluations to reduce within-task variance and potentially tighten confidence intervals. - Paired comparisons: Evaluate models on the same tasks to more reliably detect performance differences. Key takeaway. An improved benchmark score does not always mean our agent is more capable. We need well-designed evaluation tasks, calibrated human / model evaluators, and statistical confidence signals to distinguish meaningful capability improvements from noise (especially on subjective tasks). By combining these techniques into a repeatable evaluation process / roadmap, we can make building and maintaining evaluation suites more systematic, reliable, and scalable.

Was reading this thread on pretraining data for Marin and it mentions a great set of data curation papers. This paper ( was especially interesting. Had not seen it before, but the core argument is that data selection should depend on available compute. E.g., if you have tons of compute, training on lower quality data may be better than repeating high-quality data. Simple but useful! "We demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data... Our scaling laws characterize the differing 'utility' of various quality subsets of web data; account for how utility diminishes for a data point at its 'nth' repetition; and formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them."

I’m at AIE NYC this week, and I’ll be giving a talk on evaluating subjective / open-ended tasks! Talk details: - Wednesday, October 14, 3:20pm - 3:38pm, Turing Room (Riverside Ballroom) - I’ll follow up with slides, links, full details, reading material, and more resources after the talk. Motivation. Evaluating LLMs is difficult, and much of this difficulty comes from noise. How do we know if a better eval result is due to a better model or just attributable to noise / variance? We need a methodology for building evaluations that yield confident / useful information despite the non-determinism of LLMs. Subjective tasks. The problem of noise in evaluations becomes even harder when working with subjective tasks. Recent LLM research focuses heavily on verifiable tasks, where correctness is judged with a deterministic check (e.g., string matches, symbolic verification, test cases, etc.). For subjective tasks, even defining correctness is an open question. We must clearly define what constitutes a correct solution and try to capture this with an LLM judge or rubric. Then, even our verifier is non-deterministic, thus making the evaluation process noisier. The subjectivity of the evaluation process makes it really easy to build poor evals that provide no useful information about model quality. Roadmap for Highly-Subjective Evals. To address these issues, my talk will outline a roadmap for properly evaluating subjective tasks. The roadmap consists of three primary components: 1. Human evaluation: the North Star for evaluation, must be calibrated and come with clear guidelines. 2. Offline (automatic) evaluation: an efficient (but imperfect) proxy for human evaluation that allows experiments to be scaled. 3. Production monitoring: key metrics that allow us to monitor quality in the wild and create “soft” model deployments via A/B testing. Multi-stage evaluation. These components create a tiered evaluation strategy for new models and ideas: Start with offline evaluation > Human evaluate promising candidates > A/B test in production > Deploy We have multiple stages of evaluation that each are used to detect different kinds of errors. By combining them together, we can evaluate model quality holistically, detect a much larger number of errors, and (hopefully) avoid surprises in production. Handling noise. Finally, there are a variety of techniques we can use to minimize the amount of noise in this evaluation process and increase confidence in our evaluation results: - Create clear quality guidelines for human evaluation. - Calibrate the human evaluation process to ensure agreement. - Calibrate LLM judges / rubrics to ensure agreement with humans. - Invest into proper data sampling for offline evaluation (diverse, realistic, difficult, and accurate). We should also leverage statistics at every step of the evaluation process. There are a core set of statistical concepts that can be used to quantify and reduce noise in LLM evals. We can use power analysis to estimate necessary sizes and setups for benchmarks / evaluation exercises, include confidence intervals with all of our evaluation results, re-sample model / judge completions multiple times to tighten confidence intervals, or even leverage pairwise model comparisons to more confidently detect differences in model evaluation results.

Datology is one of the best (if not the best) data research labs. Their tech reports are packed with practical details and tricks for curating data properly across pre / post-training and evals. Some of my favorite resources and write ups from them: - DatBench: - DatSmith: - Pretrain vs. CPT analysis: - UberWeb: - BeyondWeb: Really cool to see them packaging these learnings / best practices into a unified platform!

Here how multi-teacher on-policy distillation (MOPD) works… TL;DR: Multi-Teacher On-Policy Distillation (MOPD) provides a way to consolidate capabilities from several specialized teacher models into a single student. Rather than training one policy to become an expert in many domains simultaneously, we can train specialists independently (often in parallel) and distill them back into one model using dense, token-level supervision. On-policy distillation (OPD) basics. MOPD builds upon the idea of OPD. Importantly, OPD is on-policy from the perspective of the student. Given a prompt, the student (not the teacher!) generates a rollout from this prompt. Then, the teacher computes probabilities for each student-generated token in this rollout, and we train the student to match the teacher using a (sampled) reverse-KL objective. Put simply, we use the teacher to score the observed behavior of the student, rather than actually generating a trajectory with the teacher. Extension to MOPD. MOPD extends this setup to multiple teachers. Instead of scoring every rollout with the same teacher, each prompt / rollout is routed to the teacher that specializes in the corresponding domain for that prompt. A SWE rollout will be scored by a SWE teacher, a chat prompt is routed to a conversational teacher, and so on. Dense supervision. One major benefit of OPD is that it provides a dense learning signal. Outcome-based RL (most common RL training setting for LLMs) assigns one reward to a full trajectory, whereas OPD provides supervision at every generated token. This dense signal can make distillation more sample efficient relative to outcome-based RL. However, OPD / MOPD and RL are not mutually exclusive—we can combine the distillation objective with a standard policy-gradient objective and learn from both teacher probabilities and task-specific rewards. Specialized teachers. The teachers used for MOPD can be independently optimized for different capabilities (e.g., reasoning, coding, SWE, search, etc.). Teachers can use their own post-training pipeline—combining SFT, RLVR, RLHF, and more depending on what works best for the domain. For example, Nemotron 3 Ultra trains ~10 teacher models by applying bespoke training pipelines (with different combinations of post-training algorithms); see image. Even though teachers often use their own training pipelines, teachers cannot be combined arbitrarily: if a teacher's behavior differs too much from the student, student-generated rollouts may be out-of-distribution for that teacher, weakening the supervision signal. This can be solved via things like MOPD warmup, where we train the student with SFT on teacher rollouts prior to MOPD to get student behavior closer to that of the teachers. Why is this helpful? Scaling RL across many domains is difficult. As we add domains, each capability receives fewer rollouts, domains can interfere with one another, and differences in rollout length / verification cost make training infrastructure increasingly complicated. MOPD gives us another approach: train domain specialists independently (often in parallel) then consolidate their capabilities into a single policy. It can also be used to recover capabilities that regress during sequential stages of post-training.

One of the most interesting details I learned from the Nemotron tech reports was that certain teachers are not compatible to use in MOPD together... Multi-Teacher On-Policy Distillation (MOPD) is a distillation technique that trains a student model on its own rollouts. Given a prompt, we: - Generate a completion from the student. - Route the prompt + completion to an expert / teacher model. - Compute token-level log probabilities from the teacher. Then, we can train the student model using a reverse-KL objective (usually a sampled approximation) between student and teacher probabilities to distill the capabilities of the teacher(s) back into the student policy. Why is this useful? There are a few different setups that can be used for MOPD that differ in how they define / create the teacher models: (1) Consolidation: if we have domain-specific specialist models that have each been created using their own training pipeline, we can consolidate their advanced capabilities into the student with MOPD. This is a great way to improve the capabilities across a broad range of domains, and specialist training is independent across domains (as opposed to multi-domain RL). (2) Recovery: if we are running a sequential / multi-stage training process, we can use checkpoints from earlier training stages as teachers to restore capabilities that regress as we continue to progress through the training process. For either of these cases, we learn from Nemotron 3 Ultra that we cannot just use an arbitrary set of models as teachers for MOPD. See the following quote: “One key finding from our MOPD trials is that teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward MOPD merge, resulting in suboptimal performance.” - Nemotron 3 Ultra Tech Report MOPD teachers need to be sufficiently compatible with the student. If a teacher was trained with a very different pipeline, its reasoning / output distribution may differ substantially from the student's. Because MOPD scores student-generated trajectories, those trajectories can then be out-of-distribution for the teacher—making its token-level supervision much less useful. This is primarily a problem for the consolidation setup rather than recovery, because the recovery setup by definition draws teacher models from the same training lineage as the student. One solution to this problem is MOPD warmup. Namely, before distillation, we can run a short SFT stage on a diverse set of trajectories sampled from the teacher models. This moves the student closer to teachers’ behavior, making student-generated rollouts more in-distribution. So, MOPD is not about training / distilling experts into a student. Ensuring that students / teachers are compatible is an important part of getting MOPD to work.

I just published my overview of the NVIDIA Nemotron series (focusing on post-training)... You can read it here: Among open models / resources, Nemotron is especially useful due to its transparency. Releases often include detailed tech reports, code, training recipes, and even data in some cases. There are many minute details involved in learning how to post-train LLMs at scale, but they are hard to learn without detailed accounts from teams that have actually trained frontier-scale models. NVIDIA Nemotron provides us with a rare opportunity to study these details directly and gain an understanding of how LLM post-training actually works. The overview covers the following, which encompasses the majority of recent Nemotron tech reports: - Llama-Nemotron: - AceReason-Nemotron: - Nemotron-Cascade: - Nemotron-Cascade 2: - Nemotron 3 Nano: - Nemotron 3 Super: - Nemotron 3 Ultra: Through these works, we can see a clear progression in post-training methodologies. Relatively simple pipelines slowly transform to become more agentic, and new components (e.g., MOPD and async / long horizon RL) are introduced over time. Beyond these high-level trends, we can see all of the small details that are useful for getting the most performance out of your model; e.g., curriculum learning, multiple training stages, synthetic data recipes, data filtering / curation / mixing, balancing reasoning efforts, harness exposure, and more. I highly recommend reading this blog (and the associated tech reports) for those who want a realistic view into the practical details of LLM post-training.

I'm getting ready to publish my blog post on the series of NVIDIA Nemotron models (focusing primarily upon post-training) tomorrow morning. Due to providing open weights, data, code, and more, these are some of the most useful tech reports for learning about recent trends in research and minute details of LLM training. I spent a ton of time while on vacation the last few weeks reading through the hundreds of pages of tech reports for these models. It's so interesting to see how trends emerge: - Training becomes more agentic over time. - RL infrastructure evolves for scale. - Varying kinds of RL training (multi-domain versus sequential) are used. - MOPD becomes more common. LLM research moves so quickly, and you can track a lot of the trends through the evolution of these papers / models. Excited to share this with everyone!

Love the fact that Jev is bringing focus to structured / open-ended classification problems with encoder-style models. There is so much to be unlocked here... it brings a new dimension to LLM workflows with a lot of interesting opportunities. Interestingly enough, this was one of my earliest research interests. See the paper below where I trained a multimodal BERT model to solve >1000 multi-class classification problems with a single model. These types of models are really capable and can do so much if you train / scale them properly! However, it's worth mentioning that Jev goes well beyond basic encoder classification. Some examples of novel components are: - Generalizing to any classification space in a scalable manner with good inference efficiency (probably some improved version of CLIP-style contrastive approach). - In-depth, data-centric research to do this properly and maximize model performance with training scale. - Figuring out how to structure the model in a way that yields really impressive inference efficiency. These are all changes that create something completely different than just encoder + classification. Even small details make a massive difference in adoptability and utility for these types of models. Jev (or similar models) provide an orthogonal tool to standard autoregressive LLMs that is really helpful and can be integrated into LLM workflows with minimal added cost. There are so many directions we can go with these models IMO. Need to do a deeper dive and will follow-up with a full writeup soon.

With the external hack of OpenAI via Claude, closed models continue to be the tip of the iceberg on AI risks, not open models. They have been 1) easier to get started with, 2) more capable & 3) shipped w/ leaky safeguards. Finetuning open models to specific attacks is harder.

I would consider helping as an independent evaluator along with the non profit I’m building. Is going to be a crucial role and there aren’t many independent people with substantial technical expertise. I think I represent an important moderate camp on AI capabilities and risks.

Great visualization and a nicely-executed idea. Many tools are available for model specialization. A lot of work focuses on harnesses or finetuning separately, but why not use them together? Bullish on efficient recipes for optimizing models in custom / task-specific harnesses!

Really awesome MOPD reading list. Here is also another well-written blog that synthesizes important work in this space (mostly from OSS model reports) for those who are interested:

@natolambert everything is so extreme / exaggerated. wouldn't be surprised if google has a comeback soon too. every company that has 3 bad months of progress is talked about as if they no longer exist / are doomed (oai, google, meta, etc.)