One of the most interesting details I learned from the Nemotron tech reports was that certain teachers are not compatible to use in MOPD together... Multi-Teacher On-Policy Distillation (MOPD) is a distillation technique that trains a student model on its own rollouts. Given a prompt, we: - Generate a completion from the student. - Route the prompt + completion to an expert / teacher model. - Compute token-level log probabilities from the teacher. Then, we can train the student model using a reverse-KL objective (usually a sampled approximation) between student and teacher probabilities to distill the capabilities of the teacher(s) back into the student policy. Why is this useful? There are a few different setups that can be used for MOPD that differ in how they define / create the teacher models: (1) Consolidation: if we have domain-specific specialist models that have each been created using their own training pipeline, we can consolidate their advanced capabilities into the student with MOPD. This is a great way to improve the capabilities across a broad range of domains, and specialist training is independent across domains (as opposed to multi-domain RL). (2) Recovery: if we are running a sequential / multi-stage training process, we can use checkpoints from earlier training stages as teachers to restore capabilities that regress as we continue to progress through the training process. For either of these cases, we learn from Nemotron 3 Ultra that we cannot just use an arbitrary set of models as teachers for MOPD. See the following quote: “One key finding from our MOPD trials is that teacher models trained with substantially different training pipelines cannot be effectively combined through a straightforward MOPD merge, resulting in suboptimal performance.” - Nemotron 3 Ultra Tech Report MOPD teachers need to be sufficiently compatible with the student. If a teacher was trained with a very different pipeline, its reasoning / output distribution may differ substantially from the student's. Because MOPD scores student-generated trajectories, those trajectories can then be out-of-distribution for the teacher—making its token-level supervision much less useful. This is primarily a problem for the consolidation setup rather than recovery, because the recovery setup by definition draws teacher models from the same training lineage as the student. One solution to this problem is MOPD warmup. Namely, before distillation, we can run a short SFT stage on a diverse set of trajectories sampled from the teacher models. This moves the student closer to teachers’ behavior, making student-generated rollouts more in-distribution. So, MOPD is not about training / distilling experts into a student. Ensuring that students / teachers are compatible is an important part of getting MOPD to work.