𝗢𝗻-𝗽𝗼𝗹𝗶𝗰𝘆 𝗱𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻 𝗶𝘀𝗻'𝘁 𝗮 𝗳𝗿𝗲𝗲-𝗹𝘂𝗻𝗰𝗵 On-policy distillation has become a default post-training tool in many open-source frontier model training recipes. Recent releases lean on it heavily: DeepSeek v4, MiMO, and Nemotron-Cascade-2 use MOPD, and GLM 5.x uses on-policy cross-stage self-distillation. It provides RL's on-policy nature reducing exposure bias, while providing token level supervision like SFT. But OPD and OPSD have their own failure modes. In this post I discuss a few of them: 1. 𝗘𝗮𝗿𝗹𝘆 𝗺𝗶𝘀𝘁𝗮𝗸𝗲𝘀 𝗮𝗿𝗲 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗮𝗹𝗹𝘆 𝘂𝗻𝗰𝗼𝗿𝗿𝗲𝗰𝘁𝗮𝗯𝗹𝗲. When the student samples a rollout and takes an early wrong turn, the per-token KL computed along that frozen rollout cannot pull it back onto a correct path. TRD proves that this failure is built into the objective rather than being a matter of noisy gradients. Even with a perfect teacher, the gradient obtained from token-level KL on the student's own rollout agrees with the ideal corrective gradient at exactly one point, the token where the student first diverged, and disagrees everywhere after it. Every later supervision target is therefore anchored to a context that the student should never have entered. Because reweighting or clipping only rescales the magnitude of each token's gradient, and here the terms point in the wrong direction, no per-token adjustment can recover the correct update. TRD's proposed fix is to distill along a teacher-refined trajectory rather than the raw student rollout, which restores a target the student can actually follow. 2. 𝗔 𝘀𝘁𝗿𝗼𝗻𝗴𝗲𝗿 𝘁𝗲𝗮𝗰𝗵𝗲𝗿 𝗰𝗮𝗻 𝗯𝗲 𝗮 𝘄𝗼𝗿𝘀𝗲 𝘁𝗲𝗮𝗰𝗵𝗲𝗿. On-policy distillation can only teach the student at states the student itself visits, and the usable signal at each of those states lives in the overlap between the student's and teacher's next-token distributions. Rethinking OPD shows that a bigger, higher-scoring teacher can fail to move a student while a weaker one succeeds, because if the teacher's token distribution places its mass on tokens the student rarely produces, the overlap is small and almost nothing transfers, no matter how capable the teacher is in absolute terms. What actually predicts success is early top-k thinking-pattern overlap. In runs that work, the shared top-k tokens carry 97 to 99% of the probability mass and the overlap ratio climbs steadily during training, whereas a run that starts with low overlap never recovers it. A teacher trained on the same recipe as the student also converges toward the student's own distribution, so its higher benchmark score does not correspond to any new knowledge it can transfer. The practical rule is to pick teachers by distributional closeness to the student, not by leaderboard rank. 3. 𝗣𝗿𝗶𝘃𝗶𝗹𝗲𝗴𝗲𝗱-𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻-𝗰𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗲𝗱 𝗢𝗣𝗦𝗗 𝗰𝗮𝗻 𝗳𝗮𝗶𝗹 𝘁𝗼 𝘁𝗿𝗮𝗻𝘀𝗳𝗲𝗿. In OPSD you distill a teacher that was conditioned on privileged information, such as the gold answer, into a student that will never have it. The Many Faces of OPD shows what goes wrong when that information is instance-specific. The student cannot recover the teacher's per-instance reasoning, since it never sees the answer, so it instead learns a single answer-free policy that effectively averages the teacher's behavior across all problems, and that averaged policy is too generic to solve any particular one. The signature is initial gains followed by collapse: rollouts grow long, fill with hedging tokens, and accuracy craters toward zero. The approach works only when the privileged information is a shared rule that applies across all instances, such as a system prompt or an alignment preference, and not when it is a per-problem answer. 4. 𝗧𝗵𝗶𝗻𝗸𝗶𝗻𝗴 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲: 𝗱𝗲𝗻𝘀𝗲 𝘀𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗶𝗼𝗻 𝘀𝘂𝗽𝗽𝗿𝗲𝘀𝘀𝗲𝘀 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹'𝘀 𝗼𝘄𝗻 𝗱𝗲𝗹𝗶𝗯𝗲𝗿𝗮𝘁𝗶𝗼𝗻. A teacher conditioned on the answer has no reason to hesitate, backtrack, or explore, so its per-token targets quietly push down the student's deliberation tokens. Diagnosing and Mitigating Thinking Collapse names this phenomenon thinking collapse: over training, the student's native reasoning behavior erodes as the exploratory tokens that carry it, words like wait, maybe, and alternatively, become progressively less frequent. The mechanism is local rather than global. The damage concentrates at high-entropy decision forks, the branch points where the student is genuinely uncertain and would normally deliberate. Exactly there, the student's top-1 token is often an exploratory marker while the answer-conditioned teacher's top-1 token is not, so the mismatch produces a strong gradient that suppresses the very tokens that make reasoning work. The result is a model whose native reasoning behavior is measurably suppressed, and downstream reasoning accuracy falls in step with it.