Xiuyu Li(@sheriyuo):On-policy distillation has the same systems bottleneck as RL: rollouts dominate training time on reasoning workloads. Going async fixes throughput but feeds the learner stale-policy data, and what staleness does to OPD specifically was unstudied. The clean finding is that KL direction decides robustness. Teacher-weighted forward KL shrugs off stale rollouts, student-weighted reverse KL breaks under them, and for the reverse-KL case nothing from async RL beats just recomputing the signal under the current student. Finite teacher-score caches then turn the estimator into a bias-variance tradeoff, which is the case for multi-sample Monte Carlo. AsyncOPD: How Stale Can On-Policy Distillation Be? Paper:

2026.06.24 09:49

On-policy distillation has the same systems bottleneck as RL: rollouts dominate training time on reasoning workloads. Going async fixes throughput but feeds the learner stale-policy data, and what staleness does to OPD specifically was unstudied. The clean finding is that KL direction decides robustness. Teacher-weighted forward KL shrugs off stale rollouts, student-weighted reverse KL breaks under them, and for the reverse-KL case nothing from async RL beats just recomputing the signal under the current student. Finite teacher-score caches then turn the estimator into a bias-variance tradeoff, which is the case for multi-sample Monte Carlo. AsyncOPD: How Stale Can On-Policy Distillation Be? Paper: