Distillation isn't a research project anymore. It's a pipeline. Weave traces → dataset curation → Serverless SFT → eval → deployment. Join us for 40 minutes on turning real production usage into a smaller student model that meets your bar.

A common technique for distillation attacks involves extracting the model’s chain of thought. To make these attacks more difficult, we are introducing a change to the Messages API: it will no longer be possible to edit Claude's context prior to thinking blocks in a multi-turn conversation. We've done this in the least disruptive way we can: it applies only to new accounts using Fable 5.1 (distillation often makes use of large numbers of new, fake accounts), although we plan to roll out this change to all users for future model releases. We hope to find better solutions to distillation attacks in the long run.

OPID does skill distillation for agentic RL using the agent's own completed trajectories as the skill source, so the supervision stays on-policy and matched to the current state distribution. Hand-written or off-policy skill libraries drift from what the policy actually visits in multi-turn interaction, which is the gap it targets. Skills are extracted in hindsight from finished rollouts and split into two levels: episode-level skills that capture global workflows and failure-avoidance rules, and step-level skills for local decisions, with a critical-first routing that distills the high-leverage ones back into the policy. OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning Paper:

On-policy distillation has the same systems bottleneck as RL: rollouts dominate training time on reasoning workloads. Going async fixes throughput but feeds the learner stale-policy data, and what staleness does to OPD specifically was unstudied. The clean finding is that KL direction decides robustness. Teacher-weighted forward KL shrugs off stale rollouts, student-weighted reverse KL breaks under them, and for the reverse-KL case nothing from async RL beats just recomputing the signal under the current student. Finite teacher-score caches then turn the estimator into a bias-variance tradeoff, which is the case for multi-sample Monte Carlo. AsyncOPD: How Stale Can On-Policy Distillation Be? Paper:

Adding an on policy distillation section to the RLHF book and it’s remarkable how bad LLMs / coding agents are at it, despite me giving them the core papers and 250 pages of context on how I present ideas.