TRL now supports MOPD (Multi-Teacher On-Policy Distillation) natively❗️ many frontier open models use it. labs train one RL expert per domain and then distill them all into one model blog with what it is, who uses it and an experiment you can run on HF Jobs with one command

how do you post-train an agent? we laid it out as a 5-step curriculum, from SFT to RL through a real harness, with livestreams, code and docs for each step

last week we closed the Training Agents series IRL at the GitHub offices in SF 4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models > Learnings from enterprise RL (@t2govind Microsoft) > Environments and benchmarks in industry (@VBarres86633 Mercor) > How to train your agents with Miles (@nblintao RadixArk) > Why we built OpenEnv Arena (@xdotli BenchFlow) full recording + slides

ICYMI, last week we completed the Training Agents live series with Class 4 where we covered RL envs with 2 demos training coding agents! Video: Slides with links to all the resources:

the fourth and last session of the training agents series finally has a date 📅 Class 4: agent environments > thursday september 10th, 5pm cest / 8:30pm ist this one is about the environments themselves, the sandbox with tools and files where the agent works for many turns, and how you build and run one with trl and openenv

catching up on some bookmarked reads from the summer, reading Antidoom from @liquidai small reasoning models get stuck more easily when the task involves a long thinking trace and a hard problem. It starts repeating the same word over and over again ("Wait", "Alternatively"…), each repetition makes the next one likelier, and the generation is spent before it reaches an answer they measured it, 10.2% of completions for an early LFM2.5-2.6B checkpoint and 22.9% for Qwen3.5-4B at greedy. After training those drop to 1.4% and 1.0% the fix is FTPO (final token preference optimization). What I like is how narrow it is, it only touches the single token where the loop starts three ways it differs from DPO: > trains one token position, mid-generation, instead of whole sequences > spreads probability across ~20 plausible alternatives instead of swapping one overtrained token for another > keeps the regularizer in logit space, no softmax, so the rest of the vocabulary stays put the third one is what makes it usable. If you want to edit one position without disturbing the model, you can't have a loss that reshuffles the other 150k logits on the way and their explanation abt the result: the training teaches the model nothing new about math or code, it clears the failure mode that was blocking answers the model could already produce full blog > FTPO itself comes from Antislop, where it was built to strip overused phrasing. LiquidAI retargeted it to doom loops and under the hood it's a subclass of TRL's DPOTrainer with compute_loss overridden, around 90 lines of loss and no new trainer we documented that pattern in TRL's docs

from the vault: on-policy distillation in TRL got 40x faster with three fixes, a generation buffer, batched teacher requests, and binary-encoded logprobs enough to distill from 100B+ teachers, a 4B student gained 39 AIME25 points learning from a 235B one

we just released a new blog "Training a coding agent using the OpenCode harness in remote HF sandboxes with TRL and OpenEnv" you can take a real coding agent (OpenCode), let it run its own tool loop against real coding problems, and train it with RL on the exact tokens it produced and every rollout runs in its own remote HF sandbox, so rollouts scale out beyond one machine the loop: - OpenCode owns its tool loop inside an OpenEnv sandbox - an in-sandbox proxy records the real token ids + logprobs, per turn - a hidden-test verifier scores the result, and that is the reward - TRL trains with AsyncGRPO, weights sync back to vLLM over NCCL blog + runnable example: