⚡ Working on a major speedup for AsyncGRPO in TRL: training steps are up to 3.5× faster so far! AsyncGRPO already has several packing optimizations. One packs samples together and balances the attention cost across DP ranks. You start with the same prompt G times. In multi-turn rollouts on a harness, the conversation can also start to fork: most of the history remains identical, while a tool result or a new turn changes the suffix. That gets expensive with GRPO because rollouts can share many long prefixes. You start from the same prompt G times, and multi-turn rollouts on a Harness can also share most of their conversation history (thanks to message-level tokenization). Each rollout is treated as its own sequence and packed on a single row, subject to a predefined token budget per DP-rank. Those rows are forwarded independently, even when most of their tokens are identical. Tree packing instead turns the rows into a prefix tree. Shared prefixes are stored and forwarded once, while we use FlexAttention to ensure that every token still sees exactly the context it would have seen in its original rollout. The loss is still defined on the original rollouts, so every trained token keeps its own target, advantage, and old log-prob. The video shows the algorithm pretty well 👇

First multi-turn agent harness RL training with sandboxing running at scale on the Hub 🎉 🤗 Spawned 9,523 sandboxes in 14h. Qwen3-Coder-30B-A3B (30B MoE) trained in full, FSDP2 + expert parallel. All on HF Jobs. No Slurm anywhere. 0 crashes 🤗

We ran async GRPO across HF Jobs with the trainer and vLLM on separate machines, no NCCL, and no shared disk. The adapter is mounted at the same path in every Job. LoRA training has landed in TRL's AsyncGRPOTrainer 🎉🎉🎉 The trainer now syncs only the adapter to vLLM ~few MB for a 1.5B model. That changes what the setup can look like. We ran async GRPO across HF Jobs with the trainer and vLLM on separate machines, no NCCL, no shared disk. The adapter travels through a 🪣 mounted on the same path in every Job. Blogpost 🧵