GLM-5.2’s shift from GRPO-style group-wise optimisation to critic-based PPO for long-horizon agentic RL is worth paying attention to. Over the past year, the common narrative around LLM RL has been fairly simple: PPO is heavy, critics are expensive, and GRPO is lighter because it removes the critic. This view makes sense for short-horizon RLVR tasks, such as maths, coding problems, and verifiable QA. In those tasks, the structure is clean. The same prompt can produce multiple candidate answers, a verifier can judge which ones are better, and the model can learn through group-relative comparison. But agentic RL is different. A coding agent fixing a bug may need to read the issue, search the codebase, edit files, run tests, fail, locate the problem again, and try another solution. Final success only tells us that the whole trajectory worked. It does not clearly tell us which intermediate step was crucial, which step was a detour, or which action had no immediate reward but enabled later success. This is where GLM-5.2’s blog becomes interesting. It mentions that long trajectories are compacted and split into multiple sub-traces. For the same prompt, different rollouts can produce different numbers of trainable traces, and their lengths can be highly uneven. In this setting, group-wise optimisation becomes less natural, while critic-based PPO can estimate token-level advantages and handle credit assignment across long, irregular trajectories more effectively. So this is not simply “PPO is back”, nor does it mean “GRPO has failed”. A better way to put it is: GRPO is better suited for comparing multiple answers, while PPO is better suited for evaluating each step in a long chain of actions. When models move from answer generators to action takers, the key RL question also changes: from “which answer is better?” to “how should credit be assigned across a long sequence of actions?”

Training and rollout logprobs matched bit for bit on ROCm. The @RLKernel team integrated RL-Align/RL-Kernel with vllm-project/vime. A 200-step Qwen3-8B GRPO run on 8× @AMD MI300X recorded zero logprob mismatches between Megatron training and vLLM rollout. The strict path aligns reduction order, intermediate precision, rounding points, and math primitives across both sides. Deep dive:

How auditing RL Environments (ie. looking at the data) will help us explain model behavior "models are benchmark shaped" bc GRPO style RL is essentially - synthetic data generation by a model - induced by the environments + harness design - behavior (trajectories) reinforced by verifier design the "data" is RL environments & tasks, a model generates a series of tokens in its exploration of that environment trying to solve the task we don't really know what it will do, but we reward behavior that aligns with the verifier Encouraged behaviors: important: the only behavior that even has an opportunity to get rewarded is trajectories our environment "induces" or "encourages". something that isn't produced, can't be rewarded if our Harness (via a tool description) or Task design encourages lots of parallel search calls over a massive document corpus to find a few facts, the model will start copying that behavior because it got rewarded for it in the future, when some other problem is roughly shaped like that search Task, the model will mimic similar behaviors whether or not that strategy is optimal models do things that their data encouraged them to do via reward Tracing Behavior to Environments: every behavior can be fuzzily traced by the accumulation of different environment design and reward combinations the model saw over training i think we all explicitly know this but we don't have the interpretability tooling to understand induced behaviors at a fine-grained level over a massive training run this is why I'm very exicting that ppl are swarming around Trace analysis over Evals/Environments I bet a lot of "weird" or "bad" behavior comes from some quirks in environment design that inadvertantly rewarded that behavior the only thing that even has an opportunity to get rewarded Vestigial Behavior RL is in part inherently exploratory. Models need to do a bunch of stuff in rollouts to figure out what works, the verifier will reward what works But the verifier won't explicitly penalize any behavior that has a neutral effect but that co-occurs in a successful rollout this is how we get behavior that used to be helpful in some tasks, doesn't hurt current tasks, and thus persists over time similar to how humans have vestigial organs like the appendix i bet we can collectively figure out a ton about model behavior by mechanistically working backwards from the data and I'm bullish humans and agents working together will make a lot of progress on this in the next year

Fatih Porikli, VP of Technology at @QCOMResearch, highlights the following CVPR papers: 🪩 DisCo - a reinforcement learning approach that uses Group Relative Policy Optimization (GRPO) to improve identity diversity in generated images using rewards such as intra-image diversity, inter-image diversity, count accuracy, and image quality 🎨 Ar2Can - a novel "architect and artist" framework that separates scene planning and composition from image rendering, similar to how a human artist might work, to produce more controllable and coherent generations. 🖼️ PixelRush tackles the challenge of generating high-resolution images efficiently on mobile devices using cascade upsampling, latent-space refinement, patchification, and semantically guided noise injection to produce high-quality images without visible artifacts. 🖌️ InverFill tackles the challenge of image inpainting using inverted, semantically steered noise to preserve the background and eliminate boundary artifacts 🎥 Dig into the papers here:

Modern TTS models can sound great — and still fail badly on pacing, pauses, and prosody. We adapted DPO + GRPO to flow-matching models to tackle the tail end of TTS behavior:

You can now fine-tune Meta Muse Glimmer 30B for free! 🔥 Our free notebook also supports GRPO RL training. Unsloth trains Muse Glimmer 1.5× faster with 50% less VRAM vs FA2 setups. Train locally with 24GB VRAM. Guide: Notebooks:

🚀 MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scale How can we make small models stronger for on-device agent deployment? MERA uses stronger models to guide an iterative loop of RL/GRPO, skill learning, and router optimization. Student failures become verified demonstrations, reusable SkillBook procedures, and LoRA updates, helping the small model take on more work over time. 🔥 Results: Qwen2.5-Coder-1.5B: 28.7% → 49.7% coding pass Qwen3.5-2B on TAU-2: 14/35 → 18/35 Fine-tuned 2B matches an unadapted 4B model Don’t just route around small models. Evolve them. 📄 💻

Timelapse #14#; 7 hours (9:00am-4:00pm) What I did: - Started experimenting with quantization aware REAP - Studied online post-training techniques (DPO, PPO, GRPO) - Tested qwen3.8 through ambassador program - benched qwen3.8 @neutralityproject - Made 2 synthetic datasets for left/right output of LLM’s - Stretched every hour + posture excises and rice bucket training - Watched rekraps newest video - Watched Martin Shkreli stream - Scrolled X Song: Memory limitations in artificial intelligence - Infinity Frequencies

Our Single-rollout Asynchronous Optimization (SAO), is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench.

RL fine-tuning is now live for @nvidiaai Nemotron 3 on Fireworks, starting with Nemotron 3 Super (LoRA). Train with GRPO and serve the model in one place. We price by GPU-hour, not per token, so long multi-turn rollouts don't blow up your bill. Training shapes →