we brought PPO back just for one reason: get way more out of each rollout
GRPO is hard to scale, you can't just pay for as many rollouts as you want on every task. for math/coding an extra rollout is a couple more compiles and checks, whatever. for RSI it's a training run that costs thousands of dollars. and in bio we literally don't have multiple identical embryos to run in parallel
we did some early stuff on keeping PPO training stable, hopefully some of it's useful to people