When you add annotations as rollouts in RL training, something counterintuitive happens — good policy actions start receiving negative advantages. OraRL names the problem and fixes it. Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs ❓ What goes wrong when you add oracle rollouts to GRPO training? 💡 "Advantage Inversion" occurs. Standard GRPO normalizes advantages across the rollout group using group mean and variance. Injecting a high-reward oracle rollout raises the group mean, which in turn causes policy rollouts that outperform the current average to receive negative advantages — a sign flip. Measuring 92,024 rollouts, naive oracle mixing inverted advantages in 42.5% of groups and 22.4% of rollouts. ❓ How does OraRL solve this? 💡 Five decoupled components. First, an oracle-free on-policy baseline is constructed without variance normalization (inversion is impossible by construction). A directional gain from the oracle-policy gap amplifies above-mean rollouts. The oracle advantage itself is calibrated against the strongest on-policy signal so it can't dominate the update. Sign-balanced advantage pruning retains equal numbers of positive and negative rollouts, compressing the residual flip rate to 0.3% and delivering a 1.48× speedup. ❓ How does it perform? 💡 Video-ORA-9B scores 73.1 on VSI-Bench, surpassing GPT-5 (55.0) and Gemini-3-Pro (55.1) by 18+ points. ReasonVOS segmentation improves +42.2 J&F and VideoHolmes gains +15.2 points over backbone. Training cost is just 2.2× SFT — under half the 4.9× overhead of GRPO with chain-of-thought. ❓ What about inference efficiency? 💡 No chain-of-thought is required, so P90 inference latency is 25.15 seconds vs 62.67 seconds for the CoT backbone. On data scaling, OraRL gains +5.2 points at 100k prompts versus +2.8 for GRPO, showing better sample efficiency throughout. #VideoMLLM# #ReinforcementLearning#