🎬 Distilled autoregressive video models are fast but tend to drift from human preferences. Astrolabe answers that challenge by doing RL alignment in the forward process, with no re-distillation and no reverse-process unrolling. Title: Astrolabe: Steering Forward-Process Reinforcement Learning URL: 📝 Overview Astrolabe is a reinforcement learning framework that aligns distilled autoregressive (AR) video models with human visual preferences. Its defining feature is doing RL in the forward process rather than via conventional reverse-process optimization. It is a large 53-page, 37-figure study. ❓ Challenges Solved Distilled AR video models suit efficient streaming generation but tend to misalign with human preferences. Worse, existing RL doesn't fit these architectures naturally: it typically needs either expensive re-distillation or solver-coupled reverse-process optimization, both heavy and hard to scale. 💡 Methodology & Proposed Approach It rests on three innovations. ・Negative-aware fine-tuning contrasts positive and negative samples at inference endpoints to establish an implicit policy-improvement direction without unrolling the reverse process ・A streaming training scheme generates sequences progressively via a rolling KV-cache, applying RL updates only to local clip windows while keeping long-range coherence through prior-context conditioning ・A multi-reward objective integrates uncertainty-aware selective regularization and dynamic reference updates to mitigate reward hacking, the collapse where only the apparent score rises 🎯 Use Cases It fits real-time streaming video generation where you want to align an efficient distilled model with preferences while preserving its speed. It applies across multiple distilled AR video models and raises quality without sacrificing inference efficiency. 📊 Significance and Results ・By avoiding the heavy paths of re-distillation and reverse-process unrolling, it addresses computational efficiency bottlenecks ・Combining forward-process negative awareness, streaming updates, and reward-hacking mitigation, it provides a robust, scalable alignment solution ・It demonstrates effectiveness across several distilled AR models, with detailed quantitative evaluation and ablations #VideoGeneration# #ReinforcementLearning#