Max Ryabinin(@m_ryabinin):RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem. We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches

Max Ryabinin

@m_ryabinin

Large-scale deep learning & research @togethercompute Learning@home/Hivemind author (DeDLOC, SWARM, Petals)

加入 October 2020

246 正在關注 2.6K 粉絲

Max Ryabinin@m_ryabinin

2026.09.18 17:17

RL with LLMs is very unstable when training and sampling policies differ. Standard fixes (matching numerics, importance sampling) work around the problem. We find the root cause of this instability from first principles and propose a way to directly cancel it. Score Centering is competitive and compatible with existing approaches — while simple to implement! 🧵 [1/6]