注册并分享邀请链接,可获得视频播放与邀请奖励。

Mariya I. Vasileva
@mariyaivasileva
Research Scientist. RSI, multi-agent safety & alignment, evaluation. ex @Meta MSL, @Amazon AGI. 🎓 UIUC (PhD), @Caltech • @WiMLWorkshop, @CVFADworkshop • 🇧🇬
加入 February 2017
2.4K 正在关注    21.6K 粉丝
I don’t often write up my thoughts, but here’s a new blog post summarizing some of my recent musings on RSI and verification: Recursive Self-Inflation. A lot of it is informed by my own research experience and by grappling with the fact that the strongest recent RSI results depend, and quite reasonably so, on fast, automatic feedback: we generate candidates, evaluate them, retain the ones that work, and keep iterating. The problem, however, is that every practical verifier has holes in it, and sustained optimization will eventually find and exploit them. That matters a bit differently in a recursive loop than in a static eval. In a static eval, a verifier mistake just gives you the wrong score. In an RSI loop, an accepted mistake becomes the seed for the next round, so errors can compound over time while the curve still looks like the model is progressing. I recently went through evidence of verifier failure across coding and science benchmarks, reward hacking, model judges, and formal proofs, and tried to characterize what I think credible RSI results would need to report, and what they need to rule out. As always, feedback welcome!
显示更多
0
6
164
15
转发到社区