註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Josh Tobin
@josh_tobin_
automating scientific discovery @recursive_si
加入 July 2011
1.2K 正在關注    14.6K 粉絲
A fun small win for automated research: We found and helped fix some edge cases that could affect inference performance in vLLM and SGLang. In our last blog post, we described a reward hacking judge we developed for performance optimization tasks. While applying the judge to some new autoresearch work, it discovered that some FlashInfer (a library underpinning vLLM and SGLang) kernels used a hard coded value of -50,000 as a masked-attention sentinel -- even though valid QK values can be smaller. Corner cases like this that are numerically wrong but silent can cause a huge amount of headache to find and fix, e.g., the historical debate around flash attention ( Great use case for AI.
顯示更多
0
12
404
43
轉發到社區