註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

wh
@nrehiew_
eng primarily, ml mostly, research previously
加入 October 2023
108 正在關注    19K 粉絲
2) A big part of what we want to do is realistic experiments so rollouts often hit a model's max context. For example with Qwen 3.8 27B, it is a dense model and doing long-context RL on it requires quite abit of work. At this context length, attention completely dominates your profile. You don't want to recompute attention twice but you also cant cache the whole attention layer or you just run out of memory even on a B300. The solution we came up with is to only cache the FA4 output and LSE, recomputing everything else. This maximises memory usage (while still not OOMing) and speeds up step time by a non-trivial amount.
顯示更多