注册并分享邀请链接,可获得视频播放与邀请奖励。

wh
@nrehiew_
eng primarily, ml mostly, research previously
加入 October 2023
108 正在关注    19K 粉丝
2) A big part of what we want to do is realistic experiments so rollouts often hit a model's max context. For example with Qwen 3.8 27B, it is a dense model and doing long-context RL on it requires quite abit of work. At this context length, attention completely dominates your profile. You don't want to recompute attention twice but you also cant cache the whole attention layer or you just run out of memory even on a B300. The solution we came up with is to only cache the FA4 output and LSE, recomputing everything else. This maximises memory usage (while still not OOMing) and speeds up step time by a non-trivial amount.
显示更多