2) A big part of what we want to do is realistic experiments so rollouts often hit a model's max context.
For example with Qwen 3.8 27B, it is a dense model and doing long-context RL on it requires quite abit of work.
At this context length, attention completely dominates your profile. You don't want to recompute attention twice but you also cant cache the whole attention layer or you just run out of memory even on a B300.
The solution we came up with is to only cache the FA4 output and LSE, recomputing everything else. This maximises memory usage (while still not OOMing) and speeds up step time by a non-trivial amount.