wh(@nrehiew_):1 intuition I have is that Delta Net is kind of a soft sliding window since the decay forces earlier tokens to have less signal (unlike RoPE) In their ablations, they show that their approach beats SWA (soft sliding &gt; hard sliding) and Full Attention (i assume flops matched)

@nrehiew_

eng primarily, ml mostly, research previously

Joined October 2023

104 Following 18K Followers

wh@nrehiew_

2026.08.27 02:50

1 intuition I have is that Delta Net is kind of a soft sliding window since the decay forces earlier tokens to have less signal (unlike RoPE) In their ablations, they show that their approach beats SWA (soft sliding > hard sliding) and Full Attention (i assume flops matched)