wh
@nrehiew_
eng primarily, ml mostly, research previously
가입 October 2023
104 팔로잉 중 18K 팬
1 intuition I have is that Delta Net is kind of a soft sliding window since the decay forces earlier tokens to have less signal (unlike RoPE)
In their ablations, they show that their approach beats SWA (soft sliding > hard sliding) and Full Attention (i assume flops matched)
더 보기
