We're open-sourcing FlashKDA — our high-performance CUTLASS-based implementation of Kimi Delta Attention kernels. Achieves 1.72×–2.22× prefill speedup over the flash-linear-attention baseline on H20, and works as a drop-in backend for flash-linear-attention. Explore on github:

We've open-sourced FlashKDA, our high-performance CUTLASS-based implementation of Kimi Delta Attention kernels. It delivers 1.72×–2.22× prefill speedup over the flash-linear-attention baseline on H20, and works as a drop-in backend for flash-linear-attention. Explore on GitHub:

Gated DeltaNet-2 (GDN-2) is now a lot faster on NVIDIA GPUs 🚀 Thanks to an amazing effort by our cuDNN team, we now have full cuDNN support for GDN-2. This covers not just prefill (forward), but also a highly optimized backward pass built on CUTLASS primitives, available now as part of the CUTLASS 4.7.0 release. On GB300 (BF16, batch 4, 64 heads, d=128), compared to the FLA Triton implementation: ⚡ Forward: up to 6.4x faster ⚡ Backward: up to 2.8x faster ⚡ End-to-end: ~3x faster training Throughput stays flat across sequence lengths from 2K to 32K, meaning the kernels are compute-bound where they should be. Benchmarks below. 👇 If you are interested to know more about GDN-2, see our manuscript below: