Ali Hatamizadeh
@ahatamiz1
LLM Tech Lead & Staff Research Scientist @NVIDIA
Co-creator of Gated DeltaNet & Gated DeltaNet-2
๊ฐ์
June 2015
180 ํ๋ก์ ์ค 4.1K ํฌ
GDN-2 scales nicely with Muon ๐
We trained hybrid MoE models at three scales (2B-7B) with Muon.
GDN-2 matches Mamba2's loss with ~10% less training compute!
