Niels Rogge(@NielsRogge):The new Kimi K3 leverages LatentMoE, a technique developed by @nvidia in January What is LatentMoE? In LatentMoE, tokens are projected from the model hidden dimension 𝑑 into a smaller latent dimension ℓ for expert routing and computation, which reduces routed parameter loads and all-to-all traffic by a factor of 𝑑/ℓ. Learn more about it here:

Niels Rogge

@NielsRogge

ML Engineer @huggingface. Building @KU_Leuven grad. General interest in machine & deep learning. Making AI more accessible for everyone!

가입 April 2010

739 팔로잉 중 21.3K 팬

Niels Rogge@NielsRogge

2026.07.17 09:01

The new Kimi K3 leverages LatentMoE, a technique developed by @nvidia in January What is LatentMoE? In LatentMoE, tokens are projected from the model hidden dimension 𝑑 into a smaller latent dimension ℓ for expert routing and computation, which reduces routed parameter loads and all-to-all traffic by a factor of 𝑑/ℓ. Learn more about it here: