The new Kimi K3 leverages LatentMoE, a technique developed by
@nvidia in January
What is LatentMoE? In LatentMoE, tokens are projected from the model hidden dimension 𝑑 into a smaller latent dimension ℓ for expert routing and computation, which reduces routed parameter loads and all-to-all traffic by a factor of 𝑑/ℓ.
Learn more about it here: