7 MoE routing strategies in production, clearly explained:
(bookmark this)
Sending every token to its highest-scoring experts can make an MoE model slower.
The problem is that many tokens can prefer the same experts. Those experts become overloaded while others remain underused.
With fixed capacity, some token assignments may be dropped. Across GPUs, uneven workloads and communication overhead can reduce the compute savings.
MoE routing has to select useful experts while distributing work efficiently.
The seven strategies in this visual handle that tradeoff differently.
> Token-choice routing
Each token selects its top-k experts using router scores. The expert count per token stays fixed, but expert loads can vary. Load-balancing mechanisms help prevent traffic from concentrating on a few experts.
> Expert-choice routing
Each expert selects its highest-scoring tokens within a fixed capacity. Expert workloads stay balanced, while individual tokens can receive different amounts of expert computation.
> Dynamic routing
The number of selected experts varies across tokens. Confidence-based variants allocate more experts when routing scores are less decisive, although router confidence doesn't directly measure token difficulty.
> Balanced assignment
A global matching procedure assigns tokens under expert load constraints. Some tokens may miss their highest-scoring expert to achieve a more balanced allocation. Computing that assignment adds overhead.
> Hash-based routing
A deterministic hash maps tokens to experts without a learned gate. This removes router learning, but expert selection has less flexibility to adapt to context.
> Soft MoE routing
Experts process weighted mixtures of token representations through slots. Each slot can combine information from multiple tokens, changing what an expert processes.
> Group-limited routing
The router restricts eligible expert groups before selecting experts. When groups reflect hardware placement, this can reduce communication across devices or nodes.
These choices affect how serving engines batch expert computation and distribute work across GPUs.
Evenly distributing tokens doesn't guarantee low latency. Expert batches may still be small, and moving tokens between devices takes time.
To dive deeper into MoE engineering, I covered everything in an article recently, including expert parallelism, grouped GEMM, topology-aware placement, memory requirements, and inference optimizations.
It follows what happens after the router selects experts and explains the tradeoffs that determine latency and throughput.
Read it below.