What if you could expand the number of expert combinations in an MoE by about 18x without adding parameters?
Title: MoRE: Mixture of Reused Experts
URL:
โ What is MoRE?
An MoE where adjacent layers share one expert pool. With reuse factor R=2, two layers share a pool of 2N experts, and the physical parameter count matches a standard MoE.
๐ก Why does it help?
With N=32 and K=4, the number of possible combinations rises from 35,960 to 635,376, about 18x. The same expert can even be used at consecutive layers.
โ Is it hard to implement?
It takes weight sharing plus one depth embedding per layer. Routers stay layer-specific, and no custom normalization or stabilization is needed, according to the paper.
โ Does it work?
At 1.15B, validation perplexity is 11.75 vs. 11.88 for DeepSeek, and the six-benchmark average rises from 47.05 to 48.12. With SonicMoE, throughput stays at about 96%.
๐ก Caveats?
Tested only up to 1.15B and about 6.55B tokens. Behavior at 100B+ is unknown, and the reuse rate is only about twice random chance.
#
MoE# #
LLM#