Register and share your invite link to earn from video plays and referrals.

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture ๆŠ•็จฟใฏๅ€‹ไบบใฎๆ„่ฆ‹ใงใ™ใ€‚
Joined May 2026
287 Following    476 Followers
What if you could expand the number of expert combinations in an MoE by about 18x without adding parameters? Title: MoRE: Mixture of Reused Experts URL: โ“ What is MoRE? An MoE where adjacent layers share one expert pool. With reuse factor R=2, two layers share a pool of 2N experts, and the physical parameter count matches a standard MoE. ๐Ÿ’ก Why does it help? With N=32 and K=4, the number of possible combinations rises from 35,960 to 635,376, about 18x. The same expert can even be used at consecutive layers. โ“ Is it hard to implement? It takes weight sharing plus one depth embedding per layer. Routers stay layer-specific, and no custom normalization or stabilization is needed, according to the paper. โ“ Does it work? At 1.15B, validation perplexity is 11.75 vs. 11.88 for DeepSeek, and the six-benchmark average rises from 47.05 to 48.12. With SonicMoE, throughput stays at about 96%. ๐Ÿ’ก Caveats? Tested only up to 1.15B and about 6.55B tokens. Behavior at 100B+ is unknown, and the reuse rate is only about twice random chance. #MoE# #LLM#
Show more