註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture 投稿は個人の意見です。
加入 May 2026
287 正在關注    476 粉絲
What if you could expand the number of expert combinations in an MoE by about 18x without adding parameters? Title: MoRE: Mixture of Reused Experts URL: ❓ What is MoRE? An MoE where adjacent layers share one expert pool. With reuse factor R=2, two layers share a pool of 2N experts, and the physical parameter count matches a standard MoE. 💡 Why does it help? With N=32 and K=4, the number of possible combinations rises from 35,960 to 635,376, about 18x. The same expert can even be used at consecutive layers. ❓ Is it hard to implement? It takes weight sharing plus one depth embedding per layer. Routers stay layer-specific, and no custom normalization or stabilization is needed, according to the paper. ❓ Does it work? At 1.15B, validation perplexity is 11.75 vs. 11.88 for DeepSeek, and the six-benchmark average rises from 47.05 to 48.12. With SonicMoE, throughput stays at about 96%. 💡 Caveats? Tested only up to 1.15B and about 6.55B tokens. Behavior at 100B+ is unknown, and the reuse rate is only about twice random chance. #MoE# #LLM#
顯示更多