๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture ๆŠ•็จฟใฏๅ€‹ไบบใฎๆ„่ฆ‹ใงใ™ใ€‚
๊ฐ€์ž… May 2026
287 ํŒ”๋กœ์ž‰ ์ค‘    476 ํŒฌ
What if you could expand the number of expert combinations in an MoE by about 18x without adding parameters? Title: MoRE: Mixture of Reused Experts URL: โ“ What is MoRE? An MoE where adjacent layers share one expert pool. With reuse factor R=2, two layers share a pool of 2N experts, and the physical parameter count matches a standard MoE. ๐Ÿ’ก Why does it help? With N=32 and K=4, the number of possible combinations rises from 35,960 to 635,376, about 18x. The same expert can even be used at consecutive layers. โ“ Is it hard to implement? It takes weight sharing plus one depth embedding per layer. Routers stay layer-specific, and no custom normalization or stabilization is needed, according to the paper. โ“ Does it work? At 1.15B, validation perplexity is 11.75 vs. 11.88 for DeepSeek, and the six-benchmark average rises from 47.05 to 48.12. With SonicMoE, throughput stays at about 96%. ๐Ÿ’ก Caveats? Tested only up to 1.15B and about 6.55B tokens. Behavior at 100B+ is unknown, and the reuse rate is only about twice random chance. #MoE# #LLM#
๋” ๋ณด๊ธฐ