註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture 投稿は個人の意見です。
加入 May 2026
287 正在關注    473 粉絲
Can MoE and looped Transformers share one scaling law? A new paper from Meta says yes. Title: Scaling Laws for Looped Mixture of Experts URL: 🔁 Overview It proposes Loop Scaling Laws, which jointly model weight-sharing recurrence (looping) and sparse expert routing (MoE). ❓ Problem Earlier laws treated the two axes separately and let the gain from looping grow without bound. In practice the gain from recurrence diminishes, and more experts let it keep paying off over more passes. 📐 Method Effective parameters are bounded: Neff(R) = N + κ₁·Nloop·(1 − e^(−(R−1)/κ₂)), with coefficients scaled by m^(−θ) for active ratio m. From compute and memory budgets, the law also gives the best recurrence count and expert count. 📊 Results ・Lowest error on unseen conditions such as R=16 (RMSE 0.0092 for dense models) ・At about 5×10²⁰ FLOPs, 0.3B MoE with E≥8 beats a 1.0B dense model ・At about 1.5×10²² FLOPs, a 0.3B-active/1.3B-total LoopMoE at R=4-5 gets BBH 31.5-31.8 and GSM8K 41.5-41.2, nearly matching a 0.6B-active/2.9B-total MoE (29.8 and 42.9) ・But overall score still trails, and inference compute is higher #ScalingLaws# #MoE#
顯示更多