Register and share your invite link to earn from video plays and referrals.

cv usk
@cv_usk
AI / Software Research Notes AI Agent, LLMOps, MLOps, Software Architecture 投稿は個人の意見です。
Joined May 2026
287 Following    473 Followers
Can MoE and looped Transformers share one scaling law? A new paper from Meta says yes. Title: Scaling Laws for Looped Mixture of Experts URL: 🔁 Overview It proposes Loop Scaling Laws, which jointly model weight-sharing recurrence (looping) and sparse expert routing (MoE). ❓ Problem Earlier laws treated the two axes separately and let the gain from looping grow without bound. In practice the gain from recurrence diminishes, and more experts let it keep paying off over more passes. 📐 Method Effective parameters are bounded: Neff(R) = N + κ₁·Nloop·(1 − e^(−(R−1)/κ₂)), with coefficients scaled by m^(−θ) for active ratio m. From compute and memory budgets, the law also gives the best recurrence count and expert count. 📊 Results ・Lowest error on unseen conditions such as R=16 (RMSE 0.0092 for dense models) ・At about 5×10²⁰ FLOPs, 0.3B MoE with E≥8 beats a 1.0B dense model ・At about 1.5×10²² FLOPs, a 0.3B-active/1.3B-total LoopMoE at R=4-5 gets BBH 31.5-31.8 and GSM8K 41.5-41.2, nearly matching a 0.6B-active/2.9B-total MoE (29.8 and 42.9) ・But overall score still trails, and inference compute is higher #ScalingLaws# #MoE#
Show more