Can MoE and looped Transformers share one scaling law? A new paper from Meta says yes.
Title: Scaling Laws for Looped Mixture of Experts
URL:
🔁 Overview
It proposes Loop Scaling Laws, which jointly model weight-sharing recurrence (looping) and sparse expert routing (MoE).
❓ Problem
Earlier laws treated the two axes separately and let the gain from looping grow without bound. In practice the gain from recurrence diminishes, and more experts let it keep paying off over more passes.
📐 Method
Effective parameters are bounded: Neff(R) = N + κ₁·Nloop·(1 − e^(−(R−1)/κ₂)), with coefficients scaled by m^(−θ) for active ratio m. From compute and memory budgets, the law also gives the best recurrence count and expert count.
📊 Results
・Lowest error on unseen conditions such as R=16 (RMSE 0.0092 for dense models)
・At about 5×10²⁰ FLOPs, 0.3B MoE with E≥8 beats a 1.0B dense model
・At about 1.5×10²² FLOPs, a 0.3B-active/1.3B-total LoopMoE at R=4-5 gets BBH 31.5-31.8 and GSM8K 41.5-41.2, nearly matching a 0.6B-active/2.9B-total MoE (29.8 and 42.9)
・But overall score still trails, and inference compute is higher
#
ScalingLaws# #
MoE#