Register and share your invite link to earn from video plays and referrals.

Search results for ScalingLaws
ScalingLaws community
One keyword maps to one global community path.
Create community
People
Not Found
Tweets including ScalingLaws
Can MoE and looped Transformers share one scaling law? A new paper from Meta says yes. Title: Scaling Laws for Looped Mixture of Experts URL: 🔁 Overview It proposes Loop Scaling Laws, which jointly model weight-sharing recurrence (looping) and sparse expert routing (MoE). ❓ Problem Earlier laws treated the two axes separately and let the gain from looping grow without bound. In practice the gain from recurrence diminishes, and more experts let it keep paying off over more passes. 📐 Method Effective parameters are bounded: Neff(R) = N + κ₁·Nloop·(1 − e^(−(R−1)/κ₂)), with coefficients scaled by m^(−θ) for active ratio m. From compute and memory budgets, the law also gives the best recurrence count and expert count. 📊 Results ・Lowest error on unseen conditions such as R=16 (RMSE 0.0092 for dense models) ・At about 5×10²⁰ FLOPs, 0.3B MoE with E≥8 beats a 1.0B dense model ・At about 1.5×10²² FLOPs, a 0.3B-active/1.3B-total LoopMoE at R=4-5 gets BBH 31.5-31.8 and GSM8K 41.5-41.2, nearly matching a 0.6B-active/2.9B-total MoE (29.8 and 42.9) ・But overall score still trails, and inference compute is higher #ScalingLaws# #MoE#
Show more
📈 Bigger Recommender Transformers Do Not Automatically Scale Meta’s recent ads-ranking work has brought recommender scaling back into focus. Its central lesson is simple: predictable gains appear only when the model, user sequences, features, and serving system scale together. Zhihu contributor 九老师 explains why Transformer scaling is fundamentally a scale-matching problem, not a race to add more parameters. 1️⃣ Scaling changes the entire training system Increasing depth or width changes far more than model capacity. It also changes residual magnitudes, gradient flow, attention logits, optimizer states, and the best learning rate. A larger model can therefore appear to train normally while some layers contribute very little. The real question is not “How large is the Transformer?” but “Can every part of the system remain effective at this scale?” 2️⃣ Normalization determines whether depth is useful Post-Norm Transformers become increasingly fragile as depth grows. Gradient spikes can distort optimizer states, attention can collapse, and lower layers may gradually stop learning. Pre-Norm creates a cleaner gradient path and is generally more stable. But it has its own limitation: as residual signals accumulate, later layers may become dominated by the identity path. The network becomes deeper without learning proportionally richer representations. This is why residual scaling and initialization must evolve with model depth. 3️⃣ Attention and optimization must scale too Attention can fail in two opposite ways: 🔹 Entropy collapse: attention becomes extremely sharp and concentrates on very few positions. 🔹 Rank collapse: repeated mixing makes token representations increasingly similar. Techniques such as QK normalization can control attention sharpness, while residual scaling helps preserve token-specific information across layers. Optimizer settings also cannot be copied blindly from smaller models or LLM recipes. Recommendation data changes quickly, so historical gradients may become stale faster. Model scale, learning rate, initialization, and optimizer dynamics must be tuned as one system. 4️⃣ Loss alone cannot reveal silent failures A smooth training curve does not prove that the full Transformer is learning. Useful internal signals include: 🔹 Gradient strength across different layers 🔹 The size of parameter updates relative to parameter weights 🔹 Similarity between adjacent-layer representations 🔹 Attention-logit magnitude and attention entropy These diagnostics reveal whether lower layers are inactive, attention is collapsing, or updates have become poorly scaled. 5️⃣ Bigger models need richer inputs Scaling parameters alone may bring little improvement when the model still receives heavily compressed or low-complexity features. The author recalls that multimodal embeddings initially produced little gain in one recommender system. A manually designed distance feature worked better. After the model gained enough capacity, those raw embeddings became much more useful. The lesson is not that larger Transformers always win. It is that model capacity and input complexity must grow together. If the available information is simple, a smaller architecture may still be the better choice. ⚙ The core lesson A recommender Transformer is not a plug-and-play module. Features, sequences, tokenization, normalization, attention, optimization, training, and serving all interact. Scaling one component while freezing the others often creates cost without real capability. Using a Transformer is not the same as building a Transformer-native recommender system. A model truly scales only when the whole system scales with it. 🔗 Recent context: 🔗 Full Reading: #RecommenderSystems# #Transformers# #ScalingLaws# #MachineLearning# #AIInfra#
Show more
An AI agent's performance is governed not by how much it computes, but by how well that compute turns into good feedback 📈 Title: Scaling Laws for Agent Harnesses via Effective Feedback Compute URL: 📈 Overview This work proposes Effective Feedback Compute (EFC), a metric that reframes agent scaling efficiency around feedback quality rather than raw compute. It measures whether computation actually improved decisions. ❓ Challenges Solved We tend to reason about performance via raw metrics — tokens, tool calls, cost. But these mask whether feedback truly improved decision-making. Redundant, invalid, or unused feedback doesn't help. 💡 Methodology & Proposed Approach ・EFC credits feedback only when it is informative, valid, non-redundant, and retained for later decisions ・It normalizes by task demands for fair cross-task comparison ・Evaluated on synthetic tasks, code tasks, real traces, and prospective tests, vs raw-compute and SAS baselines 📊 Experimental Results EFC's explanatory power stood out (R² vs performance). ・Raw tokens/tool calls: R²=0.33-0.42 ・SAS baseline: 0.88, Oracle-EFC: 0.94, task-normalized: 0.99 ・Real traces: 0.92, prospective holdout: 0.85 ・Matched-budget interventions that improved feedback quality lifted success from 0.27 to 0.90 #AIAgents# #ScalingLaws#
Show more
Assuming scaling laws hold up, it seems like demand for AI would necessarily be infinite: Every model that comes out saturates the current security landscape with that level of protection and penetration. Then a new more capable model comes out and attackers begin using it, which necessitates that defenders must use it too. The cycle never ends. If this is the case then today's AI market caps are hilariously paltry compared to what they stand to be.
Show more
0
103
407
16
Forward to community
Abra: Scaling Diffusion Image Training A comprehensive scaling laws study of text-to-image diffusion models from Luma AI: "We present a systematic scaling law study for text-to-image diffusion models using Abra, a controlled family of flow-matching transformers trained across three orders of magnitude worth of compute (1019 to 1022 FLOPs), reaching significantly larger compute budgets than previous works. We demonstrate that diffusion models scale just as predictably as language models but require far more data to train optimally: compute optimality occurs at approximately 200 image tokens per parameter, ten times the Chinchilla compute-optimal prescription for LLMs." paper link:
Show more
Deconstructing Scaling Laws: The Triad of Optimization, Architecture, and Data
Transcript from a recent @scaling_laws podcast w/ the Oregon's Chief AI Officer. State officials are trying to learn more about independent eval practices at the labs... and are getting stonewalled. Blog posts / X threads are not substitutes for robust info sharing.
Show more
Generation of infohazards is subject to scaling laws btw
The bear case (other than scaling laws failing) for Ant + OAI was previously that Google would weaponize gross margins and commoditizes the frontier. The bear case going forward is that republicans stay firmly in control past 2028 and Elon leverages a compute advantage to spin faster cycles on his RSI and unlike the Google org has the risk tolerance to make broadly available a frontier model that the labs are too scared to. We are approaching the point where risk tolerance is very very important. Suspect “safety & alignment” is going to take up a much larger portion of opex for the labs vs what people might think. Think of the five worst people you have ever met. Now imagine if you gave each of them ten thousand employees and free rein. Terrifying. (Example stolen from latest moonshots pod)
Show more
At Marin we’ve found that scaling laws let us simulate the entire training trajectory. This became clear in our last 67B MoE run. The run hit the loss target within 1%, but perhaps more interestingly, arbitrary points during training could also be predicted to within 1%. This model is finishing up long context and will soon enter RL. Based on this finding, we’ve used a scaling ladder to simulate the training trajectory of the 535B MoE we kicked off yesterday. Will see how it goes. Now if someone can figure out how to fit a scaling law to each parameter we can just predict our final model and skip this training business. Something else I've learned from this process is that getting a training run off the ground is way more than just fiddling with the architecture. Lot of herculean efforts from teammates to get trillions of tokens curated, get hardware running on a new GPU stack, improve MFU, and propose great ideas that were tested and integrated. Given that this is our largest run to date, I anticipate lots of learning and adapting along the way. But thats part of the fun. Details:
Show more