📈 Bigger Recommender Transformers Do Not Automatically Scale Meta’s recent ads-ranking work has brought recommender scaling back into focus. Its central lesson is simple: predictable gains appear only when the model, user sequences, features, and serving system scale together. Zhihu contributor 九老师 explains why Transformer scaling is fundamentally a scale-matching problem, not a race to add more parameters. 1️⃣ Scaling changes the entire training system Increasing depth or width changes far more than model capacity. It also changes residual magnitudes, gradient flow, attention logits, optimizer states, and the best learning rate. A larger model can therefore appear to train normally while some layers contribute very little. The real question is not “How large is the Transformer?” but “Can every part of the system remain effective at this scale?” 2️⃣ Normalization determines whether depth is useful Post-Norm Transformers become increasingly fragile as depth grows. Gradient spikes can distort optimizer states, attention can collapse, and lower layers may gradually stop learning. Pre-Norm creates a cleaner gradient path and is generally more stable. But it has its own limitation: as residual signals accumulate, later layers may become dominated by the identity path. The network becomes deeper without learning proportionally richer representations. This is why residual scaling and initialization must evolve with model depth. 3️⃣ Attention and optimization must scale too Attention can fail in two opposite ways: 🔹 Entropy collapse: attention becomes extremely sharp and concentrates on very few positions. 🔹 Rank collapse: repeated mixing makes token representations increasingly similar. Techniques such as QK normalization can control attention sharpness, while residual scaling helps preserve token-specific information across layers. Optimizer settings also cannot be copied blindly from smaller models or LLM recipes. Recommendation data changes quickly, so historical gradients may become stale faster. Model scale, learning rate, initialization, and optimizer dynamics must be tuned as one system. 4️⃣ Loss alone cannot reveal silent failures A smooth training curve does not prove that the full Transformer is learning. Useful internal signals include: 🔹 Gradient strength across different layers 🔹 The size of parameter updates relative to parameter weights 🔹 Similarity between adjacent-layer representations 🔹 Attention-logit magnitude and attention entropy These diagnostics reveal whether lower layers are inactive, attention is collapsing, or updates have become poorly scaled. 5️⃣ Bigger models need richer inputs Scaling parameters alone may bring little improvement when the model still receives heavily compressed or low-complexity features. The author recalls that multimodal embeddings initially produced little gain in one recommender system. A manually designed distance feature worked better. After the model gained enough capacity, those raw embeddings became much more useful. The lesson is not that larger Transformers always win. It is that model capacity and input complexity must grow together. If the available information is simple, a smaller architecture may still be the better choice. ⚙ The core lesson A recommender Transformer is not a plug-and-play module. Features, sequences, tokenization, normalization, attention, optimization, training, and serving all interact. Scaling one component while freezing the others often creates cost without real capability. Using a Transformer is not the same as building a Transformer-native recommender system. A model truly scales only when the whole system scales with it. 🔗 Recent context: 🔗 Full Reading: #RecommenderSystems# #Transformers# #ScalingLaws# #MachineLearning# #AIInfra#

An AI agent's performance is governed not by how much it computes, but by how well that compute turns into good feedback 📈 Title: Scaling Laws for Agent Harnesses via Effective Feedback Compute URL: 📈 Overview This work proposes Effective Feedback Compute (EFC), a metric that reframes agent scaling efficiency around feedback quality rather than raw compute. It measures whether computation actually improved decisions. ❓ Challenges Solved We tend to reason about performance via raw metrics — tokens, tool calls, cost. But these mask whether feedback truly improved decision-making. Redundant, invalid, or unused feedback doesn't help. 💡 Methodology & Proposed Approach ・EFC credits feedback only when it is informative, valid, non-redundant, and retained for later decisions ・It normalizes by task demands for fair cross-task comparison ・Evaluated on synthetic tasks, code tasks, real traces, and prospective tests, vs raw-compute and SAS baselines 📊 Experimental Results EFC's explanatory power stood out (R² vs performance). ・Raw tokens/tool calls: R²=0.33-0.42 ・SAS baseline: 0.88, Oracle-EFC: 0.94, task-normalized: 0.99 ・Real traces: 0.92, prospective holdout: 0.85 ・Matched-budget interventions that improved feedback quality lifted success from 0.27 to 0.90 #AIAgents# #ScalingLaws#

🦾 Why Data, Not Models, Is the Real Moat in Embodied AI The timing of this question is hard to miss. This week, the World Humanoid Robot Games released a 2,500+ hour dataset covering 12 scenario categories, 44 operations, and more than 10,000 tasks. Crucially, it also includes failures and edge cases. But raw hours tell only part of the story. Zhihu contributor 于超, an assistant professor at Tsinghua Shenzhen International Graduate School, shares his team’s view on why data has become embodied AI’s hardest-to-replicate advantage. 1️⃣ Robot scaling is fundamentally asymmetric Like language models, robot policies appear to benefit from more data, larger models, and greater compute. But these three inputs do not scale equally. Model architectures can be studied and reproduced quickly. General-purpose compute can, in principle, be purchased. High-quality robot data is different. It must be accumulated through physical interaction, and competitors cannot recreate it overnight. That asymmetry is what turns data into a moat. 2️⃣ Robot data must be manufactured LLMs inherited decades of internet data. Robots did not. Every useful trajectory must be produced through physical interaction. Even a simple cup-grasping task changes with the object, lighting, environment, camera angle, and robot body. So raw hours are not enough. What matters is the diversity of embodiments, tasks, objects, failures, and recoveries. Open X-Embodiment needed more than 20 institutions and 22 robot platforms to collect over one million trajectories. DROID used 50 collectors for a year, producing only 350 hours of data. New methods such as UMI and egocentric recording make collection easier. But every hour still requires real people, equipment, and time. 3️⃣ A successful trajectory can still be bad data Robot data quality is more complicated than whether a task was completed. On the hardware side, camera accuracy, encoder readings, force sensors, calibration, communication latency, and synchronization across modalities can all corrupt a trajectory. The human operator adds another source of noise. Teleoperating a robot is not the same as performing the action directly. Operators hesitate, pause, readjust, and develop habits for compensating for the control system. A task may succeed even when parts of the demonstration should never be imitated. Success is therefore only the coarsest possible label. One trajectory can contain both excellent behavior and inefficient or misleading actions. Training on the entire trajectory without distinction effectively tells the robot to learn both. 4️⃣ The next challenge is information density Collecting more trajectories is only half the problem. Teams must also identify which parts are worth learning from. Yu Chao’s team developed STEAM to detect local progress within a trajectory without frame-by-frame annotation or manually designed rewards. It separates useful progress from hesitation, failure, and recovery. The key question is shifting from “How many trajectories do we have?” to “How much useful information does each trajectory contain?” 5️⃣ Embodied data is physically expensive Text can be copied. Videos can be downloaded. Robot data requires a physical production process. Collecting one hour may involve a robot, sensors, teleoperation equipment, an operator, a suitable environment, task materials, and engineers who maintain and calibrate the system. Real factories, stores, and homes add even more complexity. And pressing the record button is only the beginning. Transmission, cleaning, governance, and storage can cost more than collection itself. The author offers a rough calculation. If a company wants one million hours of real-world data and reduces the combined collection and management cost to RMB 200 per hour, the total still reaches RMB 200 million. 🔑 The real moat compounds over time Quantity, quality, and cost explain why embodied AI data cannot be replicated through a short burst of spending. Large, diverse, high-quality datasets require physical infrastructure, operational discipline, and years of accumulation. As robot policies continue to benefit from scaling, the durable advantage will belong to teams that can repeatedly: 🔹 Collect broader real-world experience 🔹 Identify the most informative behavior 🔹 Preserve failures and recovery signals 🔹 Turn noisy trajectories into useful learning data In embodied AI, having data and knowing how to use it are becoming two very different capabilities. And the second may ultimately matter even more than the first. 🔗 Full analysis: #EmbodiedAI# #Robotics# #PhysicalAI# #RobotLearning# #AIData# #ScalingLaw# #Tsinghua#

Abra: Scaling Diffusion Image Training A comprehensive scaling laws study of text-to-image diffusion models from Luma AI: "We present a systematic scaling law study for text-to-image diffusion models using Abra, a controlled family of flow-matching transformers trained across three orders of magnitude worth of compute (1019 to 1022 FLOPs), reaching significantly larger compute budgets than previous works. We demonstrate that diffusion models scale just as predictably as language models but require far more data to train optimally: compute optimality occurs at approximately 200 image tokens per parameter, ten times the Chinchilla compute-optimal prescription for LLMs." paper link:

Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

To ensure the scaling law hits exponential returns after all the pain, it’s critical to maximize useful bits per token per second on a small compute and memory budget before moving into a grand palace.

HUAWEI's Tau (τ) Scaling Law is a new principle for guiding the future development of semiconductors. By 2031, HUAWEI's high-end chips are expected to feature a transistor density equivalent to 14 Å (1.4 nm) processes. Watch the livestream to learn more!

At Marin we’ve found that scaling laws let us simulate the entire training trajectory. This became clear in our last 67B MoE run. The run hit the loss target within 1%, but perhaps more interestingly, arbitrary points during training could also be predicted to within 1%. This model is finishing up long context and will soon enter RL. Based on this finding, we’ve used a scaling ladder to simulate the training trajectory of the 535B MoE we kicked off yesterday. Will see how it goes. Now if someone can figure out how to fit a scaling law to each parameter we can just predict our final model and skip this training business. Something else I've learned from this process is that getting a training run off the ground is way more than just fiddling with the architecture. Lot of herculean efforts from teammates to get trillions of tokens curated, get hardware running on a new GPU stack, improve MFU, and propose great ideas that were tested and integrated. Given that this is our largest run to date, I anticipate lots of learning and adapting along the way. But thats part of the fun. Details:

NVIDIA CEO Jensen Huang says one scaling law multiplies AI faster than NVIDIA can hire engineers. Most people know three AI scaling laws. Pre-training. Post-training. Test-time. Each one multiplies intelligence by throwing more compute at a different stage. Jensen Huang says there's a fourth and it's the one that will dominate... Agentic scaling law. "During test time, that agentic system goes off and does research, bangs on databases, uses tools," Huang says. "And one of the most important things it does is spawn off a whole bunch of sub-agents." That's the multiplier. One AI worker can become a team. Then a department. Then a company. "It's so much easier to scale NVIDIA by hiring more employees than it is to scale myself," Huang says. Now imagine scaling without a payroll constraint. "The agentic scaling law — it's kind of like multiplying AI," Huang says. "We could spin off agents as fast as you want to spin off agents." Each agent spins off sub-agents. Each sub-agent spins off more. The compute requirement compounds inside a single query. And every agent generates new data, new experiences, new edge cases. "Wow, this is really good. We ought to memorize this," Huang says. "That data set comes back to pre-training." The four scaling laws don't compete. They feed each other. Agentic systems produce data, which feeds pre-training, which smartens the base model, which enables better agents, which produce more data. A flywheel that compounds forever. The companies pricing in three scaling laws are mispricing the fourth. The fourth eats the other three for lunch. P.S. Pull the thread on any story like this and you'll find the hidden incentive at the other end. As Munger said: "Show me the incentive and I'll show you the outcome." So I wrote a short book on how to spot them and design your own. Comment "INCENTIVES" and I'll send you the details. If you're new here, follow @GeniusGTX for content on the greatest minds in economics, psychology, and history. — Jensen Huang ( @nvidia ), NVIDIA CEO, on Lex Fridman's ( @lexfridman ) podcast