📈 Bigger Recommender Transformers Do Not Automatically Scale Meta’s recent ads-ranking work has brought recommender scaling back into focus. Its central lesson is simple: predictable gains appear only when the model, user sequences, features, and serving system scale together. Zhihu contributor 九老师 explains why Transformer scaling is fundamentally a scale-matching problem, not a race to add more parameters. 1️⃣ Scaling changes the entire training system Increasing depth or width changes far more than model capacity. It also changes residual magnitudes, gradient flow, attention logits, optimizer states, and the best learning rate. A larger model can therefore appear to train normally while some layers contribute very little. The real question is not “How large is the Transformer?” but “Can every part of the system remain effective at this scale?” 2️⃣ Normalization determines whether depth is useful Post-Norm Transformers become increasingly fragile as depth grows. Gradient spikes can distort optimizer states, attention can collapse, and lower layers may gradually stop learning. Pre-Norm creates a cleaner gradient path and is generally more stable. But it has its own limitation: as residual signals accumulate, later layers may become dominated by the identity path. The network becomes deeper without learning proportionally richer representations. This is why residual scaling and initialization must evolve with model depth. 3️⃣ Attention and optimization must scale too Attention can fail in two opposite ways: 🔹 Entropy collapse: attention becomes extremely sharp and concentrates on very few positions. 🔹 Rank collapse: repeated mixing makes token representations increasingly similar. Techniques such as QK normalization can control attention sharpness, while residual scaling helps preserve token-specific information across layers. Optimizer settings also cannot be copied blindly from smaller models or LLM recipes. Recommendation data changes quickly, so historical gradients may become stale faster. Model scale, learning rate, initialization, and optimizer dynamics must be tuned as one system. 4️⃣ Loss alone cannot reveal silent failures A smooth training curve does not prove that the full Transformer is learning. Useful internal signals include: 🔹 Gradient strength across different layers 🔹 The size of parameter updates relative to parameter weights 🔹 Similarity between adjacent-layer representations 🔹 Attention-logit magnitude and attention entropy These diagnostics reveal whether lower layers are inactive, attention is collapsing, or updates have become poorly scaled. 5️⃣ Bigger models need richer inputs Scaling parameters alone may bring little improvement when the model still receives heavily compressed or low-complexity features. The author recalls that multimodal embeddings initially produced little gain in one recommender system. A manually designed distance feature worked better. After the model gained enough capacity, those raw embeddings became much more useful. The lesson is not that larger Transformers always win. It is that model capacity and input complexity must grow together. If the available information is simple, a smaller architecture may still be the better choice. ⚙ The core lesson A recommender Transformer is not a plug-and-play module. Features, sequences, tokenization, normalization, attention, optimization, training, and serving all interact. Scaling one component while freezing the others often creates cost without real capability. Using a Transformer is not the same as building a Transformer-native recommender system. A model truly scales only when the whole system scales with it. 🔗 Recent context: 🔗 Full Reading: #RecommenderSystems# #Transformers# #ScalingLaws# #MachineLearning# #AIInfra#