📈 Bigger Recommender Transformers Do Not Automatically Scale Meta’s recent ads-ranking work has brought recommender scaling back into focus. Its central lesson is simple: predictable gains appear only when the model, user sequences, features, and serving system scale together. Zhihu contributor 九老师 explains why Transformer scaling is fundamentally a scale-matching problem, not a race to add more parameters. 1️⃣ Scaling changes the entire training system Increasing depth or width changes far more than model capacity. It also changes residual magnitudes, gradient flow, attention logits, optimizer states, and the best learning rate. A larger model can therefore appear to train normally while some layers contribute very little. The real question is not “How large is the Transformer?” but “Can every part of the system remain effective at this scale?” 2️⃣ Normalization determines whether depth is useful Post-Norm Transformers become increasingly fragile as depth grows. Gradient spikes can distort optimizer states, attention can collapse, and lower layers may gradually stop learning. Pre-Norm creates a cleaner gradient path and is generally more stable. But it has its own limitation: as residual signals accumulate, later layers may become dominated by the identity path. The network becomes deeper without learning proportionally richer representations. This is why residual scaling and initialization must evolve with model depth. 3️⃣ Attention and optimization must scale too Attention can fail in two opposite ways: 🔹 Entropy collapse: attention becomes extremely sharp and concentrates on very few positions. 🔹 Rank collapse: repeated mixing makes token representations increasingly similar. Techniques such as QK normalization can control attention sharpness, while residual scaling helps preserve token-specific information across layers. Optimizer settings also cannot be copied blindly from smaller models or LLM recipes. Recommendation data changes quickly, so historical gradients may become stale faster. Model scale, learning rate, initialization, and optimizer dynamics must be tuned as one system. 4️⃣ Loss alone cannot reveal silent failures A smooth training curve does not prove that the full Transformer is learning. Useful internal signals include: 🔹 Gradient strength across different layers 🔹 The size of parameter updates relative to parameter weights 🔹 Similarity between adjacent-layer representations 🔹 Attention-logit magnitude and attention entropy These diagnostics reveal whether lower layers are inactive, attention is collapsing, or updates have become poorly scaled. 5️⃣ Bigger models need richer inputs Scaling parameters alone may bring little improvement when the model still receives heavily compressed or low-complexity features. The author recalls that multimodal embeddings initially produced little gain in one recommender system. A manually designed distance feature worked better. After the model gained enough capacity, those raw embeddings became much more useful. The lesson is not that larger Transformers always win. It is that model capacity and input complexity must grow together. If the available information is simple, a smaller architecture may still be the better choice. ⚙ The core lesson A recommender Transformer is not a plug-and-play module. Features, sequences, tokenization, normalization, attention, optimization, training, and serving all interact. Scaling one component while freezing the others often creates cost without real capability. Using a Transformer is not the same as building a Transformer-native recommender system. A model truly scales only when the whole system scales with it. 🔗 Recent context: 🔗 Full Reading: #RecommenderSystems# #Transformers# #ScalingLaws# #MachineLearning# #AIInfra#

Build and train an LLM "from scratch" yourself and you truly understand what's happening inside 🛠️ A complete educational implementation that runs on a single GPU. Title: FareedKhan-dev/train-llm-from-scratch URL: 🛠️ Overview An educational repository that implements a Transformer from scratch in PyTorch, based on "Attention is All You Need." It promises you can train your own million- to billion-parameter LLM on a single GPU. ❓ Challenges Solved LLMs are ubiquitous, but hands-on chances to train one from scratch and understand its internals are rare. ・Just using off-the-shelf frameworks leaves the Transformer's mechanics opaque ・Learners needed an end-to-end resource spanning pretraining through post-training alignment 💡 Content & Structure It covers the entire LLM lifecycle. ・Data acquisition and preprocessing (from The Pile) ・Core Transformer architecture (embeddings, attention, feed-forward networks) ・Model training (with DDP for distributed processing) ・Post-training alignment: SFT, reward modeling, PPO, DPO, GRPO ・Text generation and inference Code is organized into src/models, scripts, data_loader, configs, and a Streamlit ui. The stack is PyTorch, tiktoken, HDF5, and NumPy. 🌍 Use Cases / Audience For developers and researchers who want hands-on understanding of LLM training — from those with limited GPUs (starting at 13M parameters) to those targeting multi-billion-parameter models on enterprise hardware. #LLM# #MachineLearning#

🛠 MLOps projects often run on ad hoc instinct. This study analyzes 103 practitioner blogs and whitepapers and distills 25 architecturally significant guidelines for ML model integration and deployment. Title: Architecturally Significant MLOps Guidelines for ML Model Integration and Deployment: a Gray Literature Review URL: 📝 Overview Using a gray literature review (analyzing practitioner web sources like blogs, whitepapers, and vendor docs rather than peer-reviewed papers), this paper organizes architectural guidance for integrating and deploying ML models. ❓ Challenges Solved Even as MLOps adoption grows, there's little consolidation of architectural knowledge as reusable design decisions, so teams default to ad hoc approaches and struggle to transfer experience across projects. 💡 Methodology & Proposed Approach ・33 Google queries returned 331 sources, narrowed by criteria to 103 for analysis ・Two researchers independently extracted text, resolving disagreements in consensus meetings ・Three authors synthesized practices into guidelines and used card sorting to form five categories ・The themes are CI/CD and automation, deployment strategies and environments, design and integration strategies, model serving and inference, and ML component management 🎯 Use Cases It serves as a consolidated reference for architectural decisions in ML integration and deployment, and as building blocks toward a holistic MLOps reference architecture. 📊 Results ・It extracted 25 architecturally significant guidelines, with 72% (18) mentioned four or more times, indicating practitioner consensus ・The most-cited was containerization (27 sources), followed by establishing CI/CD pipelines (53 mentions) ・Deployment had 16 guidelines versus 9 for integration, exposing an under-documented gap on the integration side #MLOps# #MachineLearning#