Training a model to predict the next concept, not just the next token, made 8.9B-scale pretraining converge 1.95x faster. NCP-ArchPreview: Moving towards Latent Space Language Models through Next Concept Prediction The model learns discrete concepts spanning multiple tokens via vector quantization, and jointly trains token-level and concept-level prediction in one latent-space architecture. 🚀 Highlight 1: Strikingly faster convergence Trained on the exact same 5.73T tokens as OLMo-3-7B, it matches the baseline's final loss after consuming only 51.3% of the tokens, while beating it by 2.45 points on the downstream macro-average. 🧩 Highlight 2: Concept prediction genuinely drives the gains Ablations that add the Concept Module, hierarchical residual connections, and the NCP loss one at a time each independently improve the loss, and the advantage holds even under parameter- and compute-matched comparisons. ⚡ Highlight 3: The learned concept space stays useful after pretraining Domain adaptation that updates only 17M parameters gains more capability with less forgetting than full fine-tuning, and injecting concept states into a speculative drafter improves accepted length by 4.17%. Treating concepts as a first-class training target, rather than a side effect, looks like a genuinely practical blueprint for next-generation model design. #LLM# #LanguageModels#