🧠 A transformer can always attend back to the past, so it has no structural incentive to compress history into a compact latent state. That means it tends to hoard information instead of summarizing it — and that hurts generalization. The fix proposed here is to add a light auxiliary objective to next-token prediction: also predict your own next latent. By learning to guess its next hidden state given the next token, the model grows a compact internal state with consistent transition rules — theoretically, a "belief state" that's a sufficient statistic for predicting the future. The transformer stays parallel while a lightweight dynamics model enforces temporal consistency — essentially co-training a transformer and an RNN in parallel. And the payoff is concrete. On a Manhattan-taxi world model the latent rank is ~3× more compact than GPT's; on reasoning and planning it looks ahead correctly instead of taking shortcuts (Countdown 54.8% vs ~39%). On a 1.3B language model it preserves quality while variable-length self-speculative decoding makes inference 3.3× faster (MTP/JTP cap out around 1.7×). Best of all, you only add a small MLP at training time — inference stays a single transformer. Next-Latent Prediction Transformers Learn Compact World Models #Transformer# #WorldModels#