alphaXiv(@askalphaxiv):What if you have transformers learn to think in both latent states & tokens? This Microsoft paper adds a self-supervised next-latent prediction objective to standard next-token training, where a lightweight dynamics model learns to predict the transformer’s next hidden state from the current hidden state and next token. This pushes transformer representations toward compact belief states with coherent transition dynamics, improving world modeling, reasoning, planning, and enabling variable-length self-speculative decoding with up to 3.3x inference speedup.

2026.06.23 17:18

What if you have transformers learn to think in both latent states & tokens? This Microsoft paper adds a self-supervised next-latent prediction objective to standard next-token training, where a lightweight dynamics model learns to predict the transformer’s next hidden state from the current hidden state and next token. This pushes transformer representations toward compact belief states with coherent transition dynamics, improving world modeling, reasoning, planning, and enabling variable-length self-speculative decoding with up to 3.3x inference speedup.