Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included. A causal encoder builds global KV memory. For each new token, the decoder combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations. With a 48-layer decoder, the computation path after t tokens runs through 48t decoder blocks, while each token still executes a fixed number of blocks. Depth grows with the sequence and per-token cost stays the same. The same state transition is used for pretraining, SFT, sampling and RL replay, and nothing resets at the prompt-response boundary. RL replay rebuilds states under the current weights instead of reusing stale rollout states. The report is a design proposal. The author states that reasoning gains, hardware speedups and RL scaling are goals that have not been measured yet. Paper: Chat with Paper:

“Recurrent Looped Transformer” This paper makes the decoder recurrent across every prompt and response token, while a causal encoder provides reusable global KV memory. Longer sequences then create deeper latent computation paths without adding more physical layers, while keeping the same state transition across pretraining, inference, and RL replay.