Language models process each token through a single residual stream (blue arrows): reading / writing memory and applying MLPs at each layer, and finally decoding the next token.
Can we scale up this single stream computation?
Introducing Latent Lattice Language Models (LLLMs): augmenting this vertical stream with a horizontal axis of computation—the Lattice of Latents (orange arrows).
RNN 🤝 Transformers 🧵