(1/12) New Paper 📄 Linear-time attentions are reaching frontier scale, their fixed-state memory, however, still saturates as context grows, because all of its capacity is exposed from token 1. Early tokens face no pressure to compress, so they take far more of that capacity than they need. Later tokens inherit a state that is already full. We introduce incremental memory activation: a fixed-size memory should not expose all of its capacity at once. It should schedule how much is live as the context grows, imposing a bottleneck early that forces the model to summarize rather than memorize, and unlocking fresh capacity later so new information has somewhere to land that isn't already written. Proteus is the simplest instantiation of that idea. We partition the memory into E blocks and unlock them one at a time as the context advances, gating both reads and writes so locked blocks are neither retrieved from nor updated. The state itself never grows; only the active fraction of it changes. It comes at no extra cost and improves average performance on language modeling and commonsense reasoning (+0.37–1.05 avg. accuracy points). More interestingly, it yields significant gains in long-context handling: up to +8.4 NIAH points at 2× the training context length. Read more background/method/results below 🧵

World Modeling with JEPA has recently gained traction thanks to a novel anti-collapse mechanism called "SIGReg" (by @ylecun and @randall_balestr). The math is clean, but it is rarely explained from first principles, so I broke it down in a detailed blog post 📰 Here is the summary (🧵): The problem: in JEPA, both sides of the prediction loss go through the same encoder. Map every input to a single constant point, the predictor matches it trivially, and the loss hits exactly zero. Perfect score, zero information. Gradient descent finds this solution by default. Prior work patched around it with stop-gradients, EMA teacher-student models, frozen pretrained encoders, and VICReg-style losses with 6+ hand-tuned coefficients. Each adds instability, hyperparameters, or a dependency on someone else's pretraining. --- SIGReg replaces all of that with one regularizer built on a single claim: force the batch of embeddings to look like samples from an isotropic Gaussian, N(0, I). LeJEPA proves this isn't arbitrary. It's the distribution that minimizes worst-case downstream risk across linear and nonlinear probes. But "make the embeddings Gaussian" is easy to say and hard to compute. You can't estimate a density in 200+ dimensions from a small batch. The fix is a chain of four classical results: compare Fourier transforms instead of densities, turn that comparison into a single scalar, approximate the integral with ~16 points, and lift the whole thing from 1D to any dimension with a theorem from 1936. Each step is simple. The stack is what makes it work. Read the subsequent posts in this thread to understand each step. --- But why it provably works: the only distribution for which SIGReg = 0 is N(0, I), which is full rank by construction, with every eigenvalue equal to 1. A collapsed low-rank encoder cannot be a minimum. It's not just unlikely to converge there—it's mathematically excluded. And the training loop stays boring: total loss = prediction + λ·SIGReg The encoder gets gradients from both terms; the predictor gets gradients from the prediction term only. No alternating updates, no stop-gradients, no two-timescale tricks. Now check out the breakdown below. 👇