If there's one thing I've learned from AI, it's that whenever Google releases a new architecture paper, it's probably worth paying attention to. Published yesterday - Proteus is a new neural memory mechanism that can be applied to various SOTA models like SWLA, Comba, Titans, and Hope-Attention for better memory and long context. The high-level motivation is that, while recurrent-style architectures are not capped by explicit context windows like attention is, they still suffer from frontloading memory into the first few tokens that arrive in the context. Proteus instead progressively unlocks more memory for the model only as the context increases, therefore more uniformly storing information. in eli5 terms: you don't want every detail from the first five minutes of an experience consuming the same mental capacity as everything that happens afterward. You compress what came before and preserve room for what comes next. Was a fun read, and excited to see the implications of Proteus + Google's Nested Learning, for continual learning. Authors: @reza_byt, @behrouz_ali, @mirrokni, @AaronCourville