TL;DR: Cortex bakes the gains an LLM gets from context into its weights, so they persist after the context is removed. After 50 sequential factual edits it recovers 88.9% of the gain while keeping 99.8% of original performance.
Title: Cortex: Consolidating Context into Parametric Memory
URL:
Key points
・🧠 Inspired by hippocampus-to-neocortex consolidation; unifies knowledge editing, skill learning, behavior steering, and suppression as one problem
・🔮 The model generates queries anticipating how the context will be used and learns from them
・🧩 Updates only the FFN: low-rank updates (Synaptic) or new added channels (Structural), both mergeable into the weights
・👩🏫 A frozen teacher that sees the context distills its outputs and hidden states into a student that doesn't
・📊 Tested on 4B-120B dense and MoE models; batch editing hits 80-91% CR vs. 36-46% for the strongest baselines
・🤖 On ALFWorld, the consolidated model even beats supplying the context directly
・⚠️ Knowledge suppression still harms neighboring knowledge
💡 A practical path beyond RAG and prompting: folding experience into the weights.
#
LLM# #
ContinualLearning#