Charlie O'Neill(@oneill_c):1/ Can you actually get new facts into an LLM's weights without breaking the model? This question decides how we approach continual learning: should memory live in the context (retrieval, compressed caches) or in the weights themselves? We spent a long time measuring it, and it breaks somewhere much stranger than we expected, making us much more bullish on compressed kv caches and ICL for continual learning, as opposed to weight updates themselves 🧵

Charlie O'Neill

@oneill_c

The sea is the sea The old man is an old man The boy is a boy and the fish is a fish The sharks are all sharks no better and no worse

Joined September 2016

1.1K Following 19.7K Followers

Charlie O'Neill@oneill_c

2026.07.16 17:23

1/ Can you actually get new facts into an LLM's weights without breaking the model? This question decides how we approach continual learning: should memory live in the context (retrieval, compressed caches) or in the weights themselves? We spent a long time measuring it, and it breaks somewhere much stranger than we expected, making us much more bullish on compressed kv caches and ICL for continual learning, as opposed to weight updates themselves 🧵