Charlie O'Neill(@oneill_c):1/ Can you actually get new facts into an LLM's weights without breaking the model? This question decides how we approach continual learning: should memory live in the context (retrieval, compressed caches) or in the weights themselves? We spent a long time measuring it, and it breaks somewhere much stranger than we expected, making us much more bullish on compressed kv caches and ICL for continual learning, as opposed to weight updates themselves 🧵

Charlie O'Neill

@oneill_c

The sea is the sea The old man is an old man The boy is a boy and the fish is a fish The sharks are all sharks no better and no worse

가입 September 2016

1.1K 팔로잉 중 19.7K 팬

Charlie O'Neill@oneill_c

2026.07.16 17:23

1/ Can you actually get new facts into an LLM's weights without breaking the model? This question decides how we approach continual learning: should memory live in the context (retrieval, compressed caches) or in the weights themselves? We spent a long time measuring it, and it breaks somewhere much stranger than we expected, making us much more bullish on compressed kv caches and ICL for continual learning, as opposed to weight updates themselves 🧵