Charlie O'Neill(@oneill_c):13/ What I take away from all this is that all the training to create the model is engineered around crafting the best possible ICL mechanism. Further training degrades this mechanism, at least for knowledge acquisition, and maybe continual learning should just focus on loading the right information up for icl (ie into the context window) So this provides an answer to the architecture question, at least for now: the channel worth engineering for memory is the one that comes with addresses ie the context. Paper: https://t.co/ePR9dUljHv Code and data to follow shortly Done at @baseten.

Charlie O'Neill

@oneill_c

The sea is the sea The old man is an old man The boy is a boy and the fish is a fish The sharks are all sharks no better and no worse

参加 September 2016

1.1K フォロー中 22.4K ファン

Charlie O'Neill@oneill_c

2026.07.16 17:24

13/ What I take away from all this is that all the training to create the model is engineered around crafting the best possible ICL mechanism. Further training degrades this mechanism, at least for knowledge acquisition, and maybe continual learning should just focus on loading the right information up for icl (ie into the context window) So this provides an answer to the architecture question, at least for now: the channel worth engineering for memory is the one that comes with addresses ie the context. Paper: Code and data to follow shortly Done at @baseten.