Assaf Ben Kish
@abk_tau
Deep Learning | Large Language Models | Reinforcement Learning
199 Following 298 Followers
(1/7) Humans can acquire many capabilities without (catastrophically) forgetting previous ones. Why can't neural networks do the same?
Local Support Learning (LSL) takes us a step closer towards this goal.
In LLMs of up to 7B parameters, LSL augments gradient-based training to retain prior capabilities while learning new tasks at full capacity. Notably, it does so without access to prior data.
Key idea: casting forgetting as a geometric problem, and optimizing for the worst case.
Paper:
Website and code:
Show more
