注册并分享邀请链接,可获得视频播放与邀请奖励。

Jon Durbin
@jon_durbin
Human. Backend dev
加入 December 2012
146 正在关注    7.4K 粉丝
There's another magic trick (aka cool math) I believe this parallax enforced sparsity enables by the way... <- prune weights, adapt the pruned weights, sharply cuts catastrophic forgetting risk. If you have a model with the enforced sparsity, 50% of the weights are locked in to particular patterns. What if you mask the other expert weights leaving only those sparse contiguous weights to learn new data? That's the intuition, perhaps it works, perhaps not. But... if it does, it would make the base model significantly more malleable for new domains/tasks/whatever. SFT/continued pretraining with less risk of catastrophic forgetting and without lots of magic tricks and RL and re-sampling in your own words etc.
显示更多