There's another magic trick (aka cool math) I believe this parallax enforced sparsity enables by the way... <- prune weights, adapt the pruned weights, sharply cuts catastrophic forgetting risk. If you have a model with the enforced sparsity, 50% of the weights are locked in to particular patterns. What if you mask the other expert weights leaving only those sparse contiguous weights to learn new data? That's the intuition, perhaps it works, perhaps not. But... if it does, it would make the base model significantly more malleable for new domains/tasks/whatever. SFT/continued pretraining with less risk of catastrophic forgetting and without lots of magic tricks and RL and re-sampling in your own words etc.