This latest work from Base Labs looks into whether off-policy distillation (through SFT) from a smarter teacher actually improves post-RL performance, and by how much. Some highlights: - Distillation helps small models a lot more than larger ones. They even find that for 8B models, RL alone beats SFT + RL on their tasks. - Distillation helps more in tasks that require conventions or understanding not already in the base model. - SFT reduces model entropy. For a base model that is already a good RL target to begin with, this is bad because it reduces exploration. So when is it worth doing pre-RL SFT? In my view: - If the base model does a really poor job on the target domain to begin with. This can be because of poor harness use, bad syntax, missing knowledge, etc. - If there's some taste or judgement calls you want your model to pick up from a teacher.