This paper shows that knowledge transfer is hard even under perfect conditions and model access, yet it is randomly used as an argument against what i was saying about distillation. Actually it is a good read to understand why API distillation can't work, and you can just fine tune for specific style/patterns or very narrow domain knowledge *at best*.
顯示更多