注册并分享邀请链接,可获得视频播放与邀请奖励。

Horace He
@cHHillee
@thinkymachines Formerly @PyTorch "My learning style is Horace twitter threads" - @typedfemale
加入 February 2010
633 正在关注    55.4K 粉丝
Your model behaves differently during training and inference? Boy do I have a solution for you :^)
Training the model to detect when it is in training vs. deployment works too! The model can even generalize from vLLM deployment training to SGLang deployment. We do this by recording activations from layer 34 in Hugging Face Trainer's training step and in vLLM's compiled serving. Then, we swap in the attention and residual stream from the Hugging Face copy and SFT the model to predict "Yes," and to predict "No" when we swap in from vLLM instead. The model learns how its training vs. deployment environment affects its first 34 layers.
显示更多
0
24
499
21
转发到社区