註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Horace He
@cHHillee
@thinkymachines Formerly @PyTorch "My learning style is Horace twitter threads" - @typedfemale
加入 February 2010
633 正在關注    55.4K 粉絲
Your model behaves differently during training and inference? Boy do I have a solution for you :^)
Training the model to detect when it is in training vs. deployment works too! The model can even generalize from vLLM deployment training to SGLang deployment. We do this by recording activations from layer 34 in Hugging Face Trainer's training step and in vLLM's compiled serving. Then, we swap in the attention and residual stream from the Hugging Face copy and SFT the model to predict "Yes," and to predict "No" when we swap in from vLLM instead. The model learns how its training vs. deployment environment affects its first 34 layers.
顯示更多
0
24
499
21
轉發到社區