注册并分享邀请链接,可获得视频播放与邀请奖励。

wh
@nrehiew_
eng primarily, ml mostly, research previously
加入 October 2023
108 正在关注    19K 粉丝
We also had to do a bunch of tricks to keep training efficiency high: 1) You can't do weight sync the standard way in such a setup. So we do sharded HTTPs, which maximises the egress bandwidth while keeping as little overhead as possible. It turns out that there isn't much point doing delta weight sync for LoRA since the overhead of decoding the weights is not worth it given LoRA is already small in size Also, you could actually end up being CPU bound here on weight sync. If your inference replicas are busy, you dont want to add more non prefill/decode workload dealing with weight sync. Especially the case with LoRAs where the whole benefit is that you can still continue inference on rollouts from past adapters easily
显示更多