註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

wh
@nrehiew_
eng primarily, ml mostly, research previously
加入 October 2023
108 正在關注    19K 粉絲
We also had to do a bunch of tricks to keep training efficiency high: 1) You can't do weight sync the standard way in such a setup. So we do sharded HTTPs, which maximises the egress bandwidth while keeping as little overhead as possible. It turns out that there isn't much point doing delta weight sync for LoRA since the overhead of decoding the weights is not worth it given LoRA is already small in size Also, you could actually end up being CPU bound here on weight sync. If your inference replicas are busy, you dont want to add more non prefill/decode workload dealing with weight sync. Especially the case with LoRAs where the whole benefit is that you can still continue inference on rollouts from past adapters easily
顯示更多