Register and share your invite link to earn from video plays and referrals.

wh
@nrehiew_
eng primarily, ml mostly, research previously
Joined October 2023
108 Following    19K Followers
We also had to do a bunch of tricks to keep training efficiency high: 1) You can't do weight sync the standard way in such a setup. So we do sharded HTTPs, which maximises the egress bandwidth while keeping as little overhead as possible. It turns out that there isn't much point doing delta weight sync for LoRA since the overhead of decoding the weights is not worth it given LoRA is already small in size Also, you could actually end up being CPU bound here on weight sync. If your inference replicas are busy, you dont want to add more non prefill/decode workload dealing with weight sync. Especially the case with LoRAs where the whole benefit is that you can still continue inference on rollouts from past adapters easily
Show more