註冊並分享邀請連結,可獲得影片播放與邀請獎勵。

Horace He
@cHHillee
@thinkymachines Formerly @PyTorch "My learning style is Horace twitter threads" - @typedfemale
加入 February 2010
633 正在關注    55.4K 粉絲
As @johnschulman2 has put it, there are many ways we can "spend compute to save our customers compute". Here's one of them - a variant of muon that works far better for lora fine-tuning :)
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback! See our optimizers guide at
顯示更多
0
6
322
19
轉發到社區