Horace He
@cHHillee
가입 February 2010
633 팔로잉 중 55.4K 팬
As @johnschulman2 has put it, there are many ways we can "spend compute to save our customers compute". Here's one of them - a variant of muon that works far better for lora fine-tuning :)
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at
더 보기
