Horace He
@cHHillee
Joined February 2010
633 Following 55.4K Followers
As @johnschulman2 has put it, there are many ways we can "spend compute to save our customers compute". Here's one of them - a variant of muon that works far better for lora fine-tuning :)
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at
Show more
