Scaling matrix optimizers to wide networks is expensive. Our new research post introduces Echelon, a rank-anchored formulation of Muon that orthogonalizes only the top subspace in O(mnr) time while rescaling the tail: up to 10.5× faster optimizer steps, with Muon's loss at 1.3B.
Blog:
We pair the algorithm with the first Lean-checked convergence proof for spectral steepest descent.
Lean proofs: