wh(@nrehiew_):They also use Muon throughout the network - Router AdamW for stability - GR projections AdamW (suspected due to their weird shape) - ngram table Adam without weight decay - per head orthogornalization They trained the model with TP so for the parameter update, they come up with a parameter assigner to balance flops across ranks. The ranks then exchange via all to all

@nrehiew_

eng primarily, ml mostly, research previously

Joined October 2023

104 Following 18K Followers

wh@nrehiew_

2026.08.27 02:50

They also use Muon throughout the network - Router AdamW for stability - GR projections AdamW (suspected due to their weird shape) - ngram table Adam without weight decay - per head orthogornalization They trained the model with TP so for the parameter update, they come up with a parameter assigner to balance flops across ranks. The ranks then exchange via all to all