every research team needs to spend some time learning how their modeling code lowers down to gpu kernels example: rms norm vs layernorm industry assumes rms norm is cheaper. you don't need the x̂, it looks simpler, so it must be faster, so it got adopted but that's not true. both rms and layernorm are memory bound kernels (the amount of time it takes to get the data to gpu cores is longer than the amount of time it takes to do the computation on said cores) both take the same amount of time e2e so can probably train the model right with either, but maybe it makes a difference (e.g why diffusion models still use norms with an affine shift e.g adaLN) i get why. as open-source models get better and, inevitably, commoditized across the inference providers, overall serving speed determines user experience determines which model gets adopted but there's no reason to superstitiously avoid free things