Something similar to this is the invention of QB: It is not that easy to understand the math part, but if you ignore the math derivation, the solution is extremely simple (with a litte bit infra challenge as well to find the exact top m*k/n score). Compared to auxfree bias, I also felt QB is good for optimization and scaling: you do not need to worry about the lr schedule for the bias update, and you do not need to worry about the bias RMS v.s. your gate score RMS, so your model never collapses to a static routing MoE at alll (which is bad as your topk is sort of dominated by the bias?) However, my personal taste is that QB is not as important as attention residual. Actually I am surprised that why auxfree bias is not working for K3's MoE balance...although a little bit more complicated, but I personally found auxfree surprisingly work very well for most cases