jianlin.su
@Jianlin_S
Grad&Clip&EM is all you need @Kimi_Moonshot
Blog: , Cool Papers:
Interesting link:
加入 February 2025
21 正在关注 11.6K 粉丝
MoE (8): Enforcing Sequence-Level Balance
This article explores how to achieve sequence-level load balancing without incurring any loss penalty. Starting from the original Quantile Balancing (QB), we gradually derive a new method called Moving Quantile Balancing (MQB), which successfully accomplishes this objective.
Nevertheless, whether sequence-level balance is truly necessary and to what extent it should be enforced remain open questions.
显示更多
