jianlin.su(@Jianlin_S):MoE (8): Enforcing Sequence-Level Balance https://t.co/kjUITDlox7 This article explores how to achieve sequence-level load balancing without incurring any loss penalty. Starting from the original Quantile Balancing (QB), we gradually derive a new method called Moving Quantile Balancing (MQB), which successfully accomplishes this objective. Nevertheless, whether sequence-level balance is truly necessary and to what extent it should be enforced remain open questions.

jianlin.su

@Jianlin_S

Grad&Clip&EM is all you need @Kimi_Moonshot Blog: , Cool Papers: Interesting link:

加入 February 2025

21 正在關注 11.6K 粉絲

jianlin.su@Jianlin_S

2026.05.22 07:07

MoE (8): Enforcing Sequence-Level Balance This article explores how to achieve sequence-level load balancing without incurring any loss penalty. Starting from the original Quantile Balancing (QB), we gradually derive a new method called Moving Quantile Balancing (MQB), which successfully accomplishes this objective. Nevertheless, whether sequence-level balance is truly necessary and to what extent it should be enforced remain open questions.

106

轉發到社區

熱門用戶