jianlin.su(@Jianlin_S):MoE (8): Enforcing Sequence-Level Balance https://t.co/kjUITDlox7 This article explores how to achieve sequence-level load balancing without incurring any loss penalty. Starting from the original Quantile Balancing (QB), we gradually derive a new method called Moving Quantile Balancing (MQB), which successfully accomplishes this objective. Nevertheless, whether sequence-level balance is truly necessary and to what extent it should be enforced remain open questions.

jianlin.su

@Jianlin_S

Grad&Clip&EM is all you need @Kimi_Moonshot Blog: , Cool Papers: Interesting link:

参加 February 2025

21 フォロー中 11.6K ファン

jianlin.su@Jianlin_S

2026.05.22 07:07

MoE (8): Enforcing Sequence-Level Balance This article explores how to achieve sequence-level load balancing without incurring any loss penalty. Starting from the original Quantile Balancing (QB), we gradually derive a new method called Moving Quantile Balancing (MQB), which successfully accomplishes this objective. Nevertheless, whether sequence-level balance is truly necessary and to what extent it should be enforced remain open questions.

106

コミュニティへ転送

人気のあるユーザー