Horace He(@cHHillee):If you calculate out comms the dispatch takes time input_dim*comm_time_per_byte and the matmul takes time input_dim*intermediate_dim*time_per_flop. If you divide these, you find that the ratio that determines whether you can overlap compute with comms is comm_time_per_byte/(intermediate_dim*time_per_flop). Since latent_moe shrinks input_dim by 2 but keeps intermediate_dim the same, there's no impact on whether you're network bound.

Horace He

@cHHillee

@thinkymachines Formerly @PyTorch "My learning style is Horace twitter threads" - @typedfemale

加入 February 2010

612 正在關注 53.4K 粉絲

Horace He@cHHillee

2026.07.17 03:58

If you calculate out comms the dispatch takes time input_dim*comm_time_per_byte and the matmul takes time input_dim*intermediate_dim*time_per_flop. If you divide these, you find that the ratio that determines whether you can overlap compute with comms is comm_time_per_byte/(intermediate_dim*time_per_flop). Since latent_moe shrinks input_dim by 2 but keeps intermediate_dim the same, there's no impact on whether you're network bound.

151

轉發到社區

熱門用戶