Horace He(@cHHillee):If you calculate out comms the dispatch takes time input_dim*comm_time_per_byte and the matmul takes time input_dim*intermediate_dim*time_per_flop. If you divide these, you find that the ratio that determines whether you can overlap compute with comms is comm_time_per_byte/(intermediate_dim*time_per_flop). Since latent_moe shrinks input_dim by 2 but keeps intermediate_dim the same, there's no impact on whether you're network bound.

Horace He

@cHHillee

@thinkymachines Formerly @PyTorch "My learning style is Horace twitter threads" - @typedfemale

参加 February 2010

612 フォロー中 53.4K ファン

Horace He@cHHillee

2026.07.17 03:58

If you calculate out comms the dispatch takes time input_dim*comm_time_per_byte and the matmul takes time input_dim*intermediate_dim*time_per_flop. If you divide these, you find that the ratio that determines whether you can overlap compute with comms is comm_time_per_byte/(intermediate_dim*time_per_flop). Since latent_moe shrinks input_dim by 2 but keeps intermediate_dim the same, there's no impact on whether you're network bound.

151

コミュニティへ転送

人気のあるユーザー