Liam Fedus(@LiamFedus):Equal token counts don’t mean equal work. Our scheduler uses estimated compute cost to balance work across DP workers and PP microbatches. Within a sequence, later tokens attend to more preceding tokens, making them more expensive to process. Giving each GPU a mix of early and late chunks helps balance that work. This integrates nicely with the rest of our RL stack, including MoE routing replay and the vision encoder.

2026.09.18 04:53

Equal token counts don’t mean equal work. Our scheduler uses estimated compute cost to balance work across DP workers and PP microbatches. Within a sequence, later tokens attend to more preceding tokens, making them more expensive to process. Giving each GPU a mix of early and late chunks helps balance that work. This integrates nicely with the rest of our RL stack, including MoE routing replay and the vision encoder.

显示更多

Zijie Yan@zijie_y

2026.09.17 16:07

Neon’s scientific traces are long and vary widely in length. Training on them shouldn’t mean GPUs chewing through padding or waiting for each other. Here’s how we pack sequences, balance the work, and distribute long contexts across GPUs at @periodiclabs 👇

显示更多