Cyrus(@cyrusasg):multimodal inference doesn't fit the mold of standard text autoregressive generation. text-to-speech models feature different architectures, different states, and different batch shapes. we rebuilt our tts serving around that and simultaneously reduced our time to first audio while improving throughput by several fold.

Cyrus

@cyrusasg

research lead @decagonai prev @harvard post-training, inference, mlsys

加入 March 2022

255 正在關注 552 粉絲

Cyrus@cyrusasg

2026.08.20 18:11

multimodal inference doesn't fit the mold of standard text autoregressive generation. text-to-speech models feature different architectures, different states, and different batch shapes. we rebuilt our tts serving around that and simultaneously reduced our time to first audio while improving throughput by several fold.