at decagon, we’ve been building around a simple idea: the failures of your current model checkpoint should shape the training data for the next one.
Conversational voice agents need to know when a different person is speaking, but not every background voice should count.
We combined speaker embeddings with a post-trained audio-language model to determine when a speaker change matters.
Show more
multimodal LLMs are great but they have some key limitations preventing them from accomplishing general audio tasks, including some as basic as speaker-change detection!
I've written a lot about how low latency is critical in voice agents, and it's quite a hard problem when you also need to keep accuracy super high.
Here's some insight into how our team approaches a portion of this!
Show more
Fast text serving ≠ fast speech.
Here's how we got our time to first audio under 30ms with nearly 10x more audio throughput.
multimodal inference feels pretty underexplored, especially when you move beyond standard transformer architectures. fun to share what we learned when scaling our text to speech model to 10x throughput
Show more
Fast text serving ≠ fast speech.
Here's how we got our time to first audio under 30ms with nearly 10x more audio throughput.
multimodal inference doesn't fit the mold of standard text autoregressive generation. text-to-speech models feature different architectures, different states, and different batch shapes. we rebuilt our tts serving around that and simultaneously reduced our time to first audio while improving throughput by several fold.
Show more
Fast text serving ≠ fast speech.
Here's how we got our time to first audio under 30ms with nearly 10x more audio throughput.
tts naturalness is hard. sft has its limits, but rl is tricky for flow-matching models since they lack the log-probs dpo and grpo rely on.
excited to share work on adapting both methods for TTS with @cyrusasg at @DecagonAI!
Show more
Modern TTS models can sound great — and still fail badly on pacing, pauses, and prosody.
We adapted DPO + GRPO to flow-matching models to tackle the tail end of TTS behavior:
Show more
Modern TTS models can sound great — and still fail badly on pacing, pauses, and prosody.
We adapted DPO + GRPO to flow-matching models to tackle the tail end of TTS behavior:
Show more
