SambaNova
@SambaNovaAI
SambaNova is the fastest AI inference platform — running AI securely, profitably, and at scale, with ROI received at a remarkable speed that matters⚡
Joined March 2018
165 Following 13K Followers
The slow part is writing the answer, not reading the prompt.
That's why agentic inference needs a different division of labor. GPUs for prefill, SN50 for decode.
Great breakdown of Sven's talk at the @tetrateio Inference Routing Summit!
Show more
I thought long context was what made agents slow. At @tetrateio's Inference Routing Summit, Sven Mawson of @SambaNovaAI said agent sessions hit cache 90-95% of the time, so 80-95% of the time goes to decode.
The slow part is writing the answer, not reading the prompt.
Show more
