We beat out @nvidia's KV cache transfer for KL divergence, and we're using it to build the fastest inference @TryTrustAI at @ycombinator.
17.8% less TTFT, $641 less per 1M requests, and 82.5% held-out top-1 agreement on Qwen 32B, prefilled from Qwen 8B.
Show more
@TryTrustAI is building the fastest, cheapest lightweight inference by optimizing KV cache at @ycombinator.
Our group of MIT grads and olympiad medalists from @GoogleDeepMind, @JaneStreetGroup, and NeurIPS/ICML publishers are setting a new standard for cost and latency efficiency.
Show more
