Inference needs a different approach to compute. On The Startup Project, d-Matrix CEO @sidsheth discusses memory-centric compute, low-latency inference, heterogeneous infrastructure, and the path from first silicon to rack-scale systems. Listen here: #AIInference# #dMatrix#

Inference is changing, and so are the metrics we use to measure it. In a new interview with @letsdatascience, d-Matrix CTO Sudeep Bhoja shares how to evaluate next-gen inference platforms through real-world performance, efficiency, and the metrics that matter most. The conversation also explores Raptor and the role of heterogeneous infrastructure. Read more: #AIInference# #InfiniteInference#

inference companies are taking on a lot of risk right now they're signing longer contracts for compute because they never want to have customers and run out of compute at the same time. it'd be a big problem if users are growing, but they have to pause the product it's hard to get compute fairly right now, so this is all they can do. inference companies are in a particularly tough spot because they work on a pay-as-you-go model, where customers pay per token and can leave whenever. clearly, this is a problem when you buy gpus on fixed contracts, because you'll pay for them even if customers don't use them inference companies are simply stuck with no other choice. changing the way compute is bought will also make inference better!

Inference providers are hitting the funding markets. Fal has spoken to investors about raising at $15B, while Fireworks has considered raising at $30b. Both would nearly double their valuations from rounds earlier this year. w/ @validapau @aaronpholmes

Inference becomes 70 to 90% of all AI compute by 2030, and KV cache is its single biggest memory consumer. $MU has solutions for all levels of KV cache to enable Contextual Intelligence: HBM > DDR5 > disaggregated DRAM > NVMe > data lakes. Every tier is theirs.

Inference is taking the TAM, everyone knows that Five years of ASICs eat NVIDIA, and GPUs are still the majority of boxes. Only a handful of companies can even ship an ASIC. Google, AWS, Meta, Microsoft, and OpenAI. So, everyone else is still using GPUs. And even if you have the design, almost nobody can build it. TSMC, Intel, and Samsung. That’s it. $INTC