Everyday learn new things, thanks to Bernstein Global Memory: An AI memory primer TLDR: We need more CPU, GPU and memory But HBM is very import for both training and storing the model for inferencing Different workloads and hence different memory requirements: Training: Compute intensive and highly dependent on HBM bandwidth and capacity. Inferencing Prefill: Compute-bound and primarily drives TTFT (Time to First Token). Inference Decode: memory-bandwidth-bound and dominated by KV Cache access. Decode performance is often measured by TPOT (Time Per Output Token). RAG DRAM-centric because large vector indexes are commonly kept in system memory Agentic Inference demand through multiple tool calls, iterative reasoning, and repeated context injection, which can increase KV Cache usage and memory pressure and use a lot of CPU!!!