As an AI engineer, please learn:
- Learn the roofline model and why decode is memory-bound
- Deploy vLLM and SGLang, then read their schedulers
- Understand paged attention from the code, not the blog post
- Build observability before you optimize anything
- Track TTFT, inter-token latency, throughput, queue depth
- Use Grafana + Prometheus for inference dashboards
- Turn on prefix caching and find which workloads it helps
- Learn continuous batching and chunked prefill
- Run load tests with 1000+ concurrent requests
- Report p50, p95, p99, never just the mean
- Master quantization tradeoffs (FP8, INT4, AWQ, GPTQ)
- Learn speculative decoding and where it stops helping
- Set up KV cache eviction for long contexts
- Try disaggregated prefill and decode serving
- Learn Kubernetes for AI workloads and autoscale on queue depth
- Learn how inference costs break unit economics
- Build your own model router by cost, latency, quality
- Create a token budgeting system per request
- Build one inference service and benchmark it publicly
- Read inference research instead of model release news
- Start sharing your optimization benchmarks
I put together a 10-week plan that covers every one of these at 30 minutes a day. It is 50 sessions, split between reading the theory and building on your own service, and all of them feed one artifact: an inference service you deploy, instrument, load test past 1000 concurrent requests, tune, and publish as a reproducible benchmark.
It is open on GitHub and contributions are welcome, especially newer sources worth adding. I am working through it myself and will share more content on this going forward, so stay tuned.
GitHub repo:
(don't forget to star ๐)