As an AI Infrastructure Engineer, you must build these projects.
1.) High-Throughput Serving Engine Tuning
Build: Deploy vLLM or SGLang, tuning chunked prefill, continuous batching and max-num-seqs for a specific GPU architecture.
Why: Default configs waste 60% of VRAM. Squeezing the hardware is the baseline job.
2.) KV-Cache Prefix Routing Proxy
Build: A load balancer that hashes incoming system prompts and routes requests to the specific replica holding that KV-cache block in memory.
Why: Reused context is free speed. This alone can slash Time-To-First-Token (TTFT) by 80%.
3.) Custom Fused Triton Kernel
Build: Write a fused RMSNorm or RoPE (Rotary Position Embedding) kernel in Triton and benchmark it against the native PyTorch implementation.
Why: Python overhead and memory reads kill inference. Fused kernels save milliseconds that compound at scale.
4.) Speculative Decoding Pipeline
Build: A serving setup where a local 2B "draft" model generates tokens in parallel and a 70B "target" model verifies them in a single forward pass.
Why: 2x decode speed at zero quality loss is the closest thing to a free lunch in inference.
5.) Disaggregated Prefill/Decode Cluster
Build: Separate GPU pools where compute-heavy prefills and memory-bound decodes are routed to different hardware optimized for each phase.
Why: This is the exact architecture frontier labs use to maximize MFU (Model Flops Utilization) in 2026.
6.) FP8 Quantization & Calibration Harness
Build: A pipeline that calibrates a model for FP8 inference, serves it via TensorRT-LLM and measures the exact perplexity drop against throughput gains.
Why: Quantization is the only way to fit frontier models on edge GPUs and protect your cloud margins.
7.) Queue-Based GPU Autoscaler (KEDA)
Build: An autoscaler that spins up spot GPU nodes based on pending token queue depth and gracefully drains them when the queue empties.
Why: Idle H100s burn $3+/hr. FinOps and scheduling are now core infra responsibilities.
8.) Complexity-Routing AI Gateway
Build: A proxy that analyzes prompt entropy and token length to instantly route simple queries to a quantized local 8B model and complex reasoning to a frontier API.
Why: Unit economics decide if your AI product survives the next API price hike.
9.) Multi-Node Tensor Parallelism Setup
Build: Shard a 70B+ parameter model across 4 nodes using NCCL over RDMA/RoCE v2 and profile the exact network bottleneck between GPUs.
Why: Single-node inference is dead for frontier models. You must master the interconnect.
10.) Semantic Cache Layer
Build: A Redis + Vector DB proxy that intercepts requests, checks for >98% embedding similarity, and returns cached outputs without hitting the GPU.
Why: The cheapest and fastest token is the one you never generate.
11.) DCGM Observability & Throttling Dashboard
Build: A Grafana stack tracking GPU temp, SM occupancy, VRAM OOMs and PCIe TX/RX to catch silent hardware throttling.
Why: You cannot fix an SLO violation if your dashboard only shows HTTP 200s while the GPU is melting.
12.) MoE (Mixture of Experts) Load Profiler
Build: An expert-parallel deployment tracking token-to-expert routing to identify and fix expert load imbalance and communication overhead.
Why: Sparse MoE models are the 2026 standard, and naive routing destroys their latency advantage.
13.) Inference Chaos & Fault Tolerance Suite
Build: A harness that injects network partitions, simulated GPU ECC errors, and OOMs to test the cluster's ability to migrate KV-caches and recover.
Why: At scale, hardware fails daily. Your architecture must heal itself.
14.) Checkpointing & Weight Streaming Engine
Build: A system that streams model weights directly from an S3 bucket to GPU VRAM on cold-start, bypassing local disk bottlenecks.
Why: Cold-start latency kills autoscaling. Streaming weights directly to memory is the ultimate optimization.
15.) Public Inference Benchmark Teardown
Build: 3 published deep-dives with Pareto curves (Cost vs TTFT vs Throughput) comparing vLLM vs SGLang vs TensorRT-LLM on your specific hardware.
Why: Public proof of hardware mastery and methodology gets you hired instantly by top AI labs.
Systems that prove you can serve, optimize & scale when silicon meets intelligence.
Bookmark & Repost.