Most production inference services run on Kubernetes, where declarative configuration helps coordinate distributed services. Running one inference server in a Slurm job is easy; running a production-style topology across multiple containers and nodes is not. NVIDIA’s open-source srt-slurm ( provides a YAML-based orchestration layer for these deployments. It coordinates sbatch and srun, GPU placement, networking, readiness checks, and cleanup for systems involving prefill/decode disaggregation, multiple workers, Dynamo frontends, KV-cache-aware routers, and KV-cache offloading services. It’s useful for quickly deploying representative inference systems for testing or benchmarking on GPU clusters that already run Slurm. Great work to @0xishand and the rest of the @NVIDIAAI team on this!