Most production inference services run on Kubernetes, where declarative configuration helps coordinate distributed services. Running one inference server in a Slurm job is easy; running a production-style topology across multiple containers and nodes is not.
NVIDIA’s open-source srt-slurm ( provides a YAML-based orchestration layer for these deployments. It coordinates sbatch and srun, GPU placement, networking, readiness checks, and cleanup for systems involving prefill/decode disaggregation, multiple workers, Dynamo frontends, KV-cache-aware routers, and KV-cache offloading services.
It’s useful for quickly deploying representative inference systems for testing or benchmarking on GPU clusters that already run Slurm.
Great work to
@0xishand and the rest of the
@NVIDIAAI team on this!