SITUATION EXPLAINED: for anyone benchmarking or serving with mlx_lm.server: requests that carry a seed are never batched. โ ๏ธ
I traced the code. Each request goes through _is_batchable, which needs seed is None. A seeded request runs alone on the single generation thread, and nothing else is served until it finishes. Worse: if one arrives while a batch is running, the server stops taking new requests, finishes the batch, then runs the seeded one by itself. One seeded client stalls everyone. ๐
WHY: the seed resets MLX's global RNG, which can't be shared across a batch. It's a choice for reproducible output, not an MLX limit:
- llama.cpp: per-slot sampler with its own RNG, batched together
- vLLM: per-request generator/seed, stays in the batch
- mlx-serve: per-request random keys on MLX, batched
- omlx: keeps them batched, but seed is global so best-effort under concurrency
- mlx_vlm: batches them, but every request in a running batch uses the first request's sampler (seed, temp, top_p) until the server goes idle
PRACTICAL TIPS for mlx_lm:
- At temp 0 drop the seed: greedy decoding never uses it
- Compare summed per-request t/s with real throughput (tokens / wall time). If the sum is much higher, your requests didn't overlap
- --kv-bits, images and draft models also disable batching in mlx_lm.server โ ๏ธ
Code: