Register and share your invite link to earn from video plays and referrals.

Ivan Fioravanti
@ivanfioravanti
LLM addicted, Apple MLX, NVIDIA DGX, Cloud, Kubernetes, Technology Advisor, Investor and Co-Founder / Board Member of CoreView. Local AI Ambassador.
Joined June 2009
1.5K Following    45.3K Followers
SITUATION EXPLAINED: for anyone benchmarking or serving with mlx_lm.server: requests that carry a seed are never batched. âš ī¸ I traced the code. Each request goes through _is_batchable, which needs seed is None. A seeded request runs alone on the single generation thread, and nothing else is served until it finishes. Worse: if one arrives while a batch is running, the server stops taking new requests, finishes the batch, then runs the seeded one by itself. One seeded client stalls everyone. 👀 WHY: the seed resets MLX's global RNG, which can't be shared across a batch. It's a choice for reproducible output, not an MLX limit: - llama.cpp: per-slot sampler with its own RNG, batched together - vLLM: per-request generator/seed, stays in the batch - mlx-serve: per-request random keys on MLX, batched - omlx: keeps them batched, but seed is global so best-effort under concurrency - mlx_vlm: batches them, but every request in a running batch uses the first request's sampler (seed, temp, top_p) until the server goes idle PRACTICAL TIPS for mlx_lm: - At temp 0 drop the seed: greedy decoding never uses it - Compare summed per-request t/s with real throughput (tokens / wall time). If the sum is much higher, your requests didn't overlap - --kv-bits, images and draft models also disable batching in mlx_lm.server âš ī¸ Code:
Show more