๊ฐ€์ž… ํ›„ ์ดˆ๋Œ€ ๋งํฌ๋ฅผ ๊ณต์œ ํ•˜๋ฉด ๋™์˜์ƒ ์žฌ์ƒ ๋ฐ ์ดˆ๋Œ€ ๋ณด์ƒ์„ ๋ฐ›์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

Ivan Fioravanti
@ivanfioravanti
LLM addicted, Apple MLX, NVIDIA DGX, Cloud, Kubernetes, Technology Advisor, Investor and Co-Founder / Board Member of CoreView. Local AI Ambassador.
๊ฐ€์ž… June 2009
1.5K ํŒ”๋กœ์ž‰ ์ค‘    45.3K ํŒฌ
SITUATION EXPLAINED: for anyone benchmarking or serving with mlx_lm.server: requests that carry a seed are never batched. โš ๏ธ I traced the code. Each request goes through _is_batchable, which needs seed is None. A seeded request runs alone on the single generation thread, and nothing else is served until it finishes. Worse: if one arrives while a batch is running, the server stops taking new requests, finishes the batch, then runs the seeded one by itself. One seeded client stalls everyone. ๐Ÿ‘€ WHY: the seed resets MLX's global RNG, which can't be shared across a batch. It's a choice for reproducible output, not an MLX limit: - llama.cpp: per-slot sampler with its own RNG, batched together - vLLM: per-request generator/seed, stays in the batch - mlx-serve: per-request random keys on MLX, batched - omlx: keeps them batched, but seed is global so best-effort under concurrency - mlx_vlm: batches them, but every request in a running batch uses the first request's sampler (seed, temp, top_p) until the server goes idle PRACTICAL TIPS for mlx_lm: - At temp 0 drop the seed: greedy decoding never uses it - Compare summed per-request t/s with real throughput (tokens / wall time). If the sum is much higher, your requests didn't overlap - --kv-bits, images and draft models also disable batching in mlx_lm.server โš ๏ธ Code:
๋” ๋ณด๊ธฐ