This is the strangest property of LLM inference:
Your LLM's output depends on requests that aren't yours.
The same prompt can produce different logits depending on how many other requests share its batch on the GPU.
Here's what happens inside the GPU.
Most of an LLM's forward pass is spent accumulating sums.
Every matmul, RMSNorm, and attention step reduces thousands of floating-point values into a single output, so the order of those additions directly affects the result.
The catch is that floating-point addition isn't associative.
Every addition rounds the result to the nearest representable value, so (a + b) + c and a + (b + c) can produce different low-order bits.
fp16 has an 11-bit significand, so between 1024 and 2048 the spacing between representable values is 1.0.
If you compute 1024 + 0.4 + 0.4 left to right, each addition rounds back to 1024. If the two small values are added first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds to 1025.
The order of accumulation depends on batch size, because kernels choose their parallelization strategy to maximize GPU utilization.
A GPU like the H100 has 132 compute units, and a small batch doesn't give each of them enough work.
So the kernel splits each long sum into chunks, adds the chunks on different units at the same time, and then adds those partial results together.
So a different batch size gives a different partitioning, a different accumulation order, and slightly different logits.
Usually the difference is negligible, except when two tokens are nearly tied.
In a batch of 8:
- "Queens" might score 12.3417
- "New" might score 12.3415
In a batch of 32, the rounding might shift them to 12.3414 and 12.3416.
If temperature is 0, argmax will pick "Queens" in the first case and "New" in the second. Every later token is conditioned on a different prefix, so the whole completion diverges.
So the nondeterminism isn't coming from sampling, but rather from the fact that your request's numerics depend on concurrent traffic you don't control.
This is pretty evident at Temperature 0, but the nondeterminism comes from the forward pass, not the sampler.
You can fix this by making every reduction batch invariant, meaning each row is reduced in the same fixed order regardless of how many rows share the batch.
This costs throughput, because the kernel can no longer select the optimal split for each batch size. It uses one fixed strategy even when that leaves compute units idle.
The major inference engines now expose this as a flag:
→ vLLM has VLLM_BATCH_INVARIANT=1, still in beta.
→ SGLang has --enable-deterministic-inference, at a 25-45% slowdown.
→ DeepSeek's DeepInfer has DEEPINFER_BATCH_INVARIANT=1.
If you run evals, RL rollouts, or anything that needs reproducible outputs, enable it and treat batch composition as part of your experimental setup.
And in production, the batch around your request never stays fixed. With continuous batching, the server adds and removes requests at every decoding step.
I wrote an article that explains how it works and why every major inference engine uses it.
Read it below.
더 보기