Non-deterministic LLM Results Even with Temperature 0

Temperature 0 is often read as “deterministic”. It usually means “always pick the highest-scoring next token” — not “bit-identical every run”.

Seeing something like 80 different outputs in 1000 invocations (~8%) is plausible in production systems.

Why it still varies

Floating-point noise. A 70B model runs a huge number of FP ops across GPUs. Two candidates can sit almost on top of each other:

Paris = 12.441823
Lyon  = 12.441819

A rounding difference of 0.000001 can flip argmax. Once one token changes, the rest of the completion can diverge.

Non-associative floats. In exact math, (1e20 + 1) - 1e20 = 1. On a computer, near 1e20 the spacing between representable numbers is larger than 1, so 1e20 + 1 rounds to 1e20, and the expression becomes 0. Parallel GPU reductions change add order → tiny logit shifts.

Ties, MoE routing, dynamic batching. Near-tied tokens, expert routers that flip on noise, and different batch sizes on the inference server all introduce the same kind of micro-variation.

Cascading effect

Tiny numeric difference → different token → different context → different future predictions → a very different answer. Temperature 0 removes intentional randomness; it does not remove numerical nondeterminism.