Determinism in local LLM output, measured run against run

Determinism sounds like a detail until you need it. Compare two prompts, rank two models, write a regression test against generated output, or try to reproduce an answer a colleague got yesterday: each time you take for granted that the same input gives the same output. With a local model at temperature zero and a fixed seed, most people think that's safe. I tested it on my 16GB RTX 5060 Ti over several weeks (my benchmark harness needs it, so the test came for free), and the answer isn't a plain yes or no. In some configurations the output matched down to the last character on two servers and over several days. In others, a knob that seems to have nothing to do with the output changed most answers. What ties the cases together: a language model's output follows from the exact arithmetic behind it, and several settings alter that arithmetic silently. Below is each of those settings, with its measurement.

What does temperature zero actually guarantee?

It guarantees that the model picks the highest-scoring token at each step, given the scores it computed. Nothing promises that those scores come out the same every time. They come from billions of floating-point multiplications and additions on the GPU. Floating-point addition isn't associative: add the same numbers in another order and the last bits can move. Usually nobody notices. When two candidate tokens are almost tied, those bits pick the winner, and from that token on the answer goes elsewhere.

So the question to ask is whether my configuration always performs the arithmetic in the same order. Below, configuration by configuration, is the measured answer.

If you run local models in processes where reproducibility is a requirement, my hub on AI infrastructure for companies describes how I design those pipelines.

Case 1: one slot, same engine, same flags: identical

The baseline is a model served on one slot, with every flag pinned, at temperature zero with a fixed seed. In my reproducibility campaign I ran 134 deterministic items twice through a 9B model set up like that. Zero cells changed between the two passes: 0 of 134.

A tougher version of this test came later, by accident. I'd sent fifty coding problems to two production models through my everyday model loader, with a 131K context and the loader's settings. A few days later they went again through my benchmark harness, with a 65K context, another time cap and faster RAM in the machine. All 50 answers were identical character for character for both models, the dense Qwen3.8-27B and the Ornith 35B mixture of experts. A third pass, with the models run in reverse order on the same server configuration, gave 247 identical answers out of 250 across five models, and the same pass/fail result on all 250.

For llama.cpp on one slot that says a lot. Context size, serving path, day and position in a long session left the arithmetic alone for a model held in VRAM. The three answers that differed in the reverse-order pass were two degenerate loops cut at a time limit (where the cut lands is a matter of streaming speed) and one answer from the 35B MoE on a problem it failed in both passes.

On one slot with pinned flags, a model held entirely in VRAM gave the same fifty answers, character for character, across two servers and several days. That is the standard every other configuration is measured against.

Case 2: parallel slots: 3.7% of answers change

Serve the same model with four parallel slots and send concurrent requests, and the server batches them together. A batch of four sequences is computed with different matrix shapes than four single sequences, and the order of accumulation changes. On the same 134 deterministic items, run twice on four slots, 3.7% of the cells flipped between two identical passes.

3.7% sounds small. It is the same size as a difference of two items out of fifty, the size of difference people report when they say one model beats another on a small benchmark. A four-slot evaluation has a noise floor as large as the effects it is trying to measure.

Case 3: a speculative decoding head: 43 of 50 answers change

Some GGUF files include a multi-token prediction head that llama.cpp can use as a built-in draft model. In theory greedy speculative decoding cannot change the output, because the large model only accepts tokens it would have chosen. The large model verifies the draft's guesses in a small batch, and a batch changes the arithmetic. On fifty coding answers from the same Qwen3.8-27B file, with and without the head, only 7 answers were identical. The number of correct answers did not move, 34 with the head and 33 without, so the head did not make the model better or worse; it sent it down different paths. I analysed this case in detail in MTP speculative decoding in llama.cpp.

Case 4: an engine that computes part of the model on the CPU: 46 of 50 change

The most extreme case was Strata, an engine that runs mixture-of-experts models larger than the card by keeping frequently used experts in VRAM and computing the rest on the CPU. I ran fifty coding problems twice at temperature zero, on the same engine and model, a day apart with faster system RAM in between. Only 4 of the 50 coding answers were identical between the two runs, while all fifty short general answers were identical.

Where the answers split tells you something: the first difference fell, in the median, at character 517, and sometimes as early as character 10. The solved count went from 32 to 28. That's noise, yet it also means one run of that engine is uncertain by a few problems out of fifty. My hypothesis, not verified, is that the content of the GPU expert cache depends on the history of the session, so the same expert is sometimes computed on the GPU and sometimes on the CPU with AVX2, and the two produce slightly different numbers. The long code answers, with many near-ties, diverge; the short general answers, with few, do not.

The chart puts my measurements side by side: the arithmetic path decides, not the temperature.

Case 5: sampling with a seed

With temperature above zero, two runs differ by design. A fixed seed makes them repeatable only if everything that consumes random numbers is identical: the sampler settings, the number of tokens generated before each choice, and the speculative configuration, because speculative sampling consumes random numbers differently. In my reasoning-on measurements I change the seed on every repetition on purpose: three repetitions should be three samples, not three copies of one. If you need to reproduce a sampled answer, record the seed, the full sampler and the server configuration together; you need all three.

What does not affect determinism, in my measurements

Some things people worry about turned out not to matter on this card, as long as the model fits in VRAM and runs on one slot:

One note on the prompt cache. With it on, a prompt sent twice skips its prefill and comes back much faster, and that falsified my timing measurements for weeks. Could reusing a cached prefix also change the text, since the prefix was computed with another batch shape? I haven't measured it, so I keep the cache off whenever outputs are compared.

How to verify determinism on your setup

The procedure is short and fits into any evaluation pipeline.

1. Fix the configuration: one slot, pinned flags, temperature zero, a fixed seed, and the same speculative setting on every run.

2. Run the same prompt set twice, ideally hours apart, and store the answers as files.

3. Hash the answers and compare the hashes.

   $ cd run1 && sha256sum *.txt > ../run1.sha && cd ..
   $ cd run2 && sha256sum *.txt > ../run2.sha && cd ..
   $ diff run1.sha run2.sha | grep -c '^<'

   Verify: zero lines means every answer was identical. Any other number is the count of answers that changed.

4. For each changed answer, find the first differing character. An early divergence on a short answer suggests a configuration problem; a late divergence on a long answer is usually a single near-tie.

For a quick check on a single text, the hash calculator on this site computes SHA-256 in the browser, which is enough to compare two answers without a terminal.

What determinism does not give you

A reproducible answer isn't a correct one. I say it out loud because people confuse determinism with reliability all the time. On the fifty coding problems without reasoning, the five models I measured failed 148 times in total, and 146 of those failures were silent: complete, plausible code that only the tests caught. Run them again on one slot and all those wrong answers come back, character for character. Determinism makes a wrong answer stable, and just as wrong.

What it does give you is a way to reason about change. If a model's answer to a prompt moves and you know your setup is deterministic, then something real moved: the prompt, the model file, the engine, a flag. If the setup isn't deterministic, any change could be noise, and you can't tell an improvement from it. Separating the two is what you pay for, and it's worth a lot.

Determinism in agent loops

Agentic runs are a special case. An agent loop sends many requests one after another, each built on the previous answers and on what its tools printed. The model may be deterministic while the tools aren't: a test suite that prints durations, a file listing in another order, a timestamp in a log line. One tool output changes, the next prompt changes, and so does everything after it. In my agentic measurements I don't try to make episodes reproducible. I sample them at a moderate temperature with a new seed per repetition, treat each run as one observation, and put models side by side on the same tasks with paired statistics. For a single-turn evaluation, full determinism is achievable and worth having. For an agent loop, controlled randomness with enough repetitions is the design I trust.

The decision table

Beyond benchmarks

People talk about reproducibility as a scientific nicety. For a business it's an engineering property. A regression test on generated text that fails one time in thirty because of batching soon gets ignored. If an evaluation ranks two prompts one way on the first run and the other way on the second, you might as well toss a coin. And if an audit can't reproduce the answer a system gave a customer, nobody can say what happened. Each of these failures goes back to a configuration choice that looked unrelated to the output: a slot count, a draft model, an offloading engine.

Full determinism is cheap on one card: one slot, pinned flags, no speculation for the runs you'll compare, and a model held in VRAM. The configuration I use is in the recommended strategies for local LLMs on a 16GB card, the measurement traps that taught me to check this are in what llama.cpp benchmarks get wrong, and the engine whose answers changed the most is examined in the hands-on test of Strata. Why floating-point addition isn't associative is explained in the classic What Every Computer Scientist Should Know About Floating-Point Arithmetic.

With a local model you own every setting that can move an answer, which no API lets you do. Determinism comes with that ownership, and breaks only when a setting reorders the arithmetic behind your back. Measure it once on your own configuration, then guard it the way you'd guard a build pipeline. If you're building evaluations or regression tests around local models and want a setup that's reproducible from day one, you can describe the case in the free quote form: two minutes, and I tell you upfront whether I can help.