What llama.cpp benchmarks get wrong, and how to catch it

I've read a lot of local inference benchmarks in the last year, and I've produced a few wrong ones myself before building a harness that wouldn't let me. It always goes the same way: a number that is true, a title that claims something else, and a gap between the two that the reader can't see. "The 35B MoE decodes at 112 tokens per second" is true on my card. "So it's the fastest model to work with" doesn't follow, and on some tasks it's the opposite. A measurement only supports a claim under the conditions that produced the number, and llama.cpp has plenty of hidden ones. Below are the ones that bit me while measuring local models on a 16GB RTX 5060 Ti, ordered by how often I see them in published benchmarks. For each one I give the symptom, the cause, the number that exposed it on my bench, and the check that catches it. The second half describes how my own harness measures, section by section: every other article in this series links to the part of it that produced its numbers.

Why are so many local benchmarks wrong without anyone lying?

Because llama.cpp is designed to work out of the box, and a benchmark wants the opposite: nothing should work out of the box, everything should be chosen. The server adjusts parameters you didn't set, reuses work it already did, batches parallel requests together and stops a generation when a token budget runs out. In daily use you want all four behaviours; in a measurement each one is a confounder.

The second reason is statistical. One run on one prompt gives a number with no error bar, and the error bar on a modern reasoning model is wide. Two runs of a sampled model differ, two runs of some engines differ even at temperature zero, and a difference of one or two items out of fifty is usually noise.

The third reason is the choice of metric. Tokens per second is easy to measure and easy to compare, and it ranks reasoning models in the wrong order for anyone who waits for an answer.

A benchmark is a measurement plus its conditions. Publish the number without them and you've published a rumour.

If you need the same rigor applied to an inference setup for a team, the articles in my hub on AI infrastructure for companies describe how I bring measured evidence into self-hosted decisions before anyone buys hardware.

Symptom 1: the second run is suspiciously faster than the first

Cause: the prompt cache. llama-server keeps the KV cache of the previous prompt and, by default, reuses the longest matching prefix of the next one. If a benchmark repeats the same prompt three times and reports the median, the first repetition pays the full prefill and the other two don't.

I found this in my own data, three weeks after it had started affecting every latency measurement I had taken. On a long prompt, the time to first token of a 9B model was 1,600 milliseconds on the first repetition and 75 milliseconds on the second and third. On a 35B MoE, 3,340 milliseconds and then 50. The median of three was the warm cache each time, and the chart looked plausible because the warm numbers agreed with each other.

The check: look at the repetitions individually, not at the median. If the first one is an outlier on long prompts and the others agree closely, you are measuring the cache. The fix: send "cache_prompt": false in the request body for measurement runs, and keep the default on in production, where you do want the prefix reused. After the fix, an 8,543-token prompt paid 10.2 seconds of prefill on every repetition on a 27B model, with the three repetitions agreeing to within 15 milliseconds.

That's what reading a long prompt costs on this card, and a warm cache hides it.

Symptom 2: the same model loads with different speed on different days

Cause: the automatic fit. Recent llama.cpp builds enable --fit by default. It looks at the free VRAM and adjusts every memory-related parameter you didn't set explicitly, so that the model loads. If your desktop session holds 300 MB of VRAM today and 900 MB tomorrow, the server chooses a different context or a different offload, and the speed changes with it.

The check: after the server starts, read the actual context from the /props endpoint and compare it with what you passed. The fix: pass --fit off and set every parameter that affects memory and speed explicitly: GPU layers, context, both KV cache types, batch, micro-batch, flash attention, expert offload. Then a configuration that doesn't fit fails loudly instead of being silently downgraded. My harness records every one of these flags with each measurement, and numbers taken before I started pinning them are not comparable with the ones after.

Symptom 3: identical runs give different answers at temperature zero

Cause: parallel slots. With -np 4, the server batches concurrent requests together. Batching reorders floating-point operations, and on near-tied tokens that's enough to flip the choice. In my reproducibility campaign, 134 deterministic items were run twice on one slot and twice on four slots, all at temperature zero with a fixed seed. On one slot, 0% of the cells changed between passes; on four slots, 3.7% did.

That doesn't sound like much until you compare two models that differ by two items out of fifty, which is a 4% difference. The noise floor of a four-slot run is as large as the effect you're trying to measure.

The check: run the same evaluation twice and diff the outputs. The fix: one slot for any comparison or evaluation. There is a second trap hidden here: -c is the total context budget divided across slots, so -c 8192 -np 4 gives each request 2,048 tokens. If you report a long-context result from a multi-slot server, say how many slots were configured.

Symptom 4: a reasoning model looks fast in a latency test

Cause: a token budget sized for answers without thinking. A latency probe with a budget of a few hundred tokens measures the time a thinking model takes to reach that budget, and then the server stops it. The reported time is short because the model never finished. My first thinking-on measurements of a 27B model truncated 39 probes out of 48 that way.

When I gave the same kind of probes a 32,768-token budget and a 900-second timeout, nothing was truncated on that model, and the picture changed completely. Asked for a single shell command, the 27B thought for a median of 8,916 tokens and answered after 202 seconds; without thinking, the same request took 2.8 seconds.

The check: look at the finish reason of every response. A length finish means the answer is missing; it doesn't mean it was fast. The fix: set a budget the model cannot reach by accident, report truncations separately, and keep truncated responses out of every median.

Symptom 5: the fastest model in the chart is the slowest one to work with

Cause: measuring throughput instead of waiting time. Tokens per second is the rate at which the model writes. What you experience is the time until the answer is complete, and that's the rate multiplied by how many tokens the model chooses to write, most of which, on a reasoning model, you never read.

On my card the Ornith 35B MoE decodes at 112 tokens per second and the GSQ-RCO 27B at 57. With thinking off, the 27B solved 34 of 50 Python problems and the MoE 19, and both answered in seconds. With thinking on, the MoE answered a chat probe in a median of 5.3 seconds and the 27B in 21.3, because the 27B thought for almost twice as many tokens (1,032 against 586) at half the speed. Neither throughput number predicts either result.

Tokens per second is a property of the engine. Time to answer is a property of the engine and the model's habits together, and only the second one is what you wait for.

Symptom 6: one engine update changes everything, or nothing

Cause: comparing across different conditions. An engine commit, a driver version, a different quantization of the same model, a different context size: each is a different condition. Pool measurements across conditions and you get an interval narrower than the truth, on a question nobody asked.

When I compared the llama.cpp build from June with the current one on the same weights and the same flags, the difference was between +0.4% and +1.9%: three months of releases didn't move this hardware. I trust that result because nothing else changed. My harness stamps every measurement with a condition hash derived from the engine, the model file, every flag and the sampler, and it refuses to aggregate two hashes into one number.

The check: for every number you compare, list the engine commit, the model file, the quantization, the context, the slots and the sampler. If any of them differ, you've got two experiments, not one.

Symptom 7: the run order decides the winner

Cause: thermal drift and session effects. A card that has been at full power for six hours may run at a lower clock than one that has just started, and the model measured last pays for it.

This is one of the few worries that my data dismissed. I measured five models for 45 hours in a fixed order, then re-measured fifty coding problems in the reverse order. All 250 outcomes were identical, 247 answers were identical character for character, and median speeds moved by at most 1.1%. The card logged zero throttle events in the whole campaign, with a maximum of 81 °C at a 180 W power limit. On this card in this case the order didn't matter; on a laptop GPU it may. The check: log the throttle reasons and the SM clock with every measurement, and rotate the model order between sessions if you can't rule drift out.

Symptom 8: a failure is blamed on the engine, or on the model, and it was neither

Cause: the harness cut something and then fed the cut back. In my agentic benchmark, three episodes ended with an HTTP 500 from the server. It looked like an engine bug. It wasn't: in every case the previous turn had ended with finish_reason=length at the 6,000-token per-turn ceiling of the harness, to the token, the tool call in that turn had been cut in the middle of its arguments, the loop had put the broken call back into the history, and the server refused to parse it. The failure belonged to the harness budget, not to the engine and not to the model.

The check: before attributing a failure, look at the turn before it. The fix: classify those episodes separately, as budget cuts, and never count them as model failures.

Symptom 9: a single run is reported as a score

Cause: ignoring run-to-run variance. Some setups aren't deterministic even at temperature zero. When I tested Strata, a runtime that streams mixture-of-experts weights from system RAM, only 4 of 50 coding answers were identical between two runs (the second one after a RAM speed change), and the solved count moved from 32 to 28. A result that looked significant in the first run (p = 0.007 against another model) fell to p = 0.064 in the second.

Speculative decoding has a milder version of the same effect. On the same GSQ-RCO file, enabling its multi-token prediction head changed 43 of 50 answers at temperature zero, while the number of solved problems stayed the same within one (34 against 33).

The check: run twice and diff, always. The fix: report a rate with its interval, compare two models with a paired test on the same items, and call a difference you can't confirm INCONCLUSIVE instead of rounding it into a winner.

How my harness measures local models

The symptoms above are why my harness exists. It's a small Python program that drives llama-server on the bench machine, runs every test I describe in this series, stores one record per measurement with everything needed to reproduce it, and computes the statistics from those records. The application writes the data; I only write what it means. Every other article in the series links to the subsection below that produced its numbers.

The test bench and the pinned conditions

The bench is a home server built mostly from recovered hardware. These are the parts that affect a measurement:

Before every run the harness clears the room. It unloads the production models, stops the model proxy, checks that no other process holds the GPU, and records the baseline VRAM, power limit, clock and throttle state. A trap restores the production stack on exit, error or interrupt.

Every model is started with --fit off and every memory or speed parameter pinned: all layers on the GPU, an explicit context declared by each campaign (8,192 or 32,768 tokens in the September campaigns, 65,536 in the October campaign, 131,072 for the production context tests), a q4_0 KV cache on both keys and values so the fused flash attention kernel stays active, batch 512 and micro-batch 256, six CPU threads, and one slot. The harness then reads the context the server actually allocated before accepting the measurement.

Every record holds a condition hash: the engine commit and build flags, the model file's SHA-256, the quantization, the server flags, the sampler, the version of the test bank, the language and the version of the harness. Evidence is only ever summed within one condition hash. Changing a flag, rebuilding the engine or editing a test starts a new accumulation.

The suites and what each one asks

Each suite answers one question a person choosing a model would ask.

The same 50 problems run with and without reasoning on purpose: the pair isolates what thinking buys on this card. The usable threshold draws the line between "it got there" and "you would have waited for it", and a run cut at the cap counts as a failure that the tables report apart from wrong answers.

Agent tasks and the hidden oracle

The agent suites run a real loop. The model gets seven tools that mirror the ones of the coding extension I use in production (read a file, list a directory, glob, grep, analyse a directory, apply an edit, run a command) inside a sandbox, and works until it stops, runs out of turns or hits the time cap. Each turn may generate at most 6,000 tokens. Sampling is temperature 0.4 with a seed fixed per task and repetition, and every task runs three times.

The duel bank contains broken multi-file projects. Each has a visible test suite the model can run, and a hidden differential oracle it never sees: after the episode, the oracle runs the delivered code and a reference implementation on 40 to 80 generated cases and counts the disagreements. An episode is OK when the visible tests pass and the oracle finds zero disagreements, ILL (illusory) when the visible tests pass and the oracle disagrees, and FAIL when the visible tests are red. Before a task enters the bank, I write down the lazy fixes I expect and the number of disagreements each one produces, and a calibration panel of three models has to land the task in a solve-rate band between 30% and 70%. The earlier agentic bank works the same way on sixteen tasks in fourteen families, from multi-file debugging to schema migrations and concurrency bugs.

Latency, context and quantization probes

Latency. Eight probes, two per profile: a chat question, a request for a single shell command, a code review, and two long prompts. Each probe runs three times on the same server with the prompt cache off, once with reasoning off at temperature 0 and once with reasoning on with a 32,768-token budget and a 900-second timeout. The measure is the wall-clock time until the answer is complete, with the time to first token and the tokens generated beside it.

Context. Two separate measurements. The first loads the model at growing context sizes and keeps the largest one that loads and answers. The second checks that the model actually reads that context: four retrieval tasks (frequent words, a needle with several keys, a needle with several values, a variable followed through a chain of assignments) at 8K, 32K and 128K nominal tokens, with reasoning off.

Quantization. llama-perplexity --kl-divergence compares a quantized file with a higher-precision reference of the same model, on a corpus built from the bench's own tasks and prompts. It reports the mean divergence and the share of positions where both rank the same token first.

Engine and residency. llama-bench measures prefill and decode at fixed KV depths. A residency probe measures what a dedicated card costs: cold load time, VRAM held, idle power, and the time to swap one model for another.

Energy and cost accounting

A telemetry service on the bench estimates the whole machine's draw at the wall every 30 seconds. Only the GPU is measured directly, through the NVIDIA driver's energy counter; the CPU, the power supply's losses and the rest are modelled, without a wattmeter. So every energy figure is an estimate with a band, reported as the energy above the machine's idle baseline of 55.9 W during the run windows. The price is the marginal rate of my own electricity bill, 0.3266 euro per kWh including VAT and excise, fixed charges excluded: the August 2026 rate, the latest in my tariff file when the campaign ran.

Statistics, from intervals to paired tests

Every rate comes with a Wilson 95% interval. Two models measured on the same items are compared with an exact sign test on the discordant pairs, the items one solves and the other doesn't, never by setting two intervals side by side. On agent banks the pairing unit is the task family, because three versions of the same bug are one observation. A question decided before the data is preregistered with its metric, and the preregistered tests get a Holm correction; everything else is labelled exploratory. Each comparison reports the minimum detectable effect at 80% power, so that an INCONCLUSIVE verdict says whether the design could have seen a difference at all.

The verdicts are CONCLUSIVE, INCONCLUSIVE (the default) and UNDERPOWERED, when the design can't get below the threshold whatever the data say. With n task families, the smallest two-sided p-value a paired test can reach is 2/2^n: five families can never go below 0.0625. Metrics about the path, such as turns or edits, are compared only on episodes both models solved, otherwise the model that gives up first wins.

Measurements taken outside the harness

A few numbers in the series come from field tests run outside the harness, and each article that uses one says so. The Strata hands-on was a session on the production proxy with its own scripts. The September sweep was a set of exploratory runs whose loads and timings informed the harness but do not carry a condition hash. Those numbers are real measurements on the same card, and they are labelled as what they are: single sessions, not campaigns.

How to read a benchmark someone else published

The nine symptoms reduce to a short checklist you can apply to any chart before you trust it.

Benchmarks that answer all seven are rare. Ones that answer none are common; they may still be right, but they aren't evidence yet.

The models and the engine I used are all public: the llama.cpp repository documents the server flags discussed here, and the Qwen3.8-27B model card describes the dense model I measured most. The method behind my own numbers is the same one I used in the Ornith vs Qwable comparison, where two coding models tied on pass rate and I reported the tie. The same lessons turned into a configuration are in the strategies for running local models on a 16GB card.

Every one of these mistakes made it into my own data at some point, and each one was caught by a number that did not fit: a repetition too fast, a run that didn't repeat, a failure right after a cut. The habit that catches them is boring, it works, and it's the same for a hobbyist and for a company choosing a model: write down the configuration, run more than once, look at the individual records before the summary, and refuse to call a winner without a paired test behind it. If you are evaluating local models for a team and want the comparison designed and run on your own workload, you can describe the case in the free quote form: two minutes, and I tell you upfront if it's something I can help with.