MTP speculative decoding in llama.cpp, faster but not identical

People treat speculative decoding as a free lunch. A small, fast model guesses a few tokens in advance, the large model checks them all in one pass, and each accepted guess is a token you didn't wait for. In the textbook version the output is mathematically identical to the large model's solo output, since it only accepts tokens it would have picked anyway. A recent twist drops the second model. Some models are trained with a multi-token prediction layer (an extra head that predicts the next few tokens), and llama.cpp can use that head as the drafter. Qwen3.8-27B is one of them, and the GSQ-RCO quantization I run on my 16GB RTX 5060 Ti ships the head inside the file. I tested the head on one file with every other setting pinned. The speed-up was as advertised; the side effect wasn't, because the answers change. Below I diagnose both. If you use a local model to judge prompts, compare outputs or debug a regression, keep the second finding in mind.

The symptom: twice the speed and different answers

I ran fifty Python problems through the same GGUF file twice, once with --spec-type draft-mtp and once without, at temperature zero with a fixed seed, reasoning off, one slot, every other flag identical. Then I compared speed, correctness and the text of each answer.

Speed: the median decode rate went from 29.3 to 56.8 tokens per second, a paired ratio of 1.94, and all 50 problems decoded faster with the head. The middle 80% of the ratios fell between 1.82 and 2.07, so the gain barely moved from one problem to the next. The median wall time per answer fell from 17.7 seconds to 9.5.

Correctness: 34 problems solved with the head, 33 without. Five problems were solved only with it, four only without, and the paired test says what you would expect from that split: no difference.

Text: only 7 of the 50 answers were identical character for character. The other 43 differed, sometimes in a variable name, sometimes in the structure of the solution, sometimes in whether the solution was right.

Without the head, the same five workloads decoded at 29.3, 29.6, 29.5, 29.3 and 28.2 tokens per second, in the order listed in the chart.

The second symptom, the changing text, is the subject of this diagnosis.

If you run local models where outputs must repeat, for evaluation, testing or audit, my hub on AI infrastructure for companies describes how I keep those pipelines deterministic.

Why should speculation not change anything?

Because, in exact arithmetic, greedy speculative decoding only ever emits tokens that the large model would have chosen. The draft proposes a few tokens, the large model scores every position in one batched pass, and the first disagreement ends the accepted run; from there the large model's pick is used. At temperature zero that pick is the argmax, and the argmax doesn't care who proposed the candidate. The output should be the same string.

That's the theory, and it's also why many people assume a draft is safe for anything. The measurement says otherwise, so something in the practice differs from the theory.

Diagnosis: the same arithmetic in a different order

Checking a speculative batch isn't the same computation as decoding one token at a time. Verifying four draft tokens means one pass over a short span instead of four single-token passes. On paper the numbers agree. In floating point they may not, because the order of accumulation in matrix products and attention follows the shape of the batch. Floating-point addition is not associative, and a different order can move the last bits of a logit.

Most of the time a change in the last bits changes nothing. When two candidate tokens are nearly tied, they do: a difference in the eighth decimal place can flip which one is the argmax. Once one token differs, the rest of the answer goes elsewhere, since every later token is conditioned on everything before it. So the differences I saw went from a renamed variable up to another algorithm: they start at one near-tie and grow from there.

At temperature zero a model is deterministic for a given computation. Change the shape of the computation, and the near-ties decide differently.

I've seen this before on the same card: serving a model with four parallel slots instead of one changed 3.7% of deterministic answers between two identical passes, because batching concurrent requests also reorders the operations. Speculative decoding is batching inside one request, with a bigger effect: 43 of 50 long code answers changed. Long answers hold many more near-ties than short ones, so the odds that one of them flips go up with length.

This diagnosis is my reading; I haven't traced it down to individual logits. It does predict something testable. If the cause is accumulation order and not a bug, the changed answers should be no better and no worse on average, and the correctness numbers say just that: 34 against 33, five problems gained and four lost. With the head the model is as capable as before; it just walks another path through the same distribution.

How much speed, on which workload

The gain depends on how predictable the text is, because the draft only helps when its guesses are accepted. The benchmark above already shows the split by workload.

Code without reasoning gains the most, 1.94 times. Sampled reasoning gains 1.53 to 1.64 times. Code is predictable: indentation, closing brackets, repeated identifiers, boilerplate. A temperature of 0.6 makes the large model's choices less predictable for the draft, so fewer guesses are accepted. On very short answers the gain shrinks further, because the fixed cost of each verification pass is spread over few tokens: in a separate session, short general answers on the same file decoded at 45.6 tokens per second against 56.7 on longer outputs.

The same pattern appears in an entirely different engine. Strata, which I tested with its own speculative decoding on a much larger model, decoded code at 46 tokens per second and prose at 36: the draft guesses code better than English.

The cost in memory

The head is not free. On the same file at a 131,072-token context, the server occupied 15,450 MiB with the head active and 14,154 MiB without it: 1,296 MiB for the head and its draft state. At that context the model still fits, but the head-enabled configuration leaves only about 860 MiB of the card free.

An earlier probe, outside my harness and at a different context size, measured a smaller overhead of 942 MiB with a similar speed gain, from 29.7 to 58.7 tokens per second. The memory cost most likely grows with the context because the draft keeps its own state; the speed gain does not.

The head across a whole campaign

The isolation test above uses one file with and without the head. Across the full 45-hour campaign the head was always on for the GSQ-RCO build, and the comparison with the uniform IQ4_XS build of the same model, which has no head, shows how the gain behaves on different kinds of work. These ratios mix the head with the smaller file, so they are larger than the head alone:

In every pair the head-enabled build was faster. The GSQ-RCO file is 1.28 times smaller, and on a card where decoding is limited by how many bytes of weights must be read per token, that accounts for part of the gap: without the head it decodes at 29.3 tokens per second against 25.4 for IQ4_XS, a factor of 1.15. The isolation test says the head adds a factor of about 1.94 on code. The two multiply to roughly the observed 2.2.

The agentic suite is the most interesting row. An agent writes a lot of structured, repetitive text, tool calls with JSON arguments, file edits that echo existing code, and the draft guesses it well: the ratio there is as high as on plain code.

What happens with sampling instead of greedy decoding

So far I've talked about temperature zero, where the large model's pick is deterministic and speculation should in theory change nothing. With sampling (temperature 0.6, say) two runs differ anyway unless the seed is fixed. In its standard formulation, speculative sampling uses a rejection scheme that keeps the large model's output distribution intact: a draft token survives with a probability that corrects for the drafter's bias. I haven't checked how faithfully llama.cpp implements that. The distribution survives; the individual sample doesn't, because the random numbers get consumed in another order with and without the draft. A fixed seed reproduces a run only if the speculative configuration matches too.

So, for sampled generation, the head changes nothing you could rely on in the first place, and for greedy generation it changes something you might have relied on. The second case is the one that surprises people.

How to check the drift on your own setup

The check is short and worth running before you rely on a speculative configuration for anything that compares outputs.

1. Start the same model twice, identical flags, one with the head and one without.

   $ llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf -ngl 99 -c 65536 -fa on --cache-type-k q4_0 --cache-type-v q4_0 -b 512 -ub 256 -np 1 --fit off --spec-type draft-mtp --port 8080
   $ llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf -ngl 99 -c 65536 -fa on --cache-type-k q4_0 --cache-type-v q4_0 -b 512 -ub 256 -np 1 --fit off --port 8081

2. Send the same prompts to both at temperature zero with a fixed seed, and save each answer to a file.

3. Compare the answers by hash first, then diff the ones that differ.

   $ sha256sum answers-mtp/*.txt answers-nomtp/*.txt | sort | uniq -c -w64 | sort -rn | head

   Verify: identical answers produce identical hashes. Any pair with different hashes is worth a look in a diff view.

For a quick visual check of two answers, the text diff tool on this site shows exactly where the two texts diverge.

When to use the head, and when not

Turn it on for interactive use. For a developer waiting on a coding answer, a 1.94 times speed-up with no loss in correctness is the single largest improvement any setting gave me on this card.

Turn it off when outputs are evidence. If you compare prompts, evaluate models, run regression tests on generated text, or need to reproduce an answer exactly, measure without the head, or at least with the same speculative configuration on both sides of every comparison. Comparing a speculative run with a non-speculative one introduces a source of difference that has nothing to do with what you are testing.

What this changes in how I read benchmarks

A speculative head changes outputs, so any benchmark that puts a speculative configuration next to a plain one deserves a footnote. Two quantizations of one model can't be compared fairly if only one carries the head: the speed gap mixes the head with the file size, and the answer gaps mix the head with the weights. The trap is exactly there when you set the GSQ-RCO build against the uniform IQ4_XS build of the same model, which I analysed in Qwen3.8-27B GSQ-RCO vs IQ4_XS: of the roughly 2.2 times speed gap between them, 1.94 comes from the head and the rest from the smaller file.

The broader catalogue of these traps is in what llama.cpp benchmarks get wrong, and the configuration I run every day, head included, is in the recommended strategies for 16GB cards. The flag itself is documented in the llama.cpp repository, and the model card of Qwen3.8-27B describes the multi-token prediction layer that makes the self-drafting possible.

A speculative head is right for one job and wrong for another, and you learn which from a diff of the answers, not from the speed chart. On this card it nearly halved my waiting time and changed almost every long answer at temperature zero; keep those two facts apart when you decide. If you run local models for work where outputs are evidence and want a setup where speed and reproducibility don't fight each other, you can describe the setup in the free quote form: two minutes, and I tell you upfront whether it is something I can help with.