Recommended technical strategies for local LLMs on a 16GB-VRAM GPU

Most people who try a local model on a 16GB GPU give up for the wrong reason. They download a quantized 27B, start a server with whatever the tutorial said, get a crash or a crawl, and conclude that the card is too small. On the RTX 5060 Ti that runs my measurement bench, a quantized 27B holds 131,072 tokens of context with a speculative decoding head attached, and answers a coding question in under thirty seconds. Between those two outcomes sit a dozen settings that llama.cpp lets you leave implicit, and you should set each by hand. I have been running local inference as part of my personal LLM automation pipeline for a long time, and over the last weeks I built a benchmark harness that measures every configuration I recommend here, with the flags pinned, the output recorded and the statistics done properly. What follows is the condensed result: eleven strategies, each with the number that justifies it, in the order in which they bite. The card is a 16GB RTX 5060 Ti on PCIe Gen3 x8, power-limited to 180 W, in a Ryzen 5 4500 box with 32 GB of DDR4. The models are what I run day to day: Qwen3.8-27B in two quantizations, Ornith-1.5-35B-A3B, and two 9B models for contrast. Nothing in this list comes from a third-party leaderboard.

Why do the defaults fail on a 16GB card?

Because llama.cpp cares about starting, not about being reproducible. Its automatic fitting logic (--fit, on by default) looks at the free VRAM and adjusts every argument you didn't set explicitly so that the model loads. That's a kindness on a first run and a trap on every run after it: the server you think you configured isn't the server you're measuring, and the next model you load gets different adjustments. On my bench, numbers recorded before I started pinning everything aren't comparable with the ones after, because back then a configuration that didn't fit went out of memory instead of being silently downgraded.

The second reason is that the memory budget of a 16GB card is too tight for approximations. The weights of a 27B model at 4 bits take most of the card, and what is left decides how much context you get. A formula can estimate that remainder; only loading the model gives the real figure, and the two can disagree by a factor of two.

The third reason has to do with how models behave. Current models think before they answer, and a thinking model served with the wrong token budget either truncates in the middle of its reasoning or spends minutes on a question that deserved seconds. A tokens-per-second number hides all of it, so plenty of setups look fine in a benchmark and feel broken in use.

On a 16GB card the defaults do not fail loudly. They load something slightly different from what you asked for, and you only notice once you compare two runs.

If you want to see how I carry the same rigor into production systems, the articles in my hub on AI infrastructure for companies cover self-hosted inference with the same method: measured perimeter first, then the architecture.

The eleven settings, in the order they bite

1. Pin every parameter and switch the automatic fit off

Start the server with --fit off and pass every parameter that affects memory or speed explicitly: layers on GPU, context, both halves of the KV cache, batch and micro-batch, flash attention, and any expert offload. This is the configuration my harness uses for every cell:

llama-server -m Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf \
  -ngl 99 -c 65536 -fa on \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  -b 512 -ub 256 -np 1 -t 6 \
  --fit off --spec-type draft-mtp

After the server starts, read the context it actually allocated from the /props endpoint instead of trusting the command line. If the number differs from what you asked for, something adjusted it, and you'll want to know before you measure anything.

2. Size the context by loading, not by formula

The VRAM formula (weights plus KV cache per token times context plus a compute buffer) gives a first estimate and guarantees nothing. On my card it missed both high and low. The planner estimated that Qwen3.8-27B at IQ4_XS would stop at 32,768 tokens of context; loading it showed 65,536 fitting in 15,744 MiB. The GSQ-RCO build of the same model, with its MTP head active, holds 131,072 tokens at 15,752 MiB, and the Ornith 35B mixture of experts holds 131,072 at 15,340 MiB. The 9B models load 262,144 tokens with almost 4 GB to spare.

So estimate first, then load at the estimate and one step above, and keep the largest that answers a long prompt correctly. Loading is the measurement; the formula is a hypothesis.

3. Remember that the context is split across slots

The -c flag is the total KV budget, divided by the number of parallel slots -np. With -c 8192 -np 4 each request gets 2,048 tokens, and /props says so. Serve four parallel users on a 16GB card expecting the full context per user and you've read the wrong number. On this hardware one slot is the sane default for a single developer.

4. Use the same quantized type for both halves of the KV cache

Flash attention in llama.cpp has fused kernels only for homogeneous key and value types. Mixing, for example, an 8-bit key cache with a 4-bit value cache doesn't fail: it falls back to a slower path and you lose throughput without any warning. On this card q4_0 for both halves makes 131K contexts possible while keeping the fast kernel.

5. Serve one slot when you need the same answer twice

At temperature zero with a fixed seed, a model served on one slot gave identical results on 134 out of 134 deterministic test items in my reproducibility campaign. Served on four parallel slots, 3.7% of the cells flipped between two identical passes, because batching changes the floating-point order of operations. If you compare models, evaluate prompts or debug a regression, use one slot, always.

6. Decide reasoning per task, not per model

No other setting changes the day-to-day feel of a local model as much, and it reverses rankings. On fifty Python problems with executable tests, with reasoning off, the GSQ-RCO 27B solved 34 and the Ornith 35B mixture of experts 19, a difference that survives a paired test (17 problems to 2, p = 0.0007, 0.004 after the Holm correction across the preregistered tests). With reasoning on and a five-minute budget per problem, the order flips: 26 for the 35B, 12 for the 27B (16 to 2, p = 0.0013, 0.007 after Holm). The 27B didn't get worse at coding; all of its 28 failures were answers still being written when the clock ran out.

Reasoning's cost in waiting time varies just as wildly. Across eight latency probes, the median wait of the 27B goes from 5.9 seconds to 131 seconds when it thinks, the 35B from 2.7 to 8.7. Asked for one shell command, the 27B took 202 seconds and almost nine thousand generated tokens.

Treat reasoning as a budget you spend, and stop thinking of it as a quality knob. Turn it on where a wrong answer costs more than a three-minute wait, and off everywhere else.

7. Use the speculative decoding head the file ships with

Some recent quantizations include a multi-token prediction (MTP) head. With --spec-type draft-mtp on the same GSQ-RCO file, decoding on code went from 29.3 to 56.8 tokens per second, a median ratio of 1.94 over fifty problems; on sampled reasoning the gain was 1.5 to 1.6 times. The head costs 1,296 MiB of VRAM: at 131,072 tokens of context the model then holds 15,450 MiB of the card's 16,311, so it still fits, with little room left. Anyone who tests prompts should know one more thing: at temperature zero, only 7 of the 50 answers were identical with and without MTP, while the number of solved problems stayed equivalent (34 against 33). Speculation here is fast, not output-neutral.

8. Give thinking modes a token budget they cannot hit by accident

A latency probe with a budget of a few hundred tokens measures a thinking model's ability to be cut off, not its speed. In my first reasoning-on measurements 39 of 48 probes were truncated by budgets written for non-thinking answers. Set a large budget (I use 32,768 tokens) and a request timeout, and treat a truncated answer as a failure, never as a fast answer.

9. Turn the prompt cache off when you measure, keep it on when you work

llama-server reuses the KV cache of an identical prompt by default. In an editor session that's what you want; in a benchmark it's what you don't: on a long prompt the time to first token of a 9B model dropped from about 1.6 seconds to 75 milliseconds on the second repetition, and a median of three repetitions reported the warm cache. Send "cache_prompt": false in the request body when you measure.

10. Check the bus and the memory you plug the card into

The RTX 5060 Ti is electrically x8. In a Ryzen 5 4500 platform it negotiates PCIe Gen3, about 7.9 GB/s. With the whole model in VRAM that is irrelevant; with expert offloading it is the ceiling. Likewise for system RAM: with experts offloaded to the CPU, 12 GiB of host memory was enough to keep a 35B mixture of experts at full speed in my sweep, while the RAM speed mattered less: when I enabled the DOCP profile and moved my DIMMs from 2,133 to 3,600 MT/s, an engine that streams experts from RAM decoded about 6% faster on prose and 10% faster on code. If you plan to offload, measure your platform first.

11. Keep the power limit and watch the clock, not the temperature

My card runs at a 180 W limit with case fans off. Over forty-five hours of continuous benchmark load it never throttled, with a maximum of 81 °C, because the GPU fan alone handles that power. Log the throttle reasons and the SM clock with each measurement: a speed number recorded while the card was throttling is a number about your cooling.

What the settings change in practice

The table lists the eleven points with the cost of ignoring each one on this card.

The first five are about making sure you measure what you think you measure. The next four decide how the model feels to use. The last two concern the hardware under the card, and they matter only if you offload or if your cooling is marginal.

How I verified every number in this list

All the figures above come from my own harness, not from a model card or a forum thread, and the section on the test bench and its pinned conditions lists every setting. Every measurement is a record with the full server configuration, the engine commit, the GPU state and a condition hash, so that two numbers are compared only when they were produced under the same condition. Rates come with Wilson intervals, two models are compared only with a paired test on the same items, and a comparison that the data cannot decide is reported as INCONCLUSIVE, with no winner named. I wrote about the hardware side of this choice in why a 16GB consumer GPU is now enough for local inference, and the measurement discipline grew out of the comparison in Ornith vs Qwable on a 16GB GPU, where two coding models tied on pass rate and differed only in how they got there.

The engine is upstream llama.cpp, built for the Blackwell architecture of the card. Its repository lists every flag used here, and the models come from their official Hugging Face pages, starting with the Qwen3.8-27B model card, released in August 2026 under Apache 2.0.

The mistakes I see most often

Three patterns account for most of the broken local setups I come across. First, people copy a command line from a tutorial written for another card, usually one with more memory, then keep adding --n-gpu-layers until it loads. Next, they pick a model off a throughput chart, which with today's reasoning models rewards the fastest thinker rather than the shortest wait. Last, they test with one prompt, once, and take that answer as representative.

The fix for all three is the same habit: write down the configuration, measure on several tasks, and repeat on one slot. It sounds pedantic until the day an answer moves after a model update and you need to know whether the model changed or the server did.

It's for this reason that I've stopped publishing bare throughput numbers. A 35B mixture of experts decoding at 112 tokens per second and a 27B dense model decoding at 57 are both true statements, and neither settles which one to use. That comes down to whether you let them think, how long you'll wait, and how badly the task punishes a wrong answer.

None of the eleven settings requires new hardware. They require deciding, in advance, what the server is allowed to change on its own, and on a 16GB card that's nothing. Pin the flags, size the context by loading, pick reasoning on or off per task, use the speculative head when the file has one, and measure with the cache off and one slot. The upcoming articles in this series go deeper into each of these, with the full data. If you are planning a local inference setup for a team and want the configuration validated on your own workload before anybody buys hardware, you can describe the project in the free quote form: it takes two minutes, and I tell you upfront whether it fits my perimeter.