The real cost of keeping a 16GB-VRAM GPU dedicated to local LLMs

The price of a local inference setup is usually discussed as the price of the card, and then as an abstract "electricity is cheap". Both are incomplete. A card that serves a model all day is dedicated hardware: the model sits in its memory whether you send a request or not, the card can't run a game or a render without unloading it, and switching to another model costs time on each switch. The electricity is indeed cheap, but how cheap changes a lot if you count the card, the whole machine, the hours the model waits or the hours it works. I measured all of these on my bench, a 16GB RTX 5060 Ti in a small server that also runs other services, serving the two models I use every day: Qwen3.8-27B in the GSQ-RCO build with its speculative decoding head, and Ornith-1.5-35B-A3B, a mixture of experts. This article lays the costs side by side, separating what I measured from what I estimated, because the two aren't the same kind of number and should never be quoted as if they were.

What does it cost to keep a model loaded and waiting?

At the card, 15 to 16 W with a 27B or 35B model resident and no request in flight. I measured it with the model loaded at its production context of 131,072 tokens, holding the card idle for sixty seconds and reading the power draw:

Three numbers go with that idle draw. The 27B with its MTP head occupies 15,450 MiB of the card's 16,311, the 35B 15,246 MiB: in both cases about one gigabyte is left, not enough for anything else that uses the GPU seriously. The cold load, from starting the server to the first answer, took 14.0 seconds for the 27B and 17.1 for the 35B, reading from an NVMe drive (one measurement per model, and a second read of the same file may come from the operating system's cache). A swap from one to the other cost 9.6 seconds, which is effectively a full load: on a single card there is no such thing as keeping two of these models warm.

The MTP head costs 1,296 MiB of VRAM on its own, the difference between the two GSQ-RCO rows of memory. That is the price of nearly doubling the decoding speed on code; at 131,072 tokens of context the model still fits, with about 860 MiB to spare.

On a 16GB card the model doesn't share the GPU, it owns it. Fifteen gigabytes of sixteen are taken while it waits for your next request.

If you are designing a local inference setup for a team and want the residency and cost trade-offs measured on your own hardware, my hub on AI infrastructure for companies describes the approach I use for self-hosted inference.

The card is not the machine

The 15 W at the card are measured: NVIDIA's own energy counter, read by the driver. They aren't what your electricity meter sees. The meter sees the whole machine: CPU, RAM, disks, fans, power supply losses, and in my case a few other services that run on the same box.

For the whole machine I use an estimator that runs on the server every thirty seconds, combining the measured GPU power with a model of the CPU, a constant for the motherboard and RAM, and an estimated curve of the power supply's efficiency. It's an estimate with a declared 95% band, not a reading from a wattmeter, and I quote it only as such.

With the inference server running and nothing to do, the whole machine draws about 56 W, with a band from 44 to 68 W. Of those, 9.6 W are the GPU itself in that idle state. The rest is everything else a small server does when it's on. That baseline is the reference I subtract when I want the cost of the inference work alone, because the machine draws it anyway.

Every cost calculation has to pick one of the two. If the box exists anyway, as a home server or an office machine that is on all day, the inference cost is the energy above that baseline. If the box exists only to serve the model, the whole baseline is part of the cost.

How much did forty-five hours of full load cost?

I ran a measurement campaign of 645 tasks across five models for 45 hours, with the card busy most of that time. Above the idle baseline, the campaign used an estimated 8.86 kWh, with a band from 8.31 to 9.41 kWh, which at the marginal price I pay comes to 2.89 euros. The gross energy, baseline included, was about 11.4 kWh.

Of the marginal 8.86 kWh, 6.94 kWh are GPU energy measured directly by the driver; the rest is CPU, disks and power supply losses, estimated. The price is the marginal per-kWh tariff of my supply, 0.3266 euros including taxes and excluding fixed charges, which is the right price for energy you add on top of a connection you already pay for.

To put the campaign in a monthly frame: at the campaign's average, about 0.2 kW above baseline, a card working eight hours a day for twenty-one working days would use roughly 33 kWh, around 11 euros a month at that price. A card that waits for most of the day costs far less. Fifteen watts of idle draw for eight hours a day over twenty-one days is about 2.5 kWh, well under a euro a month.

Where the energy actually goes: per solved task

A cost per unit of work says more than a total. I divided the marginal energy of each model's block of tasks by the number of tasks it solved, separately for each kind of work.

The spread is the story. Without reasoning, a solved coding problem costs from half a watt-hour to five. Turn reasoning on and the same kind of problem costs 18 to 118 Wh, because the model writes six to thirteen thousand tokens, almost all of them thinking, for every answer it completes. An agentic bug fix on a small real project, with dozens of tool calls, costs 19 to 133 Wh, or 0.6 to 4.4 cents at my price.

Two things stand out. Reasoning multiplies the energy per solved problem by ten to sixty, depending on the model, which is the electrical face of the same tax I measure in waiting time. And the mixture of experts is consistently the cheapest per solved problem when it solves the problem, because it writes at twice the speed of the dense 27B for a lower power draw.

What a model swap costs in a working day

On a 16GB card you serve one of these models at a time. If your day mixes tasks that suit different models, every switch costs a load. With my two production models the swap is 9.6 seconds, and the first request after it also pays the prefill of whatever context you send, because the KV cache of the previous model is gone.

Ten switches a day cost under two minutes of waiting. I can accept that. The annoying part is elsewhere: a tool that asks for a model by name triggers a load nobody expected, and a request sent in the middle of a load waits until it ends. In my production setup the model loader keeps one model resident and swaps on demand, and as a result I stopped switching casually. I pick the model for the session, not for the request.

The costs nobody puts in the spreadsheet

Some costs are real and not electrical.

The card isn't available for anything else. With fifteen of sixteen gigabytes taken, a second GPU workload, even a light one, either fails or forces the model out. On my server that means any GPU job, photo indexing or a quick render, has to wait or run on the CPU. If you planned to share the card with a desktop session, plan again.

The machine is never really idle. A server that holds a model and waits for requests keeps its RAM, its fans and its power supply in an active state. The 56 W baseline is the price of availability, and it's paid every hour of every day the box is on.

The noise is real, and I can live with it. In a five-minute generation test at full load the dense 27B held the card at its 180 W limit with the fan at 98%, the 35B MoE at 144 W with the fan at 76%. The case fans are off by choice, and over the long campaign the card never went above 81 °C. In forty-five hours of continuous load the card never throttled.

Time is the largest cost of all. A reasoning model that thinks for three minutes on a simple question costs a fraction of a cent in electricity and three minutes of your attention. The reasoning tax article in this series puts numbers on that cost; in money it dwarfs the power bill.

How these numbers were produced, and what they are not

The card-level numbers come from the NVIDIA driver and are measurements. The whole-machine numbers come from an estimator with a declared band, calibrated on the same hardware but without a wattmeter at the wall: they are estimates, and every figure derived from them is one too. The per-task energy subtracts the idle baseline over the exact time windows of each task, recorded with the measurements. The price is my own marginal tariff and won't be yours; the kWh figures are the portable part.

The NVIDIA driver documents the power fields it exposes in its nvidia-smi reference, and the inference engine is the upstream llama.cpp server. The configuration under which all of this was measured, from pinned flags to the KV cache type, is described in the recommended strategies for local LLMs on a 16GB card, and the hardware rationale for this class of card is in why a 16GB consumer GPU suddenly matters.

Energy by model over the whole campaign

The per-task figures come from a larger accounting: the marginal energy of each model's entire block of work during the 45-hour campaign, from the first task to the last, with the idle baseline subtracted over the same window. The blocks differ in length because slower models need more hours for the same set of tasks.

The table explains a result that surprises people: the 9B models aren't the cheap option. They ran longer than the 35B mixture of experts on the same task set, because they think more slowly and fail into long loops more often, and they ended up costing one and a half to two times as much energy for fewer solved tasks. The uniform IQ4_XS build of the 27B is the most expensive of all: it's the slowest decoder on the card, so it stays busy longest.

How much to trust the estimated numbers

Every whole-machine number in this article comes with a band, and two separate error sources feed it. One is the model of the components that aren't measured directly: CPU power is modelled from its counters, motherboard and RAM are a constant, and the efficiency curve of the power supply is an estimate, never characterized on a bench. Then there's the baseline: it's an estimate too, with the same error sources, so the marginal figure gets a band computed on both.

The two errors partly cancel. Because the baseline and the loaded measurement use the same coefficients and the same supply curve, a systematic error in the model tends to shift both in the same direction, and the marginal energy is more reliable than either absolute value. The correlated band for the campaign, 8.31 to 9.41 kWh, rests on that assumption. If you take the two errors as independent, the prudent band grows to 7.24 to 10.48 kWh. Neither includes the uncertainty of the price.

What I can say without any estimator is the GPU part. The driver's energy counter on this card agreed with an independent once-per-second power sampler to within 0.1% to 0.3% per model, and 6.94 of the 8.86 marginal kWh were GPU energy measured that way. Whatever error the estimator has, it lives in the remaining 1.9 kWh. The only way to remove the word "estimated" is a wattmeter at the wall, which costs money I chose not to spend; the band travels with every number instead.

Is a dedicated card worth it?

On electricity alone, the running cost is almost a rounding error; the real cost is the card itself, the dedicated machine and the time you wait for answers. Forty-five hours of full load cost less than three euros. A card waiting eight hours a day costs under a euro a month at the card; the whole machine idling around the clock, other services included, is about 40 kWh and 13 euros a month. On their own, these figures decide nothing.

What decides is whether you can accept a GPU that belongs to the model, a machine that stays on, and the waiting time of the model you choose. For a developer who would otherwise send code and documents to an external API, the electricity is irrelevant and the trade is about privacy, availability and speed. For a team that already has a server running all day, the marginal cost of adding inference is the card and a few kilowatt-hours.

The comparison with paid APIs, where the card's purchase price finally becomes the dominant term, is the subject of a later article in this series. In short, the power bill is the smallest line in the budget, the dedicated card is the largest, and your waiting time sits between them in a currency the spreadsheet doesn't have. If you are deciding whether a dedicated inference box makes sense for your team and want the numbers measured on your own workload, you can describe the case in the free quote form: it takes two minutes, and I tell you whether it's something I can help with.