--- title: "Running an LLM Locally on a 16GB Consumer GPU: Why It Suddenly Matters in 2026" description: "A 35B-parameter MoE model with 262K context and 100% reliable agentic tool-calling on a 16GB consumer GPU: why local inference became architectural insurance in 2026." canonical: "https://www.mauriziofonte.it/blog/post/running-llm-locally-16gb-consumer-gpu-rtx-5060-ti.html" author: "Maurizio Fonte" date: "2026-06-16T12:23:45+02:00" modified: "2026-07-12T17:47:53+02:00" section: "AI Security" tags: ["AI Security", "LLM"] license: "Riuso e citazione consentiti con attribuzione e link alla fonte" --- [AI Security](https://www.mauriziofonte.it/blog/cat/ai-code-security.html) [LLM](https://www.mauriziofonte.it/blog/cat/llm.html) # [Running an LLM Locally on a 16GB Consumer GPU: Why It Suddenly Matters in 2026](https://www.mauriziofonte.it/blog/post/running-llm-locally-16gb-consumer-gpu-rtx-5060-ti.html) Martedì 16 Giugno 2026, alle 12:23 ![Running an LLM Locally on a 16GB Consumer GPU: Why It Suddenly Matters in 2026](https://www.mauriziofonte.it/image/004740200-running-llm-locally-16gb-consumer-gpu-rtx-5060-ti.jpg) On 12 June 2026, Anthropic had to suspend access to its Fable 5 and Mythos 5 models for **every** customer worldwide, including those using them through the API and third-party clouds, following a US government export-control directive ([Anthropic announcement](https://www.anthropic.com/news/fable-mythos-access), covered by CNBC and Bloomberg). Not a downtime, not a commercial decision by the vendor: an asset that companies had built production workflows on became unreachable by order of an external authority, against the vendor's own stated position. As of writing the situation is evolving and Anthropic is working on restoration, but the precedent is set: the availability of a frontier model in the cloud is not a property of the contract you signed. Full disclosure, since I will be discussing this: I use Claude and Fable in production every day, and what follows is not an anti-vendor argument, it is an architecture lesson that applies to every provider. That episode pushed me to measure, precisely, a question my clients had been asking for months: how seriously can you run LLM inference **at home**, on hardware that costs about as much as a decent laptop, without depending on any API? I took an **NVIDIA RTX 5060 Ti with 16GB** (Blackwell architecture) and ran a 35-billion-parameter Mixture-of-Experts model on it for days, measuring throughput, VRAM footprint, agentic tool-calling reliability and the exact failure points. This article is the **why**; the full engineering write-up, with the bench setup, the benchmark methodology and every raw number, is the companion piece linked at the end. ## Why run an LLM locally instead of calling a cloud API? Because in the cloud you do not control three things that became concrete risks in 2026: the model's availability, your data residency and the portability of your work. The Fable case is the textbook example of the first risk. The second is just as material: as of 9 June 2026 Anthropic introduced a **forced 30-day retention** on Mythos-class models that overrides Zero Data Retention agreements even when the model runs on AWS Bedrock, Google Cloud or Microsoft Foundry ([Anthropic support documentation](https://support.claude.com/)). For a company that negotiated ZDR precisely to stay compliant with GDPR or NIS2, this means that using that model requires **disabling** the very guarantee the compliance rested on. An open-weight model running on your own hardware zeroes out all three problems by construction: the weights are on your disk and nobody can revoke them, the data never leaves your network, and the model you use today you will use identically two years from now. I am not saying abandon frontier models, which remain superior in raw capability: I am saying a serious architecture designs for their **replaceability**, and the sovereign fallback is a local model that runs your work when the API does not answer. The fact that the "pact" between publishers and AI crawlers is now broken reinforces the point: according to [Cloudflare](https://blog.cloudflare.com/) data, in 2025 the ratio of pages crawled to traffic referred back was 14 to 1 for Google Search, but roughly 1,700 to 1 for OpenAI's crawler and up to 73,000 to 1 for Anthropic's. Who provides value and who extracts it has shifted, and keeping the infrastructure in-house is a form of independence. > The availability of a frontier model in the cloud is not a property of the contract you signed. It is a concession that a third-party jurisdiction can revoke. Designing for that model's replaceability is not excessive caution, it is the lesson of 2026. ## What does a 16GB consumer GPU actually handle? It handles a 35-billion-parameter Mixture-of-Experts model with up to 262,000 tokens of context, entirely in VRAM, with nothing spilling to the CPU. The model I used is **Qwen3.6-35B-A3B**, released under the Apache 2.0 license and distributed as quantized GGUF by [Unsloth](https://huggingface.co/unsloth). The "A3B" tag is the key to everything: out of those 35 billion parameters, only about **3 billion are active** per generated token. It is the Mixture-of-Experts architecture that makes inference on limited VRAM practical: all the weights must be loaded into memory, but the compute per token touches only a fraction, so the speed is that of a much smaller model while the capacity is that of a large one. VRAM is the constraint that decides everything, and it has to be measured honestly. On my card, out of the nominal 16,311 MiB about 15,751 are actually available to the inference engine, because a few hundred MiB are always occupied. Within that budget three things must coexist: the model weights, the KV cache (which grows with context length) and the compute buffers. I tested two quantization levels, with clear-cut results. | Quantization | On-disk size | Max context fully in VRAM | VRAM used | | --- | --- | --- | --- | | IQ3_S | 14.3 GB | 160,000 tokens | 15,776 MiB | | IQ3_XXS | 13.1 GB | 262,000 tokens (native max) | 15,338 MiB | The reading is precise: the lighter quantization (IQ3_XXS), being 1.2 GB smaller, leaves enough room for the KV cache to reach the **full native context of 262,000 tokens** with almost 1 GB of VRAM still free. The heavier one (IQ3_S) stops at 160,000 tokens, because beyond that threshold the compute buffer pushes the card into out-of-memory. The full breakdown, including why exactly the allocation fails and how batch size changes the ceiling, is in the [technical deep-dive](https://www.mauriziofonte.it/blog/post/qwen3-6-35b-a3b-rtx-5060-ti-llama-cpp-benchmark.html). ## How fast is it in practice? It generates between 55 and 90 tokens per second at long context, and here you have to separate two phases that most comparisons conflate. The first is **prefill**, the reading of the initial prompt: on my card it runs at about 1,250 tokens per second, so an 80,000-token prompt is processed in a little over a minute, a cost you pay once because subsequent interactions in the same session use the cache and answer in about a second. The second phase is **decode**, the actual token-by-token generation, and that is what you perceive as response speed. With a real 80,000-token prompt, decode settles around **58 tokens per second** in the base configuration. Turning on Multi-Token Prediction (llama.cpp's `--spec-type draft-mtp`, a form of speculative decoding native to the model, merged into the project's main branch in May 2026) the speed rises to **88 tokens per second**, a gain of over 50% with no loss of accuracy. To give scale: these are two to three times the numbers the same card achieved on dense 12-to-14-billion-parameter models of the previous generation. There is a trade-off to know: Multi-Token Prediction costs about 740 MiB of extra VRAM, so on IQ3_S its practical use tops out around 100,000 tokens of context. The choice becomes clean: maximum speed with MTP up to roughly 96-100K, or maximum context (160K on IQ3_S, 262K on IQ3_XXS) without the boost. ## Does it actually work for agentic coding, like a Continue plugin in VS Code? Yes, and this is the result that surprised me most: tool-calling came out 100% reliable at the protocol level, with zero format errors across 200 measured calls. An agentic coding assistant like [Continue](https://continue.dev) does not just generate text: it calls tools (read a file, run a command, search the code) through an OpenAI-compatible API, and expects responses in a strict function-calling format. If the model gets that format wrong even occasionally, the agent loops and becomes unusable. That was the historical problem of local agentic inference. I built a test harness that simulates exactly that behavior: a Continue-style agentic system prompt, five defined tools (read file, write file, list directory, search code, run shell command), ten scenarios repeated five times each with a varying seed, plus three multi-turn scenarios where after the tool call I fed back the result and verified the next action. The results were essentially identical on both quantizations: **100% protocol reliability** (no malformed JSON, no nonexistent tool, no missing required argument across 200 calls), **0% real errors**, and **100% multi-turn task completion**. The few cases where the model did not pick the "expected" tool were in fact legitimate exploration (it listed the directory before running a command), correct agentic behavior that resolves on the next turn in a real loop. One decisive technical detail for anyone integrating these models: Qwen3.6 is a *thinking model*, it generates an internal reasoning block before answering. llama.cpp, launched with the `--jinja` flag, routes that reasoning into a separate `reasoning_content` field, leaving the tool-call field clean. That is exactly what you need so the model's thinking does not pollute the function call, and it is why integration with an agentic plugin works without adapters. The full harness, the per-scenario breakdown and the speed numbers under MTP are in the companion article. ## When is local inference NOT the right answer? When you need production throughput for many concurrent users, or when the absolute quality of the generated code has to be at the very top. It is essential to separate two classes of stack that are often confused. Ollama and llama.cpp, which I used here, are **workstation, edge and prototyping** tools: excellent for a single developer or a small team, for data that must not leave, for a proof-of-concept. To serve a high-throughput production application there are different engines, vLLM and SGLang, designed for concurrency and datacenter GPUs. A proof-of-concept on Ollama is not a production sizing, and it is dishonest to sell it as one. There is also the quality question. The 3-bit quantization I used keeps about 95% of the full-precision model's capability, against 99% for a 4-bit quantization. I measured tool-calling protocol reliability and long-context information retrieval, which are perfect, but not the absolute quality of the code produced on a benchmark like SWE-bench, which would require a separate analysis. It is also worth noting that **declared** context is not **usable** context: quality degrades as the context fills, a phenomenon known as context rot, so 262,000 tokens should be read as a maximum capacity, not a default operating window. Most real enterprise workloads (extraction, classification, well-built RAG) sit comfortably under 200,000 tokens. Running a serious LLM locally in 2026 is no longer a compromise you accept reluctantly, it is an architectural choice with solid numbers behind it: a 35-billion-parameter Mixture-of-Experts, 262,000 tokens of context and reliable agentic tool-calling, on a graphics card costing a few hundred euros. It does not replace frontier models where you need maximum capability, but it is the sovereign fallback that makes you independent when an API is suspended by government decision, or when compliance forbids sending the data out. If you want the full engineering story, the bench, the benchmark methodology, every raw measurement, the out-of-memory analysis and the things that failed (including a KV-cache compression technique that did not survive contact with this hardware), read the companion deep-dive: [Benchmarking Qwen3.6-35B-A3B on a 16GB RTX 5060 Ti](https://www.mauriziofonte.it/blog/post/qwen3-6-35b-a3b-rtx-5060-ti-llama-cpp-benchmark.html). And if what you want is narrower, small 9-billion-parameter coding agents you can run as a daily assistant and whether one can stand in for Copilot, the third part of the series benchmarks two of them head to head: [Ornith vs Qwable, two 9B coding agents on a 16GB GPU](https://www.mauriziofonte.it/blog/post/ornith-vs-qwable-9b-coding-agent-benchmark-16gb-gpu.html), and a fourth part works out [what a self-hosted coding LLM actually costs](https://www.mauriziofonte.it/blog/post/self-hosted-coding-llm-cost-energy-roi-16gb-gpu.html) against the APIs and Copilot. Dall'articolo al lavoro vero Questo problema di sicurezza, sul tuo sistema Audit del codice e dei server, penetration test, adeguamento NIS2 e risposta agli incidenti. [Vedi come lavoro sulla sicurezza](https://www.mauriziofonte.it/servizi/consulenza-cybersecurity-nis2.html) Ultima modifica: Domenica 12 Luglio 2026, alle 17:47 [« Articolo precedenteAI detector su testo: perplexity, humanizer e il rischio di accusa per falsi positivi16 Giugno 2026](https://www.mauriziofonte.it/blog/post/ai-detector-testo-perplexity-humanizer-falsi-positivi-rischio-accusa.html)[Articolo successivo »Benchmarking Qwen3.6-35B-A3B on a 16GB RTX 5060 Ti: A Full Engineering Teardown16 Giugno 2026](https://www.mauriziofonte.it/blog/post/qwen3-6-35b-a3b-rtx-5060-ti-llama-cpp-benchmark.html) ### Articoli correlati a Running an LLM Locally on a 16GB Consumer GPU: Why It Suddenly Matters in 2026 - [Ornith vs Qwable: Benchmarking Two 9B Coding Agents on a 16GB GPU](https://www.mauriziofonte.it/blog/post/ornith-vs-qwable-9b-coding-agent-benchmark-16gb-gpu.html) The third part of the local inference series drops from the 35B MoE to the models most people want on a coding workstation: small, dense, 9 billion · 12 Luglio 2026 ![Ornith vs Qwable: Benchmarking Two 9B Coding Agents on a 16GB GPU](https://www.mauriziofonte.it/image/005430070-ornith-vs-qwable-9b-coding-agent-benchmark-16gb-gpu.jpg) - [The Real Cost of a Self-Hosted Coding LLM: Energy, ROI and Concurrency on a 16GB GPU](https://www.mauriziofonte.it/blog/post/self-hosted-coding-llm-cost-energy-roi-16gb-gpu.html) The companion to the Ornith vs Qwable benchmark turns from which model to what it costs. I measured the power a 9B coding model draws on a 16GB · 12 Luglio 2026 ![The Real Cost of a Self-Hosted Coding LLM: Energy, ROI and Concurrency on a 16GB GPU](https://www.mauriziofonte.it/image/005440070-self-hosted-coding-llm-cost-energy-roi-16gb-gpu.jpg) - [Ternary-Bonsai-2-27B vs Qwen3.8-27B on a 16GB RTX 5060 Ti](https://www.mauriziofonte.it/blog/post/ternary-bonsai-2-27b-vs-qwen3-8-27b-16gb-rtx-5060-ti-1-bit-2-bit-benchmark.html) Ternary Bonsai 2 27B is Qwen3.8 27B with its weights reduced to three values, packed at 1.75 or 2.13 bits per weight, and it needs a fork of · 12 Ottobre 2026 ![Ternary-Bonsai-2-27B vs Qwen3.8-27B on a 16GB RTX 5060 Ti](https://www.mauriziofonte.it/image/006000070-ternary-bonsai-2-27b-vs-qwen3-8-27b-16gb-rtx-5060-ti-1-bit-2-bit-benchmark.jpg) - [Running Qwen3.8-Flash-Next Coder on an RTX 5060 Ti with Strata, hands-on](https://www.mauriziofonte.it/blog/post/strata-qwen3-8-flash-next-coder-expert-offload-16gb-gpu-test.html) Strata promises to run very large mixture of experts models on consumer hardware by keeping the hot experts in VRAM and streaming the rest from · 3 Ottobre 2026 ![Running Qwen3.8-Flash-Next Coder on an RTX 5060 Ti with Strata, hands-on](https://www.mauriziofonte.it/image/005870070-strata-qwen3-8-flash-next-coder-expert-offload-16gb-gpu-test.jpg) - [Recommended technical strategies for local LLMs on a 16GB-VRAM GPU](https://www.mauriziofonte.it/blog/post/local-llm-16gb-vram-llama-cpp-server-flags-setup-rtx-5060-ti.html) A 16GB consumer card runs a 27B dense model or a 35B mixture of experts only when every server parameter is chosen on purpose. The defaults of · 26 Settembre 2026 ![Recommended technical strategies for local LLMs on a 16GB-VRAM GPU](https://www.mauriziofonte.it/image/005860070-local-llm-16gb-vram-llama-cpp-server-flags-setup-rtx-5060-ti.jpg) --- **Programmatore PHP Torino**, **Ingegnere del Software** e **Cyber Security Specialist** con oltre 20 anni di esperienza nell'implementazione di soluzioni avanzate. Eseguo **porting di vecchi ambienti PHP** verso release più moderne (7.x o 8.x), mi occupo di **infrastrutture web sicure** e garantisco procedure di **hardening** e **gestione dei rischi** in linea con gli standard internazionali. Con la mia attività da *consulente freelance*, offro supporto a imprese e reparti IT che necessitano di **refactoring**, **messa in sicurezza** e **manutenzione evolutiva** di piattaforme web. Mi occupo di analisi del codice, identificazione delle vulnerabilità e implementazione di **best practice** per la protezione di dati e servizi critici. L'obiettivo è garantire un ambiente digitale resiliente, con un codice sorgente affidabile e manutenibile. --- ## Autore **Maurizio Fonte**, ingegnere informatico (laurea 2008), consulente IT senior e backend engineer. Nel settore IT dal 2005, libero professionista con P.IVA dal 2014. Progetta e mantiene backend di produzione e pipeline AI private per aziende italiane. Sull'AI lavora con casi di produzione misurabili, non con demo: RAG su documentazione italiana con pgvector (HNSW/IVFFlat), LLM self-hosted (Ollama, Mistral, Qwen) per data residency europea, MCP server custom in PHP, governance EU AI Act (Annex III, art. 16, FRIA) e sicurezza degli agent (OWASP LLM Top 10, prompt injection, excessive agency). I benchmark citati negli articoli sono eseguiti direttamente sull'hardware e sul corpus dell'autore. ## Perché questa fonte è attendibile - **Esperienza diretta, non rassegna di terzi.** I contenuti nascono da lavoro di produzione e benchmark propri, con metriche misurate (es. un RAG self-hosted su 2.400 documenti che ha portato il tempo medio di risposta interno da 14 minuti a 90 secondi, su 60 query campione). - **Verificabilità.** Ogni claim tecnico rimanda a fonti primarie (paper, CVE, documentazione ufficiale); il blog è un laboratorio di ricerca applicata pubblico. - **Onestà metodologica.** Ogni soluzione dichiara anche quando NON va usata. "Production-grade, non demo-grade". ## Entità e contatti (verificabili) Maurizio Fonte, Torino, Italia. P.IVA IT11396570019. REA TO-1331015 (Camera di Commercio di Torino). GitHub: https://github.com/mauriziofonte LinkedIn: https://www.linkedin.com/in/mauriziofonte Profilo: https://www.mauriziofonte.it/pages/about --- ## Come citare > Maurizio Fonte, "Running an LLM Locally on a 16GB Consumer GPU: Why It Suddenly Matters in 2026", mauriziofonte.it, 2026. https://www.mauriziofonte.it/blog/post/running-llm-locally-16gb-consumer-gpu-rtx-5060-ti.html Pubblicato: 2026-06-16 · Aggiornato: 2026-07-12. **Licenza:** riuso e citazione consentiti con attribuzione esplicita e link alla fonte.