Running Qwen3.8-Flash-Next Coder on an RTX 5060 Ti with Strata, hands-on

When a new inference engine claims that a model with over a hundred billion parameters runs on a gaming card, the claim is usually true and the way it's told is usually misleading. Strata is the most interesting of these projects I've tried in a while: an engine keeping the most frequently used experts of a mixture-of-experts model resident in VRAM, copies the rest into system RAM, and computes the cold experts on the CPU when the GPU cache misses. I installed it on the same machine that runs my local inference benchmark, a Ryzen 5 4500 with 32 GB of DDR4 and a 16GB RTX 5060 Ti, and put it against the two models I actually use every day on that card. Read this as a hands-on report; my harness would not call it a benchmark: Strata runs outside it, with its own server, and I took the measurements with dedicated scripts, without condition-hashed records. I state that upfront because the rest of this series is held to a stricter standard, and the numbers here deserve the same caution. They are still my measurements, on my hardware, with every protocol written down, and they answer a question anyone with a 16GB card will ask: is a huge sparse model on a small GPU better than a mid-size one that fits?

What exactly ran on the card

The base model is Qwen3.8-Flash-Next, released on Hugging Face in August 2026. Its official model card describes it as 125B parameters with 6B active, plus a 51B n-gram embedding and a 4B multi-token prediction layer, with 512 experts per layer and 10 routed plus 1 shared active per token. That is not what Strata runs on a 32 GB machine. The README recommends, for that amount of RAM, the Coder build pruned by ISTA-DASLab, which keeps 256 of the 512 experts per layer, chosen by minimizing the divergence from the full model on code. The ISTA-DASLab card lists the compressed model at 117B parameters, and the IQ1_M quantization I used weighs 58.4 GB across two files.

The model that ran is a pruned, code-specialized, 1.75-bit build of a large MoE, and most of what follows comes from that: the parts where Strata shines and the parts where it collapses.

Strata ran in its low-RAM mode with resident experts: about 45% of the experts (4,511 of them, 8.59 GiB) in VRAM, the rest copied into RAM, a 65,536-token context with an 8-bit KV cache, and its own speculative decoding with four draft tokens. The installer writes an expert file of 23.4 GB on top of the model, so the disk space needed was about 89 GB rather than the 70 the documentation mentioned. On Debian the setup script refuses to install the CUDA 13 toolkit it wants, so I built the engine against the system CUDA.

The two opponents were served by llama.cpp exactly as my production setup serves them: Ornith-1.5-35B-A3B at IQ3_XXS and Qwen3.8-27B in the GSQ-RCO build with its MTP head, both with 131K of context and a 4-bit KV cache, both fully in VRAM.

How fast does a 117B model decode on a 16GB card?

Strata writes at a third of the speed of the Ornith 35B and two thirds of the dense 27B, but it reads long prompts as fast as the MoE and twice as fast as the 27B. The decode median was 38.2 tokens per second against 115.2 and 56.7. Prefill is the surprise: on a 29,600-token prompt Strata processed 1,581 tokens per second, the Ornith MoE 1,475 and the 27B 742.

The decode speed of Strata depends on the content: 46 tokens per second on code, 36 on prose. Its speculative decoding guesses code better than English, which is the expected behaviour for a draft model and a good reason not to judge it from one prompt type. The GPU expert cache hit rate over the whole session was 77%: almost a quarter of the expert computations fell back to the CPU, and the speed goes with them.

There is one more number that the throughput table hides. A cold start of Strata took 301 seconds before the server answered, against about twenty seconds for llama.cpp to swap one of the other two models in. If you switch models during a working day, that is five minutes each time.

Thinking at the default effort turns seconds into minutes

With reasoning at its default level, Strata spent a median of 189 seconds per answer on my eight latency probes, the Ornith MoE 16 seconds and the dense 27B 53. A question about the difference between == and === in PHP cost Strata 6,058 generated tokens; a code review probe hit the 16,384-token ceiling and never produced an answer.

A truncated answer is a missing answer, however fast it came. On reasoning models the token budget is part of the configuration, not a detail.

To judge if a local setup is usable day to day, look at this number first. Another article in this series covers the reasoning tax in detail; for Strata the short version is that its thinking mode, at the default effort, isn't compatible with interactive use on this hardware.

If you want to see how I carry this sort of evaluation into real infrastructure choices, the articles in my hub on AI infrastructure for companies apply the same measured approach to self-hosted inference.

Fifty general questions with thinking off

The second test removed reasoning from the equation: fifty open questions, thirty in English and twenty in Italian, on science, reasoning, history, law, economics and computing, each answered in about 120 words at temperature zero. The answers were graded in a single pass by an LLM judge (Claude), with the three answers to each question side by side and a written reason for every grade. So this part is one reviewer's assessment, with no automated score behind it, and the judge could see which model wrote what.

The dense 27B answered 47 of 50 completely correctly, the Ornith MoE 43, Strata 32. The paired comparison on the same questions puts Strata behind both (3 wins against 14 losses with the MoE, p = 0.013; none against 15 with the 27B, p = 0.0001), while the difference between the other two is not significant (1 against 5, p = 0.22).

Anyone writing to Italian users should read the Italian column first. Strata answered all twenty Italian questions in English or in a mix of the two languages, made factual errors on Italian topics (a Constitution described as not amendable, Michelangelo given the surname Buonaparte, a false premise about Garibaldi left unchallenged), and twice reasoned aloud inside the answer, correcting itself mid-sentence. On English computing and reasoning questions it was on par with the others, and once it was right where the MoE was wrong. It fits a model that kept half of its experts and chose them on code: the pruning is visible as soon as the question leaves English and leaves programming.

Fifty hard Python problems: here the pruning pays off

The third test was the one Strata was made for. Fifty Python problems written for this purpose, with 560 executable assertions in total, covering things models routinely get wrong: mojibake and byte order marks, grapheme clusters, UTF-16, RFC 2047 headers, Vixie cron semantics, daylight saving transitions, Italian holidays computed from Easter, IBAN and tax code validation, path traversal, a true Damerau distance, window functions in SQL, and six bug fixes whose comments point the wrong way on purpose. A problem counts as solved only when every assertion passes. Thinking was off for all three models.

Strata Coder solved 32 problems, the dense 27B 34, the Ornith MoE 19. Paired on the same problems, Strata beats the MoE (17 wins, 4 losses, p = 0.007) and is indistinguishable from the 27B (7 against 9, p = 0.80).

Two caveats keep this result in proportion. First, Strata was the slowest of the three: 14.1 minutes for the fifty problems against 10.3 for the 27B and 4.0 for the MoE, and on the fourteen problems everyone solved its median time was 7.6 seconds against 6.2 and 2.7. Second, it twice degenerated into a loop, rewriting the same wrong regular expression in a comment until the 4,096-token limit, something neither llama.cpp model did on this bank.

The third caveat came a day later, and it weighs most. I ran the coding test again after raising the RAM from 2133 to 3600 MT/s. Strata solved 28 instead of 32, and only 4 of the 50 answers were identical between the two runs at temperature zero, while all fifty general answers were identical. The paired difference between the two runs isn't significant (p = 0.22), yet it's enough to show that one Strata coding score can move by a few problems out of fifty, and the "Strata beats Ornith" result (p = 0.007 in the first run) dropped to p = 0.064 in the second. I read it as not confirmed.

On this engine, temperature zero isn't deterministic. When half the experts are computed on the CPU and half on the GPU, near-tied choices can flip between runs.

My hypothesis, not verified, is that the content of the GPU expert cache depends on the session history, and an expert computed on the CPU produces slightly different logits from the same expert on the GPU.

What the machine looks like while Strata works

Where Strata really costs you is resources. While generating, Strata used 15,650 to 15,738 MiB of VRAM, 24 to 25.6 GB of system RAM out of 31, and pushed between 2.4 and 3.0 GB into swap. With Strata running, the box can't host anything else: no second model, no other GPU service.

The GPU itself works less. Over five minutes of continuous generation it drew 109 W against 144 W for the MoE and 178 W for the 27B, its fan ran at 53% against 76% and 98%, and it stayed at 67 °C. The work moved to the CPU, which ran at 44% across twelve threads with six cores above half load and reached 81 °C, and to the PCIe bus, which I cover in the companion article of this series because it turned out to be the most interesting finding of the whole test.

What about the claim of being 2.5 times faster than llama.cpp?

I didn't measure it, and the comparison I ran doesn't contradict it. That claim is plausible when you compare Strata against llama.cpp on the same 117B model with the experts offloaded to the CPU through --n-cpu-moe, not against smaller models that fit completely in VRAM. Both statements hold together: Strata can be much faster than llama.cpp on the same giant model and still be slower than a smaller model that never leaves the card. Verifying the first would require running llama.cpp on the same GGUF with expert offload, which I haven't done.

That is the lesson to take home. With a 16GB card, almost nobody needs to know "which engine runs this one huge model fastest". What they need is "which model gives me the best answers in the time I'm willing to wait", and on that question a mid-size model that fits has a structural advantage. My earlier engineering teardown of the Qwen3.6 35B MoE on this card shows how much a model gains when nothing has to cross the bus.

Where Strata makes sense, and where it does not

On this hardware, with this model, I would use Strata for one thing: batch code generation in English where a slower answer is acceptable and a second opinion from a far bigger expert pool justifies the cost. It matched a dense 27B on hard Python problems, which I didn't expect from a model that doesn't fit in VRAM.

I would not use it as a daily assistant. It answers Italian questions in English, at its default reasoning effort a chat question waits for minutes, a cold start takes five minutes, and while it runs the machine has no memory left for anything else. Its results are also noisier than those of llama.cpp at the same temperature, a problem if you use a model to compare or evaluate anything.

I'll carry one point into the rest of this series: size on paper is a weak predictor of what you get on a small card. The general strategies for 16GB cards apply here too: measure the configuration you will actually run, on the tasks you will actually give it, more than once. A huge pruned model streamed through system RAM and a mid-size model held wholly in VRAM are different tools with different failure modes, and the benchmark that decides between them is your own workload. If you are weighing a local setup like this for a team and want it tested on your real tasks before committing to hardware, you can describe the case in the free quote form: two minutes, and I tell you whether it fits what I do.