Ternary-Bonsai-2-27B vs Qwen3.8-27B on a 16GB RTX 5060 Ti

Every few months a new compression scheme promises to fit a large model into a small card with little loss, and the claim is always half right. Ternary-Bonsai-2-27B is one of the more interesting recent cases, because it isn't a new model trained from scratch: according to its model card, it is "derived from Qwen3.8-27B", with the architecture unchanged and the weights reduced to ternary values. That makes it directly comparable to the conventional GGUF quantizations of the same model, which I had already measured on my 16GB RTX 5060 Ti at 4, 3 and 2 bits. In late September I ran both ternary packings, PQ2_0 at 2.13 bits per weight and PTQ1_0 at 1.75, on the same card, with the same deterministic bank and the same engine benchmark. The comparison has one structural limit that I state before any number, because it decides how every number must be read: the ternary files need a fork of llama.cpp, so they ran on another engine, while the conventional quants ran on upstream llama.cpp. That rules out the paired statistical test I use everywhere else in this series. What remains is a careful side-by-side, and it is still informative.

What exactly is being compared

Qwen3.8-27B is the dense model Alibaba released on Hugging Face in August 2026 under Apache 2.0. The conventional quantizations I measured come from a community packager: IQ4_XS, Q3_K_M and Q2_K, between roughly 4.3 and 2.6 bits per weight, all running on upstream llama.cpp. The ternary versions store each weight as one of three values. PQ2_0 places each ternary digit in a two-bit slot, 7.21 GB on disk; PTQ1_0 packs them densely at 1.75 bits per weight, 5.95 GB. Both require the PrismML fork of llama.cpp: the card states plainly that stock llama.cpp will not run these files.

All five ran with the same server parameters: every flag pinned, automatic fitting disabled, a 4-bit KV cache, flash attention, one slot, temperature zero, and reasoning off, which was the default of my servers in that period.

Why is there no winner in this article?

Because the two groups ran on different engines. My comparisons rely on a condition hash: every measurement records the engine build, the model file, every flag and the sampler, and two measurements are compared with a paired test only when they share everything except the variable under study. The ternary files ran on the fork, built from a different commit; the conventional quants ran on upstream llama.cpp. A difference in results could come from the weights, from the fork's kernels, or from changes between the two engine commits, and once the runs are done, statistics can't pull those causes apart.

When two measurements differ in more than one thing, a statistical test tells you that they differ, never why. So I report them side by side and refuse the ranking.

So this is a descriptive comparison. The numbers are measurements; the conclusions are bounded by the engine difference, and I keep that bound visible in every section.

To have aggressive quantization tested on your own tasks with the engine variable controlled, my hub on AI infrastructure for companies explains how I set up those comparisons.

Size: a 27B model in six gigabytes

The most dramatic result needs no statistics. A 27B model at 1.75 bits per weight is a 6 GB file, and on the card the PTQ1_0 build occupied 6,030 MiB in that run, the PQ2_0 build 7,174 MiB. The conventional IQ4_XS build of the same model needs most of the card: with a 65,536-token context it occupied 15,744 MiB.

That changes what a 16GB card can do with a 27B model. With a ternary build, a long context, a second model or an embedding service still fits next to it. A 4-bit build leaves space for little else. On an 8GB card, where the conventional quants of a 27B don't fit at all, the ternary builds do.

The VRAM figures were not taken at the same context size for all builds, so read them as an order of magnitude, not as a precise ratio.

Speed: almost twice the decode, very different prefill

On the engine benchmark, at empty context and at 8K tokens of depth, the ternary builds decode much faster than the conventional quants (the 8K and prefill figures are in the depth table further down), and their prompt processing tells two different stories.

Decoding is where the ternary builds shine: 48 and 44.5 tokens per second against 26 for the 4-bit build. Generation on a card like this is limited by how many bytes of weights must be read for each token, so a file three times smaller decodes faster. The Q2_K build lands in between, as you'd expect from its size.

Prefill is where the two ternary packings part ways. PQ2_0 reads prompts at about 1,020 tokens per second, on par with IQ4_XS. PTQ1_0, the densely packed one, reads at 456, less than half. Unpacking densely stored ternary digits costs compute, and prompt processing is compute-bound, so the smaller file pays for its density every time it reads a long prompt. The 2-bit slot layout of PQ2_0 avoids most of that cost by spending a quarter of a bit more per weight.

The smallest file is not the fastest model. It decodes fast and reads slowly, and on a long prompt the slow reading dominates the wait.

What the bits per weight mean

A ternary weight takes one of three values, worth log2(3), about 1.58 bits of information. Real formats can't store exactly that: values are grouped in blocks, and each block carries a scale factor so that the three levels can be stretched to the range of the original weights. PTQ1_0 packs the ternary digits as densely as the format allows: 1.75 bits per weight including those scales. PQ2_0 gives each digit a two-bit slot, wasting part of a bit per weight, for a total of 2.13 bits per weight. The extra space buys a layout the GPU can unpack with simple bit operations, which is why PQ2_0 reads prompts more than twice as fast as PTQ1_0 while being only 1.26 GB larger.

The conventional quants work differently: they keep many more levels per weight, typically 4 to 16 per block for the 2 to 4 bit formats, with scales and sometimes per-block minimums, and they mix precisions across tensors. That is why a Q2_K file of a 27B model is still larger than PQ2_0 even though both are nominally around two bits: the K-quant formats spend more on scales and keep sensitive tensors at higher precision.

How speed falls as the context fills

Both families slow down as the context grows, because every new token attends to every previous one. The rate of loss is similar across builds, which suggests that the ternary kernels scale with context like the conventional ones:

The ternary builds lose a little more decode speed with depth in relative terms, two to four points more than the conventional quants, but they start so far ahead that they stay well ahead at 8K. Measurements at larger depths were not part of that run, so I make no claim about 64K or 128K.

The fork as an operational dependency

Running the ternary builds meant building and maintaining a second llama.cpp tree on the bench, with its own commit, its own build flags for the card's architecture and its own server binary. Everything in my harness that assumes one engine had to accept a second engine identity, and for this reason the results are kept in separate comparison groups. For a single developer this is an afternoon of work; for a team it's a dependency with its own update cycle, and it should be weighed like any other: who updates it, how quickly it follows upstream fixes, and what happens to your deployment if nobody maintains it anymore.

Time to answer a chat question

With reasoning off, on short chat questions, the median time to a complete answer comes down mostly to decoding speed and answer length. PQ2_0 answers fastest, in a median of 1.85 seconds, followed by Q2_K at 2.63, PTQ1_0 at 3.01, Q3_K_M at 3.24 and IQ4_XS at 3.41. PTQ1_0 is slower than its decode speed suggests, again because of its slower prompt processing; Q2_K is the fastest of the conventional quants. These are short questions with short answers, where every build feels instant; on long prompts the prefill difference would weigh much more.

Quality: 134 deterministic items, side by side

The deterministic bank has 134 items in seven categories, each with one right answer checked automatically: extraction of fields from text, precise instruction following, structured output against a schema, arithmetic, reasoning traps, refusal of unsafe requests, and Italian items.

The ternary builds passed 107 (PQ2_0) and 105 (PTQ1_0). The conventional quants passed 113 (IQ4_XS), 113 (Q3_K_M) and 114 (Q2_K). Seven to nine items separate the two groups, on a bank where, as a later audit showed, close to half of the items are passed by every capable model and cannot separate anyone.

The category view says where the gap is. On reasoning traps, refusals and structured output, the five builds are within one item of each other; on extraction and instruction following, within two or three. The gap lives in arithmetic and Italian. On twelve math items the ternary builds solved 6 and 4, against 7 for IQ4_XS and a surprising 10 for Q2_K. On thirty Italian items they solved 22 and 23, against 26 for IQ4_XS and 24 for Q2_K.

That pattern is plausible for an aggressive compression: arithmetic and a second language are the capabilities that depend most on precise, less frequently exercised weights, and they are the first to blur. But I cannot confirm it with a test, for the reason explained above: the ternary builds ran on a different engine.

One more observation, because it surprises people: among the conventional quants, the 2-bit Q2_K did not fall behind the 4-bit IQ4_XS on this bank. That matches what I found when I measured the full quantization curve of this model on agentic tasks: no pair of quants produced a difference that survived a paired test. Aggressive conventional quantization of a 27B costs less than intuition suggests, at least on short deterministic tasks.

The decision in yes-or-no form

When a ternary build makes sense

When memory, not quality, is the constraint. On an 8GB card the ternary builds are the only way to run this 27B at all, and on a 16GB card there's still space beside them for a second model or a very long context. If the alternative is a much smaller model, a ternary 27B is worth testing on your tasks.

When you need fast decoding and prompts are short. PQ2_0 decodes almost twice as fast as the 4-bit build and reads prompts as fast. For short interactive requests it is the quickest way to get answers from this model on this card.

Not when you need upstream tooling. The fork is a dependency: it lags upstream, it has its own bugs, and every tool that assumes stock llama.cpp may need adapting. On my card it was also a separate engine to build and maintain, which is a real operational cost.

Not, without testing, when arithmetic or Italian matter. Those are the two categories where the ternary builds trailed in my bank, and they're where I'd focus a test before adopting one.

A fair compression comparison needs one more thing. Comparing the ternary builds with the conventional quants properly would require running the conventional quants on the same fork, or the ternary ones on upstream once support lands, so that only the weights differ. Until then, every chart that ranks them mixes two variables. The method behind that caution is in what llama.cpp benchmarks get wrong, the conventional builds of the same model are compared head to head in Qwen3.8-27B GSQ-RCO vs IQ4_XS, and the general configuration is in the recommended strategies for 16GB cards.

How I would test it properly, given the chance

The experiment that would settle the quality question is simple to design and long to run. Build the fork once, run the conventional GGUF quants on it as well as on upstream, and check that their results on the fork match their results on upstream: if they do, the engine variable is neutralised and the ternary builds can be compared to the conventional ones with a paired test on the same items. If they don't, the fork itself changes answers, which is worth knowing before adopting it for anything. Add a few hundred items in the two weak categories, arithmetic and Italian, because the current bank has only twelve and thirty there, too few to confirm a gap of the size observed.

Compressing a model to three values per weight and still getting most deterministic questions right, like its 4-bit parent does, is a serious piece of engineering, and nobody can argue with the size and decoding numbers. On quality, this comparison sketches the outline and stops there, because the one variable that would let it go further wasn't held constant. To have that comparison run on your own tasks with the engine variable controlled, you can describe the case in the free quote form: two minutes, and I tell you upfront whether it's something I can help with.