Ornith-1.0-9B vs Ornith-1.5-9B on a 16GB RTX 5060 Ti

When a model family ships a new version, the release notes describe improvements and most users upgrade without checking. The question I wanted to answer on my 16GB RTX 5060 Ti was simple: if you already run Ornith-1.0-9B for coding and day-to-day questions, does Ornith-1.5-9B do your work better? I measured the two generations side by side in late September on the same card, with the same server flags, at the same 8-bit quantization, on three kinds of task: fourteen families of agentic coding tasks in a tool loop, a deterministic bank of 134 short items across seven categories, and engine and latency probes. The measurements belong to an earlier regime than the rest of this series: at that time my servers ran with reasoning off by default and the system RAM was at 2133 MT/s, so the numbers here compare the two generations with each other, never with the results measured later with reasoning on. What came out is a textbook case of a version bump that changes the model's character more than its capability, and of why a tie in one test doesn't carry over to the others.

Why a generation comparison is a clean experiment

Two versions of the same family, quantized the same way and served the same way, are as close to a controlled experiment as model comparisons get. The architecture class is the same, the size is the same, the tokenizer family is the same, and the card and the engine are identical. Whatever differs is mostly what the authors changed in training. That makes it a good place to ask a sharp question and expect a sharp answer.

The sharp question has a statistical shape: on each item both models see, which one gets it right? Only the items where they disagree carry information, and a paired test on those disagreements tells you whether the difference leans one way more than chance would allow. Pass rates side by side, without pairing, would hide most of what follows.

If you're deciding whether to upgrade a model in production and want the decision based on your own workload, my hub on AI infrastructure for companies describes how I run this kind of controlled comparison.

Round one: agentic coding, an exact tie

The agentic bank at the time had fourteen families of coding tasks, each a small project with a bug to fix or a feature to add, worked on by the model in a loop with file and shell tools. Each model ran each task several times, 48 runs in total per model, and the visible test suite decided success.

Ornith-1.0-9B passed the oracle in 41 runs, Ornith-1.5-9B in 43. Reduced to the fourteen families, the two generations had exactly the same result. They succeeded on the same families and failed on the same families, including the two that neither solved. The metric I had declared in advance for this comparison, the number of edits needed to reach a solution, came out UNDERPOWERED: the design could not have detected a difference of plausible size.

There is a caveat on the bank itself that I state every time I use these numbers. That agentic bank turned out to be saturated for capable models: in a later audit it discriminated between models on only two of its sixteen instances, both already at the ceiling. A tie on a saturated bank is a statement about the bank as much as about the models. Still, on routine agentic work of this difficulty, the upgrade didn't break anything.

Round two: 134 deterministic items, and a hidden difference

The second bank is a set of short items with deterministic answers, checked automatically: extracting fields from text, following precise formatting instructions, producing structured output against a schema, arithmetic, reasoning traps, refusals of unsafe requests, and Italian items that mirror English ones. Temperature zero, one slot, one pass per item.

Overall, Ornith-1.0-9B passed 112 items and Ornith-1.5-9B 102. The older model alone solved 21 items, the newer one 11, and the paired test gives p = 0.11: not significant. Two earlier versions of the same bank, before a handful of items were replaced, gave 111 against 98 and 111 against 101: consistent in direction, never confirmed.

The total hides where the difference lives. Broken down by category, almost all of it is one category:

On instruction following, the older model solved 9 items that the newer one missed, and the newer model solved none that the older one missed (p = 0.004). This is an exploratory finding, one category of seven that I did not single out in advance, but it survives a correction for having looked at seven categories, and the same direction appears in both earlier versions of the bank: 21 against 12 each time, nine items to none.

The instruction-following items look like what people actually ask for: produce exactly three bullet points, answer in fewer than twenty words, put the answer in a specific format, do not use a given word. The newer model tends to answer the question well and ignore the constraint around it.

The other categories point both ways and none is significant on its own. The newer model is slightly better on reasoning traps (4 items to 1) and math (6 to 3), the older one on Italian items (6 to 1) and structured output (2 to 0). Extraction and refusal are identical.

A tie on the total can hide a clear loss in one place and small gains in others. The upgrade didn't make the model weaker; it made it worse at following instructions and slightly better at reasoning.

The nine instructions the newer model broke

The instruction-following difference is easier to believe when you see the items. These are the nine English items that Ornith-1.0-9B passed and Ornith-1.5-9B failed; the reverse list is empty.

None of these topics is hard for a 9B model; both generations know what a systemd timer or a bind mount is. The newer one loses the constraint halfway through the answer. Italian instruction items follow the same pattern, where the older model passed six that the newer one failed, for example describing the KV cache in at most forty words with the word "memoria" exactly twice. The newer model won one Italian instruction item, an explanation of speculative decoding in lowercase letters only, between forty and seventy words.

On arithmetic the newer generation wins: it solved four chained calculations and two modular arithmetic items that the older one missed, against three items in the other direction. On reasoning traps it solved four, including a logic grid and a spatial reasoning item, against one. These categories are small and none of the differences is confirmed, but they are the improvements a release note would mention, and they are real in direction.

Round three: speed and length of answers

On the engine benchmark, both generations decode at essentially the same speed. Prompt reading is another story.

The older model reads prompts 18% to 35% faster depending on context depth, while generation is within 3%. On long prompts you see that difference directly, as a longer wait for the first word.

The second difference is in how much each model writes. On short chat questions with reasoning off, Ornith-1.0-9B answered in a median of 72.5 tokens and 1.7 seconds; Ornith-1.5-9B in 151 tokens and 3.5 seconds. The newer model is more verbose by a factor of two on the same questions, which doubles the wait for the same decode speed. Sometimes the extra words help the reader, sometimes not; on the deterministic bank they didn't produce more correct answers.

Waiting time by kind of request

The latency probes of that period, with reasoning off, show the verbosity difference on every kind of request:

The long-prompt row uses only the first repetition of each probe: in that period the server's prompt cache was on, and later repetitions of the same long prompt skipped the prefill, a measurement defect I describe in what llama.cpp benchmarks get wrong. The time to first token on the long prompts, about 1.5 seconds for both models, is the real prefill cost. Short prompts are unaffected by that defect, because their prefill is a few dozen milliseconds either way.

Round four: long context

The 9B models load contexts beyond 262,144 tokens on this card, so I measured whether these two still use what they load: four retrieval and tracking tasks (common words, multi-key needle, multi-value needle, variable tracking) at 8K, 32K and 128K tokens.

Ornith-1.0-9B passed 4 of 4 at every length; Ornith-1.5-9B passed 3 of 4 at 8K and 4 of 4 at 32K and 128K. Four probes per length can show where a model falls, not a precise rate, and neither falls. Both read a 118,000-token haystack correctly, which is remarkable for a 9B model on a consumer card.

What the reproducibility test added

The older model was also the one I used to measure reproducibility of the whole bench. On 134 deterministic items, run twice on one server slot, 0% of the cells changed between passes; run twice on four parallel slots, 3.7% changed. That result shaped every later measurement in this series, which runs on one slot. It also means the differences between the two generations reported above aren't run-to-run noise: on one slot at temperature zero, a model repeats its answer identically on every run.

The decision in yes-or-no form

Should you upgrade from 1.0 to 1.5?

If your use depends on precise formatting and constraints, keep 1.0 or test carefully before switching. Structured prompts, templated outputs, assistants that must answer in a fixed shape: on those, the older model was clearly better in my measurements, nine items to none.

If your use is agentic coding of routine difficulty, the upgrade is neutral. The two generations tied on every family of my agentic bank. On harder tasks, measured later for the newer model only, Ornith-1.5-9B behaves like any careful 9B: it rarely accepts a false premise, and in an agent loop it still ships illusory fixes, as the comparison between Ornith-1.5-9B and NeoHorse-1-9B shows.

If you care about latency, pick the older model: it reads prompts faster and writes half the words.

A higher version number doesn't tell you which way the model moved. A new generation can be better at one thing and worse at another, and one pass rate on a saturated benchmark reports a tie over the gain and the loss. The earlier comparison of Ornith against Qwable at 9B taught me to look past pass rates; this one taught me to look inside categories. The measurement traps I had to fix along the way are collected in what llama.cpp benchmarks get wrong, and both models are available as GGUF on Hugging Face, the newer one on the official Ornith-1.5-9B page.

How to run the same check before your next upgrade

The procedure fits in an afternoon, and it'll tell you more than any release note. Keep the old model's outputs on a fixed set of prompts that represent your use, with temperature zero and one slot so they are reproducible. When a new version comes out, run the same prompts through it with the same flags. Compare item by item; totals hide too much: list what the new model solves that the old one did not, and the reverse, and look at the reverse list first, because it is the list of regressions. Group the items by the kind of constraint they test. If one group concentrates the regressions, as instruction following did here, you have found the thing to test before switching.

1. Freeze a prompt set and the old model's answers, one file per prompt, with the old model loaded on one slot.

   $ mkdir -p baseline && for f in prompts/*.txt; do jq -n --rawfile p "$f" '{messages:[{role:"user",content:$p}],temperature:0,cache_prompt:false}' | curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @- | jq -r '.choices[0].message.content' > "baseline/$(basename "$f")"; done

2. Load the new version with identical flags and run the same loop into a candidate directory.

   $ mkdir -p candidate && for f in prompts/*.txt; do jq -n --rawfile p "$f" '{messages:[{role:"user",content:$p}],temperature:0,cache_prompt:false}' | curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @- | jq -r '.choices[0].message.content' > "candidate/$(basename "$f")"; done

3. List the prompts whose answer changed, then grade those by hand or with your own checks, grouped by the kind of constraint they test.

   $ diff -rq baseline candidate

   Verify: the two runs used the same engine build, quantization and context; otherwise you are comparing two setups, not two models.

A model generation is an engineering change like any other, and it deserves the same regression test you'd give a library upgrade. On this pair the test found a clean tie where you would expect it, a clear regression where release notes would never mention it, and a speed difference in favour of the older model. If you maintain local models in production and want a regression check like this built into your upgrade process, you can describe the setup in the free quote form: two minutes, and I tell you upfront whether it's something I can help with.