Green tests and wrong code, how local LLMs fake a fix

Every agentic coding benchmark has to decide what "solved" means, and the cheap answer is "the tests pass". It's cheap because the tests are already there, and it's wrong because the agent can see them. Push a model to turn a red suite green and it'll find the shortest path to green, and on a non-trivial bug the shortest path is very often a fix that satisfies each visible assertion and is still wrong. I measured this on my 16GB RTX 5060 Ti with five local models, five broken projects and three runs per model and project, with a second judge the agent never sees: a differential oracle that runs the fixed code and a reference implementation side by side on randomly generated inputs. If you evaluate agents by pass rate, you won't like the result. The two 9B models produced more illusory fixes than real ones, the 35B mixture of experts produced a few, and the two 27B models produced none. This article shows the numbers, the signatures of the wrong fixes, and how to build the hidden check yourself, because you can reuse the method on your own code; my particular results you can't.

What counts as solved when an agent fixes a bug?

A fix is solved when the visible tests pass and an independent oracle agrees with a reference on inputs the agent never saw. Every episode ends in one of three outcomes, each with its own name:

• OK: the visible suite is green and the hidden oracle finds zero disagreements with the reference.
• ILL (illusory): the visible suite is green, the hidden oracle disagrees. The agent thinks it's done, the reviewer sees green, the code is wrong.
• FAIL: the visible suite is still red when the agent stops.

A FAIL is a failure you notice. ILL is the dangerous one, because nothing in the agent's own workflow tells you. In a real repository it becomes a merged pull request with a green CI badge.

A passing test suite measures how convincing a fix is to the tests the agent could read. That's all it measures.

If you build agentic workflows for a team and want illusory fixes caught before code reaches review, my hub on AI development tooling covers how I design guardrails around coding agents.

The tasks: five real bugs with a trap each

Each task is a small project, one to three files to change, with a bug report written the way a colleague would write it and a visible test suite that starts red. These bugs pass review when nobody looks closely:

Every task has a hidden oracle: a reference implementation and a generator of random inputs, compared on 40 to 80 trials. Before any model ran, I wrote the plausible lazy fixes for each task, ran them through the oracle, and recorded how many trials each one gets wrong. That number is the signature of the lazy fix. When an agent's output disagrees on the same number of trials, it's almost certainly the same mistake.

The models ran with reasoning on, inside an agent loop with file and shell tools, three repetitions per task, with time and turn ceilings per task.

What did the five models actually deliver?

The two 27B models delivered 29 correct fixes in 30 episodes and no illusory ones. The two 9B models delivered 9 correct fixes and 15 illusory ones.

Two numbers in those totals stand out. One is the ratio of illusory fixes to everything that came back green: 7 of 12 for Ornith-1.5-9B and 8 of 12 for NeoHorse-1-9B. When these 9B models report success, more often than not they are wrong. The other: the larger models aren't immune either, and the Ornith 35B mixture of experts shipped 3 illusory fixes out of 13 green ones.

These are descriptive numbers. With five task families, the arithmetic alone rules out a significant pairwise comparison between two models, so I don't claim that one model beats another here. The repetitions reduce noise inside a task; they don't add independent observations. The data does support the shape: illusory fixes concentrate in the smaller models, and they exist even in a 35B.

The trap that caught eight episodes with the same signature

On snapkeep a predicted failure showed up most plainly. The bug report says the nightly run deleted a snapshot that operations needed for a restore drill. The correct fix needs four changes in three files: a time zone sign in clock.py, the first day of the week and the daily rule in rules.py, and one change in plan.py. The visible tests and the log point at the first three. The fourth comes from one sentence in the README: a snapshot on hold is never pruned and also counts toward no rule, as if it did not exist. The shipped code already keeps held snapshots; it still lets them take a place that a rule should give to another snapshot.

Before any model ran, that lazy fix gave 47 disagreements out of 80 against the reference. In the campaign, eight episodes came back green with the hidden oracle reporting exactly 47 of 80: two from the Ornith 35B, three from Ornith-1.5-9B and three from NeoHorse-1-9B. Eight of eight with the same signature, all of them editing clock.py and rules.py and never opening plan.py for an edit. Every episode that solved snapkeep, seven of them, also changed plan.py.

When a wrong fix is predicted before the run and then appears with the same signature eight times, you are looking at a habit of the model.

The other illusory fixes follow the same pattern with other signatures: on shiftpay the 9B fix that groups shifts by date and then merges them disagrees on 57 trials of 60, on tailer the fix that counts characters instead of bytes disagrees on 34 to 40 trials of 40.

Building a hidden differential oracle, step by step

The method is portable to any codebase where you can write a reference, even a slow one. Here's a minimal illustration in Python, on a simplified retention rule instead of the code of the task above.

1. Write a reference implementation that is obviously correct, not fast.

   # Slow but readable reference: no optimisation, only the rule as the README states it.
   def reference_should_delete(snapshot, now):
       if snapshot.legal_hold:
           return False
       age_days = (now - snapshot.created).days
       return age_days > snapshot.retention_days

2. Write a generator that produces random inputs biased toward the edges: boundaries, empty values, the special cases the README mentions.

   import random
   from datetime import datetime, timedelta
   
   def random_snapshot(rng):
       # Biased toward the edges: zero days, the exact limit, an active hold.
       created = datetime(2026, 1, 1) + timedelta(days=rng.choice([0, 1, 29, 30, 31, 365]))
       return Snapshot(created=created,
                       retention_days=rng.choice([0, 30, 31]),
                       legal_hold=rng.random() < 0.3)

3. Run the candidate and the reference on the same inputs, with a fixed seed, and count disagreements.

   def differential(candidate, trials=80, seed=20261008):
       rng = random.Random(seed)
       now = datetime(2026, 6, 1)
       disagreements = 0
       for _ in range(trials):
           snap = random_snapshot(rng)
           if candidate(snap, now) != reference_should_delete(snap, now):
               disagreements += 1
       return disagreements

4. Before trusting the oracle, attack it with lazy fixes you write yourself. Each one should be green on the visible suite and fail the oracle with a distinct count. Record those counts: they are the signatures you will recognise later.

5. Keep the oracle and the reference out of the agent's workspace. Copy them in only at grading time, in a scratch directory, so the agent can never read them or adapt to them.

Most people skip the fourth step, and without it you can't trust the oracle. An oracle that has never rejected a plausible wrong fix hasn't been tested. In one of my tasks this step found a bug in the oracle itself: two extreme values in the random generator overflowed when averaged, and two different correct fixes disagreed with the reference on the same trial. The fault was in the oracle, and I fixed the generator before running any model.

Why the visible tests cannot be fixed by writing more of them

The obvious objection: just put the hidden cases in the visible suite. It can, for the cases you know. The point of the hidden oracle is the cases you did not think of, generated from the specification instead of listed by hand.

There is also a subtler reason. An agent that can read the tests optimises for the tests. In my runs, the agents that produced illusory fixes weren't cheating on purpose; they made the change that turned red into green, verified it with the suite, and stopped. The suite was the only evidence of correctness available to them. A model that checks its fix against every sentence of the specification, including the ones no test exercises, behaves differently, and that is what the correct episodes did.

The same failure shape appears without an agent loop. On fifty single-turn Python problems with reasoning off, the five models together failed 148 times, and 146 of those failures were silent: the model returned complete, plausible code that only the tests could reveal as wrong. A failure you notice costs you a retry. One you don't notice costs you whatever it breaks downstream.

What this means if you use a local model as a coding agent

I take three rules from these runs, and they apply to cloud models too.

First, a green CI is not acceptance for agent-written code. Treat an agent's passing suite as a claim to verify, with tests the agent did not see, or with a reviewer who reads the specification instead of the diff.

Second, size the model to the cost of a silent error. On these tasks a 9B model is a fast way to produce something that looks finished. On a task where a wrong but plausible fix is expensive, only the 27B models never shipped one.

Third, write the lazy fix before you trust the agent. Ten minutes spent writing the obvious wrong fix and checking that your tests reject it shows whether they can catch the failure you're most likely to get.

The statistical method behind these numbers, including why running a task again doesn't add independent evidence, grew out of the Ornith vs Qwable comparison, where two 9B agents tied on pass rate. The harness configuration for running these models on a 16GB card is in the recommended strategies for local LLMs, and the measurement traps that shaped how I record every episode are in what llama.cpp benchmarks get wrong. For the property-based side of differential testing, start from the Hypothesis documentation if you prefer a library to a hand-written generator.

What separated the episodes that got snapkeep right

I expected the difference to be in reading: the larger models opening the README, the smaller ones going straight to the failing test. The trajectories say otherwise. In all fifteen snapkeep episodes, every model's second action was to open README.md. All of them read the sentence about held snapshots. What differed was what they did with it: the seven correct episodes changed plan.py, where held snapshots are handed to the retention rules, and the eight illusory ones never edited it. They read the specification and didn't act on every sentence of it, and the visible suite gave them no cause to act on that one.

Time spent doesn't separate them either. On snapkeep the GSQ-RCO 27B took a median of 786 seconds per episode and solved all three; the 9B models stopped between 380 and 783 seconds, with a green suite and a wrong fix. On shiftpay the 27B needed a median of 744 seconds to deliver three correct fixes, while Ornith-1.5-9B ran for more than 3,000 seconds in one episode and still shipped the lazy fix.

There's a cost side to this, and it isn't the one I expected. The larger models are slower per token on a 16GB card, so I assumed a wrong small-model fix would at least come back sooner. On the three long tasks, the GSQ-RCO 27B's correct fixes took a median of 744 seconds; the 9B models' illusory fixes on the same tasks took a median of 764. The wrong fix didn't even come back faster. Add the time someone spends finding and repairing an illusory fix after it's merged, and it isn't close.

If you're choosing a model for an agentic workflow, look at that chart first. It answers what a reviewer really wants to know: when the agent says it's done, how likely is it to be right?

A note on what the oracle cannot see

A differential oracle is only as good as its reference and its generator. If the reference shares a misunderstanding with the agent, the oracle agrees with both and catches nothing. If the generator never produces the input that triggers the bug, the oracle reports zero disagreements on a broken fix. Both failure modes are real, and the defence is the same as for the agent: write the lazy fixes first, and require the oracle to reject each of them with a count you understand. In my tasks every oracle was validated against two independent correct implementations, which must both score zero, and against three to five lazy fixes, which must all score above zero. Only then did a model see the task.

An agent that makes your tests pass has told you something about your tests. It has told you something about your code only if a check exists that it couldn't see. On my bench that check turned half of the 9B successes into failures and left the 27B successes untouched, and a trap written before the run caught eight episodes with the same fingerprint. If you are bringing coding agents into a team and want the acceptance gates designed around a hidden check like this, you can describe the setup in the free quote form: two minutes, and I tell you if it's something I can help with.