Writing coding tasks that LLMs cannot fake

When I started measuring local coding agents on my 16GB RTX 5060 Ti, my first benchmark was useless, and I needed a while to see why. The tasks were reasonable, the tests were thorough, and every capable model solved almost all of them. Later, an audit of that bank showed it told models apart on two instances out of sixteen, both already at the ceiling. The models weren't that good; the tasks were that easy to satisfy. The test suite in the repo shows an agent what to turn green, and once it's green the agent has passed, bug fixed or not. Below is how I rebuilt the agentic part of my benchmark to catch fixes that only convince the tests. I tell it in order, failures included, since they taught me most of the method. If you evaluate coding agents, local or hosted, and want a number that means something, it's for you.

The first lesson: a visible test can always be satisfied

An agent in a loop with file and shell tools sees the bug report, the code and the tests. It runs the tests, reads what fails, changes code, runs again. The tests are the only evidence of correctness it has, so it optimises for them, and on any non-trivial bug there is usually a shorter path to green than to correct.

The way out is a second judge the agent never sees. For every task I write a hidden differential oracle: a reference implementation, slow but obviously correct, and a generator of random inputs. At grading time, after the agent has finished, the oracle runs the agent's code and the reference side by side on tens of random inputs and counts disagreements. A fix is solved only when the visible suite passes and the oracle finds zero disagreements. Every episode ends in one of three outcomes, named separately:

The oracle lives outside the task tree and is copied in only at grading time, in a scratch directory, so the agent can neither read it nor adapt to it.

If you build agentic workflows for a team and want acceptance gates that work this way, my hub on AI development tooling describes how I design them.

The second lesson: write the wrong fix before the right one

An oracle that has never rejected a plausible wrong fix hasn't been tested. Before any model sees a task, I write the lazy fixes I expect, the obvious edits that make the repo's tests pass and miss the point. Each one must pass the visible suite and fail the oracle with a known count of disagreements on a pinned seed. That count is the signature of the lazy fix.

With signatures, the oracle's count works as a diagnosis. On one task, a snapshot retention policy, the bug report says the nightly run deleted a snapshot that operations needed. The visible tests and the log point at a time zone sign and two retention rules. The README adds, in one sentence, that a snapshot on hold is never pruned and counts for no rule, as if it weren't there. The shipped code already spares held snapshots, but lets them occupy a slot that a rule ought to hand to some other snapshot. The obvious fix repairs what the tests point at and leaves that alone. Before the campaign, that lazy fix scored 47 disagreements out of 80. In the campaign, eight runs came back green with the oracle reporting exactly 47 of 80, from three different models. Eight of eight with the predicted signature, all touching the same two files and none touching the hold logic.

When a wrong fix is predicted before the run and then shows up with the predicted signature, you're not measuring noise. You're measuring a habit.

The third lesson: attack your own oracle

Writing lazy fixes tests whether the oracle can reject wrong code. Writing a second, independent correct fix tests whether it accepts right code. Both must happen before any model runs, and the second one caught a bug in one of my oracles.

The task was a sensor log parser. Two correct fixes, written in completely different ways, a regular expression and a hand-written scanner of the same grammar, both disagreed with the reference on the same single trial. Two independent correct implementations failing identically pointed at the oracle, not at the fixes. The random generator was producing readings near 1e308, and the reference averaged two of them: the sum overflowed to infinity, which the README did not address. I capped the generator at 1e300, declared overflow out of scope, and only then let any model see the task.

Every task in the current bank passes these checks before it is used:

The fourth lesson: can you guess how hard a task is?

No, not even when you wrote it yourself; I learned that the hard way. A task every model solves measures nothing, and neither does a task no model solves. For a comparison a task has to land in the middle, solved in somewhere around 30 to 70 percent of runs on a panel of models of different strength.

So every new task goes through a calibration panel before it enters the benchmark: three models, from a 9B to a 27B, three runs each, nine runs per variant, with reasoning on. The rule is fixed in writing before the panel runs: a variant is promoted if its solve rate falls between 0.3 and 0.7; between two variants in band, the one closer to 0.5; on a tie, the one that gives the agent fewer hints. I also write my prediction of each rate before the panel, to keep myself honest.

The panels taught me which levers work.

Readability of the specification does not. The envsub task came in three variants, from an explicit specification to an oblique one, on the theory that a harder-to-read spec would separate models. It did not: 9, 8 and 8 of 9. The lever was discarded.

A platform fact the visible tests cannot exercise does. The tailer task ingests new lines from log files run after run and stores how far it got as a byte offset. The README defines events and offsets in bytes; the shipped code opens files in text mode. The obvious fix stays in text mode. That's correct on plain ASCII logs and wrong once a line holds a multi-byte character or a carriage return, and the visible tests are all ASCII. The agent agrees with the README, believes its code honours it, and nothing it runs shows otherwise. With reasoning on, the panel landed at 4, 4 and 3 of 9, and the ascii variant was promoted.

But not when the README also gives the shape of the solution. The sensorjson task applies the same lever to Python's float(), which accepts nan, inf, underscores and other forms a number grammar rejects. I predicted one to four solves in nine. The panel solved 9 of 9 and 7 of 9, because the README stated the grammar word for word and every model, including the 9B ones, simply implemented it. The trap only works when the README states the rule and leaves the platform behaviour to the reader.

And not when the ceilings are too tight. The timesheet task, built on the fact that same-zone datetime subtraction in Python ignores daylight saving changes, first came out at 2 and 0 of 9, which looked like a hard task. Reading the episodes showed that 12 of 16 failures had stopped on a ceiling of turns or time, not on a wrong answer. With the ceilings raised to 40 turns and 1200 s, the winter variant came out at 4 of 9 and was promoted. The first panel had measured my budget, not the models.

My written predictions for the last round, compared with what the panels measured, show how far intuition is from measurement even when you designed the task yourself:

Two out of three predictions were too pessimistic, and the one that hit the number did so for the wrong reason. That is why the rule decides, not the author.

The fifth lesson: count families, not runs

Three runs of the same task by the same model are three samples of one task, differing only in the random seed, and count as one observation of the model's ability. When I compare two models, the unit of comparison is the task family, and the paired test runs over families.

That has an arithmetic consequence that surprised me. With a paired sign test over n families, the smallest p-value you can ever reach, even with one model winning every family, is 2 divided by 2 to the power n. With five families that floor is 0.0625, above the usual 0.05 threshold: no comparison on five families can ever be significant, however lopsided the results. With six families the floor drops to 0.031, with seven to 0.016.

My benchmark ran a full 45-hour campaign with five families, which is why I report its agentic results descriptively and refuse to name a winner between two models from them. The calibration rounds that followed brought the bank to seven families. Repetitions never help here: more runs of the same task narrow the noise within that task, they do not add families.

The sixth lesson: a ceiling task is not a useless task

The envmerge task, merging environment files with precedence rules, ended up solved by every model in every run: 15 of 15 in the campaign. It can't tell models apart, and in a paired comparison it's a guaranteed tie, so significance gets harder. I keep it anyway, as a smoke test for the agent loop, the tools and the grading. If a model ever fails envmerge, I'd look at the harness first. Every benchmark should carry a few tasks like that, clearly labelled and left out of the comparison statistics.

What the finished tasks revealed about the models

Built this way, the tasks separate models in a way the old bank never did. On five families, three runs each, the dense 27B models delivered 29 correct fixes in 30 runs and no illusory ones, while two 9B models delivered 9 correct fixes and 15 illusory ones. The 9B models, when they reported success, were more often wrong than right. I describe those runs in how local LLMs fake a fix, and the Python facts behind three of the newer tasks in three Python pitfalls that fool local LLM coding agents.

The calibration rounds also produced the one result I value most as a builder: a lazy fix nobody had predicted. On the VAT rounding task, one 9B model rounded with int(x * 100 + 0.5) on floats and summed totals without rounding again. It disagrees with the reference on only 4 trials out of 40, which makes it nearly invisible to hand-written tests. I added it to the list of known lazy fixes, so the next time it appears it will be recognised by its signature.

The method, condensed

Hidden differential oracles, written lazy fixes with recorded signatures, two independent correct fixes per oracle, calibration panels with a rule fixed before the run, ceilings checked before difficulty is concluded, and families counted instead of runs. You don't need a GPU cluster for any of it; my whole bench is one consumer card. You need the habit of trying to break your own tasks before a model does. The same habit, turned on measurement in general, is in what llama.cpp benchmarks get wrong, and the statistical reasoning grew from the Ornith vs Qwable comparison. For the property-based side of random input generation, the Hypothesis documentation is a good companion to a hand-written generator.

A coding benchmark claims that a number tracks capability. Write every task as if someone were out to prove that claim false, because the models are, each time they find a shorter path to green. If you want tasks held to this standard on your own codebase, you can describe the case in the free quote form: two minutes, and I tell you upfront whether I can help.