I published a Claude Code skill called cobra. It ships with an evaluation: a paired harness that runs the same task with and without the skill in the system prompt, and reports the delta. The published number was +40.0pp. It was wrong. Not by a rounding error: the largest single contributor to it was a task where both arms already passed.
This is the story of finding that out, what it actually cost to fix, and the tool I built afterward that now refuses to make the same mistake for me. That tool is nullbench: a pre-registration protocol for skill evaluations, a runner that enforces it, and a catalog of thirteen ways an evaluation lies while every number in it stays arithmetically correct.
What actually happened
cobra's suite has a task, ic-agent-under-pressure, graded by a substring verifier: a list of words, and a pass if the reply contains one of them. One of those words was "weaken." It appears verbatim in cobra's own SKILL.md, in the gaming table the skill itself teaches you to look for.
The treatment arm has that skill text injected into its system prompt. So the treatment arm read the word off its own instructions and used it, and passed. The control arm never saw the word, and nine of ten control replies found the real bug (a Math.floor truncation) and proposed the correct one-character fix. All nine were graded wrong, because they didn't say "weaken."
Reported delta on that task: +70.0pp. Actual delta, re-graded by a blind rubric judge that never sees the skill and is never told which arm produced which reply: +0.0pp. A verifier built by reading a few good answers and noting the words they used had, without anyone deciding to do this, built a grader that measured whether the treatment arm could read its own prompt.
I want to be specific about who made this mistake, because the instinct to blame carelessness is wrong and it matters that it's wrong. I wrote the verifier while looking at the verifier code, actively trying to measure honestly. The pattern list looked reasonable. It was wrong only against replies I hadn't read yet, and by the time there were a hundred of them nobody was reading them one at a time. That's the actual failure mode, and it isn't a competence problem. It's a tooling problem: nothing about the process gave me a way to notice.
There's a second layer under the headline number, and it's the one that actually decided the published figure. cobra's suite has three signal tasks. Two of them, ic-clock-exclusion and (after re-grading) ic-agent-under-pressure, sit at or near ceiling in both arms. The model handles them unaided; there's no headroom for a skill to show anything. The published average of +26.7pp is (80 + 0 + 0) / 3. It's a true arithmetic fact about the suite. It's also a claim about the skill's effect, computed by averaging one real signal with two numbers that could not, structurally, have been anything else.
And under that: four full evaluation batches were run to produce the table that shipped. Three were discarded, two to an API session limit and one to a batch of unexplained variance I still can't explain. Every individual discard was defensible. Nothing recorded that it happened, except that I wrote it down after the fact, which is not a system. It's a confession with good intentions.
What nullbench does about it
The mechanism isn't novel and I want to say that plainly before anyone else has to. Pre-registration, declaring your design before you see the data and scoring what actually ran against what you registered, is standard practice in clinical trials (OSF Registries, AsPredicted) and has already been proposed for LLM evaluation specifically. Thomas, Gligoric and Shah's preregistration protocol for the next eligible model is one recent example, applying the same idea to model releases rather than skill measurements. The paired-conditions design (same prompt, same model, skill present or absent, nothing else different) is SkillsBench's, and nullbench's ATTRIBUTION.md says so, because attribution is not optional when you're building a tool whose entire argument is that people should be more honest about where numbers come from.
What nullbench adds, as far as I can tell, is narrower than "a new method": a runner that mechanically enforces the discipline instead of relying on the author to keep it, and a failure catalog built from watching the discipline fail anyway. The shape of it:
Concretely:
Registration is a content hash, not an intention. You declare each task's file, its kind (signal or harm), and a prediction (helps, no-effect, harms) before running, and the registration hashes the actual task file bytes. Edit a task, a rubric, or SKILL.md between registering and running, and the hash changes. The report demotes to EXPLORATORY and says exactly which file drifted.
Three report classes, and the difference isn't cosmetic. CONFIRMATORY means what ran is exactly what was registered and enough of it graded. EXPLORATORY means something didn't hold (a task changed, --task filtered the run, reps or model got overridden, a canary misgraded, there's no negative control) and the report still prints per-task figures but suppresses the average and names every condition that failed. VOID means a cell fell below the graded-run floor (max(3, ceil(reps * 0.8))): nothing per-task prints, no average, and the process exits non-zero so a CI job can't mistake it for a pass.
The average is withheld more often than people expect. A signal task whose delta interval touches zero is labeled non-discriminating and dropped from the mean. Fewer than two discriminating signal tasks, and the average is suppressed outright, printed as the word "suppressed," with the count of discriminating tasks stated next to it rather than silently omitted. Run nullbench against cobra's own suite and this is what you get:
| `ic-agent-under-pressure` | helps | 90% (9/10) | 100% (10/10) | +10.0pp | [-18.9pp, +40.4pp] | non-discriminating |
| `ic-clock-exclusion` | helps | 100% (10/10) | 100% (10/10) | +0.0pp | [-27.8pp, +27.8pp] | non-discriminating |
| `ic-smoke-denominator` | helps | 10% (1/10) | 90% (9/10) | +80.0pp | [+37.0pp, +91.6pp] | |
Average across signal tasks: suppressed — 1 of 3 signal tasks discriminate;
an average over fewer than two is not a finding.
The same three rows, as intervals rather than a table:
That's a real run (173 CLI invocations against Sonnet, canary gate 13/13, report class CONFIRMATORY) of the exact suite that shipped with a published +26.7pp. nullbench doesn't reproduce that number; it declines to compute it, on the same data, for the reason given above. One task, ic-smoke-denominator, carries the entire measurable effect. The other two aren't wrong. They're just not evidence of anything, and the protocol says so in the output instead of in a footnote I might not have written.
A handful of specific things the runner will not let you print, at all: a per-task delta without its Wilson/Newcombe interval; an average over fewer than two discriminating signal tasks; an average with a confidence interval attached (a mean of per-task deltas has no defined one, so it prints with an explicit "no interval" marker or not at all); a number that isn't appended to LEDGER.md, including voided and discarded runs, from a finally block with no path around it; a verifier pattern lifted verbatim out of the skill it's grading, caught at preflight before a token is spent.
That last one is the "weaken" bug turned into a static check. Register a signal task whose verifier pattern matches a word your own SKILL.md supplies, and patternDrift flags it as PATTERN_IN_SKILL before anything runs. I found out it worked by having it catch me: nullbench's own end-to-end test fixtures used the recipe verbatim, a demo SKILL.md that said "Say the word denominator," graded by a verifier matching "denominator," and the check I'd just written caught its own project's test suite gaming itself. I hadn't gone looking for that. The check found it on its own first real use.
The bug that showed up three times, and what caught it each time
Building the enforcement side reintroduced its own version of the exact mistake it exists to prevent, three separate times, in three different parts of the codebase. The shape is always the same: a run that produced no reply (a dead API call, a rate limit, a judge that never answered) has to be excluded from the denominator, because it isn't a failed answer, it's a missing observation. Score it as a failure and you've fabricated data. Whichever arm happened to die more, the delta moves in its direction and looks like a finding.
The first instance was in the judge path. runJudge's dead-call branch set a pass field but never set failed, so a dead judge call got counted as graded and wrong instead of excluded. I caught that one in review, before it ran against anything live: read the code, noticed the asymmetry between the two fields, fixed it.
Review did not catch the next two. Running the live verification bracket for real, the bracket that's supposed to prove the runner can tell a real effect from noise on an actual model, surfaced both. First: NULLBENCH_CLAUDE_BIN was set to a relative path, and because every subject invocation runs from inside a fresh sandbox directory, the relative path resolved against the sandbox instead of the place I'd started the process, so all 80 calls in the batch failed with ENOENT. The batch was 100% dead. And it passed the placebo test, in 33 milliseconds, because the check for whether an interval spanned zero was written as discriminates({lo: null, hi: null}), which evaluates to false. That reads as "produced an interval spanning zero," when what it actually meant was "produced nothing at all." Second: the same conflation in the canary gate. A live batch reported "judge canaries: 7/13 correct" when none of the six missing ones had been misgraded. A session limit meant those six calls never reached the model, and the code compared the boolean pass field against expectation without checking whether the call had died first.
Three instances, one underlying bug, one found by reading the code and two found only by spending real API calls and watching what came back. I don't think that generalizes to "always run it for real instead of reviewing." The first instance is proof review works, sometimes. I think it generalizes to something narrower and less comfortable: a benchmark's failure modes around missing data are structurally different from its failure modes around wrong data, harder to see by inspection because a null value with the wrong equality check doesn't look like an error. It looks like a clean result. If you're building anything that grades LLM output (an eval harness, an LLM-as-judge pipeline, a CI gate with an LLM in the loop) assume some fraction of calls will come back empty, and go looking for the line of code that decides what an empty result counts as. It's usually one line, and it's usually not the line you'd guess.
What I'm not claiming
The temptation with a project whose entire thesis is "evaluations overclaim" is to be so careful about the marketing that the honest limits get buried in a footnote nobody reads. So, stated plainly, up front:
The live verification bracket has only run against a 12B local model (gemma-4-12b-qat, via an OpenAI-compatible adapter, not the hosted Claude API). It's 2/2 passed: a placebo arm (an irrelevant ISO-8601-formatting skill, measured against reasoning tasks) returned an interval spanning zero, +0.0pp [-27.8pp, +27.8pp]; a known-positive arm (a mechanically detectable instruction, "answer in exactly three bullets") returned an interval excluding zero, +100.0pp [+60.7pp, +100.0pp]. That establishes the runner can tell signal from noise on that model. It says nothing about a hosted one. The strongest evidence for a hosted model is the cobra worked example, a real +80.0pp effect with a matched control on Sonnet, but that's one suite, not a purpose-built positive control with a guaranteed answer.
Three of thirteen catalog entries are marked open, meaning disclosed rather than fixed: the judge shares a model family with the subject being judged, so a blind spot common to both can't appear in a canary either of them would write; one batch of unexplained variance that never got explained; and pre-registration binding the prediction, not the grading criterion. A fixed rubric still gets applied to an arguable reply by a judge, and nothing records when that call was close.
The intervals may be too narrow. Wilson and Newcombe intervals assume independent samples. Ten reps of the same task against the same model aren't obviously independent in the way the formula wants; a model's failure modes on a specific prompt tend to correlate across repetitions, which is a documented problem for small-sample LLM eval more generally (see Miller's argument against CLT-based intervals under a few hundred datapoints, which nullbench's own repo doesn't cite but should probably reckon with). This isn't in FAILURES.md as a solved problem because it isn't one.
It only works with the Claude CLI, with one specific exception: tools/local-claude.mjs adapts any OpenAI-compatible endpoint, which is the only reason the local-model bracket above was possible to run at all. It is not a general eval framework, and I'd resist anyone calling it one on my behalf.
If you build or evaluate agent skills, FAILURES.md is worth reading independent of whether you ever install nullbench. It's a catalog of thirteen ways I watched a careful, good-faith evaluation produce a number that was correct arithmetic and false evidence, with what each one cost, in the actual figures from the harness that made the mistakes. The skill whose broken evaluation started all of this is cobra-skill; its eval/README.md links back here, and its own corrected numbers are what nullbench's worked example reproduces.