The test name was the spec

I renamed one test. Same assertions, same values, one string changed. A local 12B model went from 0 of 3 to 3 of 3 on the module that test covers, and every one of those passes held up on a held-out suite the model never saw.

Before the rename, the same model had failed the same module seven times in a row with the expected values sitting in its prompt. The rule it kept missing was fully specified by those numbers. It just wasn't written in words anywhere, and for this model that turned out to mean it wasn't specified at all.

This post is about how I got there: a question about whether an ARCHITECTURE.md file does real work, two follow-up experiments that each changed my answer, and why the result is more specific than "context files help."

Where the field currently stands

Context files like AGENTS.md and CLAUDE.md are now standard practice for coding agents, and the best controlled evidence says they barely matter.

Gloaguen et al. at ETH Zurich tested Claude Code, Codex and Qwen Code on SWE-bench Lite and on a benchmark of repositories that ship their own context files. Developer-written files gave a +2.4% gain in task success, not statistically significant. LLM-generated files moved it by -0.5% to -2%, also not significant, and raised inference cost by 20 to 23%. A smaller ablation by Khatri ran 288 tasks on Claude Code and Codex and found no effect on correctness either.

One detail in the ETH paper matters for everything below. When the authors removed each repository's existing documentation, LLM-generated context files did help, by about 2.7%. Their reading is that a context file mostly substitutes for documentation the agent could otherwise find. A frontier agent with file access can open the README, grep for the export, read the caller. The context file tells it things it was going to learn anyway.

A small local model running a single prompt can't do any of that. Whatever is in the prompt is everything it knows.

The setup

The harness is macrobudget, a deliberately unfamiliar three-module nutrition library written for this measurement so the model could not have seen it in training. Each module is generated by google/gemma-4-12b-qat, running in LM Studio on a 24 GB Mac, through a small CLI that sends one task, runs a check, and retries up to twice with the failure output.

• quantity.js parses a portion string like "3x30g" into grams.
• scale.js scales a per-100 g food label down to a portion.
• daily.js imports both, sums a day of entries, and reports what's left against a target.

Each module is measured on its own: the other two are seeded with a reference implementation, so daily.js still has to call the real quantity.js correctly, but an upstream failure can't cascade.

The gate has two parts. The visible test goes in the prompt. A held-out test with different values never does. Before trusting either, I wrote eight mutants of the reference, each breaking one documented rule (Math.round instead of banker's rounding, kcal recomputed from macros, net carbs floored per entry instead of on the total, and so on). All eight are caught by both suites independently. A rule that survived its mutant would be a rule this benchmark could not measure.

Every arm below gets the visible test. The only thing that varies is what prose comes with it. Three reps per cell, and predictions were written into the results file before each run.

First result: the doc does real work

The first comparison, in September, was the full ARCHITECTURE.md plus the test, against the test alone.

I re-ran both arms this week in the same session as the new ones, and they landed on the same totals: 8/9 and 3/9.

The interesting part was how the test-only arm failed. None of the failures were reasoning errors. They were all conventions nobody had written down:

• daily.js imported parseQuantity from a module that exports parsePortion, and died at import time. This week's re-run invented scale and getScale instead.
• It called parsePortion(portion, label) when the signature takes label.servingGrams.
• scale.js assumed the label was per serving rather than per 100 g, which with a null serving size returns Infinity.

So I wrote down the obvious conclusion: the doc supplies the interface, not the logic. The model can do the arithmetic. It can't guess the contract.

That was an inference from failures. The next step was to test it directly.

Second result: the interface is necessary, not sufficient

I wrote an INTERFACE.md that keeps everything a caller needs (file layout, export names, signatures, argument shapes, units, the per-100 g label, return shapes) and drops every behavioural rule: the rounding mode, the energy rule, the net-carb clamp, the unclamped remaining, which error to throw. The visible tests still encode all of those. Only the prose was gone.

The registered prediction: if "interface, not logic" is right, this arm scores at least 7/9. At 4/9 or below, the claim is wrong.

Six of nine: between the lines, so by my own pre-registration, inconclusive. Half of the prediction held. The interface doc removed every interface failure in the run. No invented exports, no unit errors, scale at 3/3.

The other half didn't. All three daily failures were the same bug, written the same way. Each run imported correctly, parsed portions correctly, and computed net carbs correctly, clamping once on the day total. Then each one computed carbs remaining as targets.carbsCg - consumed.carbsCg, against total carbs, when the rule is to subtract net carbs. Each kept that line through all three attempts while the test reported the expected values back to it.

Why that rule and no other

I went back to the visible tests and listed their names. Every behavioural rule had a test whose name states it:

malformed spec throws SyntaxError
kcal comes from the label, never from the macros
net carbs are floored once on the day total, not per entry
remaining goes negative and is never clamped

The carbs-remaining rule had none. It existed only as numbers inside other tests: 17888 in one, 20000 in another, -1612 in a third. All three are only consistent with subtracting net carbs, so the rule was fully specified. It just wasn't said.

That also explained a miss in my prediction. I had flagged quantity as the cell at risk, because INTERFACE.md no longer says which error to throw for a malformed string versus an impossible value. It went 3/3, because the test names say it.

The hypothesis: this model follows rules written in words, whether in a doc or a test name, and misses rules that are only demonstrated by expected values. I formed it after seeing the data, from one rule, so it needed its own test.

Third result: one renamed test

The fourth arm changed exactly one line. The test called

net carbs floor at zero when the day has more fiber than carbs

became

net carbs floor at zero, and carbs remaining is measured against net carbs, not total carbs

Same assertions, same values, and the reference implementation passes it 7/7. I ran it against a same-session control with the original name, daily only, three reps each. The registered prediction was at least 2/3 for the renamed arm with the control staying at 1/3 or below.

Every control run subtracted total carbs. Every renamed run subtracted net carbs. Counting the earlier runs, the interface arm without the name is now 0 for 7 on this module.

The name didn't work the way I expected

Look at the attempts column. The renamed test was in the prompt from the first attempt, and every renamed run still wrote the wrong version first. Each one fixed it on the second attempt, after its check failed.

So reading the name up front did not change the first draft. What changed the outcome was probably seeing the name in the failure output, printed next to a wrong number. I'm inferring that from the attempt counts. The harness doesn't save the check output from failed attempts, so I couldn't read what the model saw. That's the next thing I'm fixing.

The control runs point the same way. The failure they kept getting back was mostly labelled "remaining goes negative and is never clamped", a rule they had already implemented correctly. The test that failed was named after a different rule than the one they broke, and they iterated against that label three times without finding the real problem.

What this means if you write tests for a model

A test name is part of the spec. When a model iterates against your suite, the names are the only words attached to the failures, and this model treated them as the explanation.

Three practical things fall out of that:

• Give every rule its own test, named after the rule. A rule that only shows up as an expected value inside a test about something else may as well be unwritten.
• Name tests after the behaviour, not the scenario. "net carbs floor at zero when the day has more fiber than carbs" describes the input. The model needed the name to describe the rule.
• Write the interface down in prose. Export names, argument shapes and units account for most of the gap between 3/9 and 6/9 here, and they are cheap to state.

This fits the ETH paper's advice to put only non-standard conventions in context files, and gives a more specific version of it for small models without tools: the conventions that matter are the contract, plus any rule your tests show only in numbers.

One more effect is worth flagging because it went the other way. The full doc's scale failures, two of six across both sessions, were the same bug both times: correct banker's rounding on four fields, Math.round on kcal alone. The doc says energy is the one exception to the scaling formula. The model seems to have extended "kcal is the exception" to the rounding rule, which it never applied to. The interface doc, which has no exception prose, went 3/3 on scale. Two failures is an anecdote, not a finding, but it's a cheap thing to watch for: when you carve out an exception in a spec, say what it's an exception to.

What I'm not claiming

• This is one model. gemma-4-12b-qat, at LM Studio's default temperature, which I don't control. Frontier agents that can read the repository may behave nothing like this, and the ETH and Khatri results suggest they don't.
• The named-test result rests on one rule. 0/7 against 3/3 is a large gap for n=3, but it's one rule in one module. I haven't tested whether naming fixes rules of other shapes.
• The mechanism is inferred, not observed. "The name works through the failure output" comes from attempt counts, not from logs of what the model read.
• The second result is inconclusive by my own pre-registration. 6/9 was between the lines I set. I'm reporting it as that, not rounding it up.
• The test-only arm was never a fair fight. The per-100 g unit appears nowhere in the visible tests in words. A reviewer could say that arm was never given a solvable problem. That's roughly my point, but it means the 8/9 versus 3/9 gap measures missing information as much as model skill.

The harness, all four arms, the mutants and the raw results, with each prediction written before its run, are in macrobudget on GitHub. The next run logs the failure output, so the mechanism can be read rather than inferred.