The spec named nothing

Yesterday I switched off a prompt default because its benefit refused to reproduce. Today I found out why it had ever seemed necessary. It was a patch for a spec that named none of its tests and never said where a short code comes from — and once the spec said both, the default I had spent a day agonising over measured as an exact null, four arms out of four.

The model was not guessing. It was doing what the spec allowed.

shortener's suite reported 72.7% coverage and green, on two test functions. Here is what one of them did:

store := NewMemStore()
store.Save("http://example.com")
req := httptest.NewRequest("GET", "/r/0", nil)   // the code is whatever Save assigned. Not "0".

--- FAIL: TestRedirectHandler   want 301, got 404

A 404 against handlers that are perfectly correct. The model reached past the API, planted a link in the store directly, and then asked for a code it had invented. And the spec had never said otherwise: it described the flow in prose — "POST /shorten to get the code, then GET /r/{code}" — named zero test functions, and showed no code at all.

So the purpose now names all eight, and shows the six lines that take the code out of the response, ending in the only sentence that matters:

var created Link
if err := json.NewDecoder(rec.Body).Decode(&created); err != nil { t.Fatal(err) }
// created.Code is the ONLY code that exists. Use it:
h.ServeHTTP(rec2, httptest.NewRequest(http.MethodGet, "/r/"+created.Code, nil))

Measured, two runs, one server process:

                 test funcs   coverage   verdict
old spec              2         72.7%     GREEN
fixed spec            8         73.1%     GREEN   // x2, identical

guessed codes: 0        reads created.Code out of the response: 19

The eight are exactly the eight named. The 404 class is gone at the root — not gated, not nudged, not retried. It cannot happen, because the test no longer has a code to invent.

And the default I agonised over is a null

The rule I spent yesterday on — write one focused test function per scenario — broke this exact spec three runs out of three when the spec left its tests implicit: ten test functions, a test file that would not compile, 0.0% coverage. That was the evidence that sent it off by default.

Re-run against the fixed spec, arms alternated, one process:

arm         test funcs   coverage   verdict
with 1/2        8            73.1%    GREEN
without 1/2     8            73.1%    GREEN

Four arms, four identical results. The rule's harm was never the rule's. It was a function of the spec's silence — and with the silence gone, the rule is not harmful, it is simply nothing. It was a patch over an under-specified spec, and I had been debating the patch.

Then the second spec refuted the theory

Twelve specs in the suite name two or fewer test functions. shortener was one, and it had a real, measured defect. The obvious next move was to fix them all. logstats looked identical: names zero, ships two top-level test functions, no subtests, no tables.

I ran grep -c ErrBadLine on its test file, got 1 where the spec enumerates four bad-line cases, and had my confirmation. Then I read the file instead of the grep. It tests all four, in a loop. The literal appears once because the assertion is err == nil.

Named-test count does not predict under-testing. logstats names nothing and covers everything; shortener was under-tested because it bypassed the API, not because its tests were unnamed. Same symptom, different defect, and no suite-wide sweep is justified. The generalisation died on the second data point, which is the cheapest place for a generalisation to die.

What logstats did have was the mirror image of shortener's bug. The spec said those four bad lines "all return ErrBadLine via errors.Is". The test asserted only that something failed — so a ParseLine returning any random error passes, and the sentinel contract the file exists to pin was not tested at all. Cases kept, assertion hollowed out. The spec had already said errors.Is, in prose, and the model wrote err != nil anyway. So show it instead of saying it:

if _, err := ParseLine(bad); !errors.Is(err, ErrBadLine) {
    t.Errorf("ParseLine(%q): err = %v, want ErrBadLine", bad, err)
}

errors.Is(err, ErrBadLine) checks:  0 → 4     // one per bad line, transcribed exactly
coverage:                        82.0% → 82.0%  // unchanged to the decimal

Every real gain today was invisible to both numbers

That flat 82.0% is the third independent instance in two days of the same thing, and by now it is not an anecdote:

TestListSorted        a spec-NAMED test the model dropped        coverage +0.0
shortener             2 test functions became 8                  coverage +0.4
logstats              4 sentinel assertions where there were 0   coverage +0.0

The shortener suite now checks that a redirect returns 301, that Location equals the original url, that a missing code 404s, that a bad request 400s, that Hits is exactly 1 after exactly one hit. Before, it checked none of them and reported 72.7%. Those assertions execute the same lines either way.

Green says the tests that exist pass. Coverage says the tests that exist reach the code. Neither says the assertion is the one the contract asked for — and this project reports exactly those two numbers.

Twice, the instrument was the thing that lied

Both of today's wrong turns came from a grep, not from the model. The ErrBadLine count that read as "one case of four" was one literal over four cases. And a column I generated to ask "does this spec show code?" answered no for ratelimit — the one spec that shows a helper's body in full — because the regex only looked for h := assignment lines. I nearly published that table.

The rule that came out of it is duller than the one I wanted and more useful: read the file, not the grep, and test a heuristic against a case you already know the answer to before you trust its verdict on cases you do not.

What this leaves

Two days ago the question was which prompt default to ship. Today the answer is that the default was never the lever. A spec that names its tests gets them. A spec that shows the code gets the code. A spec that leaves either one implicit gets whatever the model improvises — and then a default gets invented to argue with the improvisation, measured, agonised over, and eventually turned off, while the actual defect sits in a YAML file nobody re-read.

Which is the same law this project keeps re-earning from a new direction: implicit means broken, prose loses to shown code, and the compile-and-test machinery cannot recover what the specification never said. The machine's job is to catch what the model gets wrong. It was never going to catch what the spec forgot to ask for.