← Research log
Report #20 · 2026-07-12

The tests were wrong. So was the ruler.

The last report ended with a promise: coverage was going in, and the next sweep would measure both numbers, because a coverage gain paid for with a green loss is not a gain. This is that sweep. Two of three specs went red — and not one of the reds was the model's fault. All three were tests that no correct implementation could have passed. Then the ruler that was supposed to score the work lied three separate times, and the last lie was told by the gates.

Why coverage at all

Green is a shallow claim. It says the tests that exist pass. It says nothing about whether those tests reach the code that shipped. The store packages sat at a flat 50.9% for weeks, and the reason turned out to be arithmetic: eight methods, four for Task and four mirrored for Project, and only the Task four were ever called. A method nobody calls is a method nobody has shown to be correct. So the specs grew a models_test.go and the Project half of the store tests, and the sweep began printing green and coverage side by side. Neither is the score. The pair is.

Three tests that failed against a correct implementation

An unreachable state. workapi's config test was asked to prove that an empty AuthToken fails Validate(). The model wrote correct code and the test failed anyway, because the spec's own defaulting rule makes that state impossible to reach:

func getenv(key, fallback string) string {
        if v := os.Getenv(key); v != "" { return v }
        return fallback                    // AUTH_TOKEN="" is UNSET, so → "secret"
}

t.Setenv("AUTH_TOKEN", "")
got, _ := Load()                           // token is now "secret" — NOT empty
err := got.Validate()                      // correctly nil. Test fails.

A constructor that validates before it returns can never hand you an invalid value — that is its entire job. On failure it gives back the zero value, and the zero value usually passes the very check you were trying to fail. The spec named the assertion and left the construction unpinned, so the model reached for Load() like its neighbours did, down a path that cannot produce the state. taskapipro carried the identical latent bug and passed only because its model happened to build the struct literal directly. Nothing in the spec required it. A stochastic pass is not a pass.

An unsatisfiable row. The spec said write separate focused test functions. The model built a table anyway — and dropped a valid "done" status into the bad-status table with expected: nil, under a loop whose assertion demands an error from every row:

{"bad status with done status", Task{Status: "done"}, nil},   // expects nil…
...
if err == nil { t.Errorf("expected %v, got nil", tc.expected) } // …but the loop demands an error

The tell is the error message itself — expected <nil>, got nil, a sentence that cannot mean anything. That row is unsatisfiable by construction; no code that could ever be written makes it pass. The default already said prefer focused functions over a table, and the model built a table anyway. So rather than say it louder, the rule now makes the table safe: one function, one outcome — every row expects the same kind of result, because the assertion after the loop is shared and can only express one.

A default is only as good as the noun the model acts on

The third red is the sharpest thing in this report. taskapipro's router test reported want 409, got 400 — against a handler that maps ErrExists to 409 perfectly correctly. The bug is in the test, and it is a real Go trap:

req := httptest.NewRequest("POST", "/tasks", bytes.NewBufferString(body))
h.ServeHTTP(w, req)                // first call DRAINS the body
w = httptest.NewRecorder()         // fresh recorder…
h.ServeHTTP(w, req)                // …same req. Body is empty now.

A request's Body is an io.Reader. The first ServeHTTP reads it to EOF, so the second call sends an empty body, json.Decode fails, and the handler answers 400 — long before any duplicate check runs. The 409 path is never reached.

Here is what makes it sharp. The rule was already there, and it was already right:

HTTP-TEST HYGIENE: build a FRESH request body for every request — an
io.Reader/bytes.Buffer is drained after one read, so reusing it sends an
empty body on the second request (a re-POST then wrongly returns 400 instead of 409).

It names the mechanism. It predicts the exact wrong status code, in advance, in parentheses. And the model walked straight into it — because the sentence names the body, and the object the model reused was the request. It even thought about freshness: it built a fresh recorder, which nothing had asked for, while handing the same req to ServeHTTP twice.

The last report's finding was that a gate is only as good as the sentence it listens for. This is its twin on the other side of the loop: a default is only as good as the noun the model acts on. Bind a correct rule to the wrong noun and it behaves exactly as if it were not there. The rule now names *http.Request, forbids reusing req, and simply shows the two-request shape.

The ruler lied three times

With the reds understood, the remaining question was whether the coverage work had actually bought anything. Answering it meant building an instrument — and the instrument was wrong three times before it was right once.

It measured a different thing than the baseline. The first numbers disagreed with the recorded history: store 50.9% → 58.5%, models 0% → 25%. That is not progress, it is a definition swap. Two different questions had been given the same name: does this package have tests of its own, and is this code executed by any test in the module. The models package scores 25% on the second while scoring 0 on the first, purely because the store and api tests construct models values while testing themselves. A metric that quietly changes definition manufactures progress out of nothing. Both are now printed, and labelled.

It double-counted. Under -coverpkg=./... every test binary instruments every package, so the merged profile lists each block once per binary — store's blocks with count>0 from the store binary, and the same blocks with count=0 from the models binary. Summing the lines naively counts a block N times in the denominator and once in the numerator. It reported the coverage push as a regression.

And then the gates poisoned it. Even deduplicated, the ruler put internal/api at 58.5% while go itself said 69.0%. That is not a finding — it is impossible. The second number is a strict subset of the first: code executed by any test cannot be less than code executed by the package's own tests. Only the impossibility caught it. The profile held the same block twice, one line apart:

middleware.go:14.60,15.37       middleware.go:18.2,18.10
middleware.go:15.60,16.37       middleware.go:19.2,19.10   // the same code, shifted

A cached coverage row is keyed to the line numbers it was recorded at. The gate chain shifts lines — that is exactly what it does when it inserts a missing import, and it is the same physics that made the gates corrupt the files they repaired two reports ago. Go's test cache had rows from before the shift and rows from after, and the merge kept both: every statement counted twice in the denominator, once in the numerator. The line-shifting gates do not only damage the files they fix. They poison the instrument that measures them. -count=1, everywhere, forever.

A bug with no sentence at all

All three rules went into the Builder's permanent defaults rather than into the three specs that surfaced them — each is latent in every spec of its shape, and the request trap alone is live in five. But a default is global, so it has to be swept: seven other specs, regenerated from scratch. Six came back green. The seventh, ratelimit, went red — and it is worth the space, because it is not a regression and it is not a model that can't write Go.

The defaults only enter the prompt for *_test.go files. The bug is in api.go:

func NewRouter(reg *Registry) http.Handler {
        mux := http.NewServeMux()
        mux.HandleFunc("GET /ping", PingHandler)
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                if r.URL.Path == "/ping" {
                        PingHandler(w, r)     // RateLimit is never applied
                } else {
                        mux.ServeHTTP(w, r)
                }
        })
}

RateLimit is declared, and never called. Asked to limit one route and leave another unlimited, the model hand-rolled a dispatcher and dropped the middleware on the floor. Every request is allowed; every rate-limit test fails.

And here is the part that matters. An unused package-level function is legal Go. The build is green. go vet is green. The compiler has nothing to say — so the gate that exists for exactly this bug cannot fire, because there is no error message for it to listen for. Not the wrong sentence, as in the last report. No sentence. The only thing that notices is the test. This is the honest floor of the deterministic layer: gates repair what the compiler can name, and a semantic hole that compiles cleanly is not theirs to catch. The spec now shows the exact wrapping shape instead of describing it — the same lever that took the workapi wall down: make the ask concrete, and there is nothing left to invent. The model wrote it straight back:

mux.Handle("GET /ping", RateLimit(reg)(http.HandlerFunc(PingHandler)))

Green, 75.4% of statements executed. Ten specs regenerated from scratch across this session — three for coverage, seven to sweep the defaults — and all ten are green.

What it actually bought

                green      store (own)      models (own)     exec-total
  taskapi         ✅       50.9% → 100%       0% → 100%         62.8%
  taskapipro      ✅            → 100%        0% → 100%         64.1%
  workapi         ✅            → 100%        0% → 100%         70.9%

Every mirrored Project method in every store is now executed. Every field rule in every models package is now tested directly, where before it was reached only by accident, through somebody else's test. All three specs are green, and the coverage that was there before is still there. The trade the last report was afraid of — buying depth with green — did not happen. It only looked like it had, for as long as the ruler was broken.

This week's pattern was mechanisms that look like they work. Today it went one layer out. Every red in this sweep was upstream of the model: three times it wrote correct Go and was marked wrong by a test that no correct program could satisfy. And every time the measurement disagreed with reality, the measurement was what was broken — including, in the end, because the repair machinery was corrupting its own scoreboard. None of it was visible until something counted, and the one lie that got caught was caught only because it was arithmetically impossible, not because it looked wrong. Ten specs regenerated from scratch, ten green, the Builder's suite at 290 tests. The model has not changed since April. The bill is still $0.

Postscript: I walked into it myself

The fix in this report broke something. The rewritten HTTP rule went into the global defaults, the regression sweep ran — as a global default forces you to — and taskapipro came back red with every seeded case failing at once: the duplicate answered 201, the record just POSTed answered 404, the list came back empty. Against a perfectly correct handler, again.

newRouter().ServeHTTP(resp, httptest.NewRequest(...))   // seed → a NEW store
resp = httptest.NewRecorder()
newRouter().ServeHTTP(resp, httptest.NewRequest(...))   // assert → ANOTHER new store

The model had started building a fresh router — and therefore a fresh, empty store — inside every request. The POST that seeds the precondition wrote to a store thrown away on the next line.

I caused that. My new rule said every ServeHTTP call gets its own *http.Request, and my example called h.ServeHTTP(…) without ever showing where h came from. Read next to the older rule — ISOLATE STATE: construct your OWN fresh instance — the model did the consistent thing and freshened the router too. The rule was right; the example failed to pin the invariant it needed preserved.

Which is this report's own thesis, collected from its own author. It is not that the model is careless. It is that a rule lands on a noun, and if you do not say which noun, the model will pick one — and it will pick consistently, and it will be wrong. Two things are fresh at two different rhythms, and the default now says so out loud: the router and its store, once per test function, because that is what isolates one test from another; the request and the recorder, once per call, because a body is drained by its first read. The example now opens with h := newRouter() and threads that same h through both requests. The model writes it straight back.

Nothing but the sweep would have caught it. That is the argument for paying for one every single time a global default moves. Final state: eleven specs regenerated, eleven green, the gate chain still driving five of the archived red projects to green on its own, and the config seam — the code path every deployment goes through, which no test had ever executed — now covered in all three backends.

All training, serving, benchmarking and Builder runs are local on an M1 Max with Apple MLX — total cloud spend: $0. The gates, the go/ast rewriters, the audit, the archive and the Builder loop: github.com/guildlm/builder.