The backlog was empty — auditing the gates, then moving upstream

Gates get written one at a time, by whatever failure happens to be in front of you. That is a fine way to start and a poor way to continue: you have no idea whether the thing you just fixed was the common case or a curiosity. So before writing a sixth gate, I stopped and measured — and the measurement said not to write it.

The corpus was already there

Every failed Builder run leaves its artifact on disk. That pile is not garbage: it is 138 real, model-written Go projects, most of them broken, each carrying whatever the 7B actually gets wrong. It is the closest thing this project has to a ground-truth dataset, and nobody had ever read it as one.

The audit copies each artifact, runs the gate chain to a fixpoint, and asks what is left. Residual compiler diagnostics get normalised to their shape — names and line numbers filed off — and ranked. It separates the two backlogs that always get conflated: artifacts that still do not compile (a gate could still help) from those that compile and fail a test (only the spec or the model can). It needs no model server, so it is free.

It caught the harness eating its own evidence

The first run reported the middleware gate as never firing — a gate I had verified by hand, on a real artifact, hours earlier. The artifact was gone. _ab_run.sh rm -rfs the output directory at the start of every run, so a broken project survived only until the next roll of the same spec.

That is worth saying plainly: the system was deleting the only hard evidence of what the model gets wrong — the thing every gate is verified against, and the only honest input to deciding what to build next. Failing runs are now archived. Failures are the corpus.

Then it caught its own first answer being wrong

The initial ranking put too many arguments in call to newAPI at the top, on four artifacts. All four were the same spec, rolled four times. Counting artifacts lets one spec's quirk outrank a class that is genuinely broad — which is the exact opposite of what a ranked backlog is for.

Counted by distinct spec, the whole ranking collapses:

=== residual COMPILE classes, by how many DISTINCT SPECS hit them ===

    2  undefined: store
    2  operator ! not defined on X() (value of type string)
    2  tt.X (type []models.X has no field or method X)
    1  too many arguments in call to X
    1  X redeclared in this block
    1  … (everything else)

Nothing exceeds two specs. Most classes are hit by exactly one. There is no broad ungated compile class left. The five gates cover the mechanical surface, and what remains is a long tail of one-offs. Of the thirteen broken artifacts the current system produces, four already compile clean and fail only an assertion — past where any gate can reach.

That is a negative result, and a load-bearing one. It says: stop hunting gates. Which is not a conclusion I would have reached by looking at the next failure, because there is always a next failure.

Upstream, then

With the mechanical surface closed, the leverage is in the specs — and two findings there were worth more than another gate.

The Builder's own default was teaching the model the failing pattern. Six specs in a row died the same death: a duplicate → 409 case that got 201, a GET existing → 200 case that got 404. The ISOLATE-STATE prompt default told the model, correctly, that every test case must build its own fresh instance of the system under test. That is half a rule. A fresh instance is empty — so a "duplicate" case has nothing to duplicate, and it fails no matter how correct the handler is. The model obeyed literally and the default walked it into the bug, six times, while I patched the specs one at a time. The rule is now whole: isolate state, then seed it.

And a spec that no implementation can satisfy should not cost a GPU run. Every expensive debugging session this week ended at a contradiction that was sitting in the YAML the entire time. So the Builder now lints the spec before it generates anything. The rules are just the failures, written down: a fresh instance per case that also wants a duplicate; a test calling a constructor no implementation promises; one name asked to be both an interface and a struct; and a purpose that forbids the model's idiom ("do NOT model it as an interface") — because empirically the spec loses that argument, and the ban only decides how it breaks.

A linter that passes everything proves nothing, so it was replayed against history: checked out at the commit before each fix, it catches 4 out of 4 of the runs that actually failed. And on its first pass over the current suite it immediately found two more instances of the seeding trap that my own hand-grep had missed, because they were phrased differently.

The arc here is the whole thesis in miniature. Writing gates is patching. Reading the corpus, ranking what is left, discovering the backlog is empty, and then fixing the thing upstream that was generating the failures — that is building a system. The moat was never the model, and this week it wasn't even the gates: it was a default that was wrong, a harness that deleted its evidence, and four specs arguing with a 7B they were always going to lose to. The Builder's test suite stands at 253, on an unchanged model, at $0.

Postscript — three more mechanisms that only looked like they worked

Once you start counting instead of assuming, it is hard to stop. Three more, each found the same way.

Best-of-N had never selected anything. The Builder samples two candidates per file and keeps the clean one. Grepping the logs: 230 instances of "no clean candidate; using last of 2", and zero of "kept candidate". Not a bias — a total absence. The check that decides whether a candidate is clean rejects any file importing something non-stdlib, which is how a small model gets stopped from reaching for gorilla/mux. But a project's OWN packages carry a domain too — guildlm.dev/workapi/internal/store — so in every multi-package project, every file looked like a foreign dependency. No candidate was ever clean, and best-of-N quietly degenerated into "keep the last sample", which is not a selection, it is a coin flip. It had been drawing a second sample and throwing it away for months. Measured against artifacts that are green — they build, vet and pass -race, so they are as clean as Go gets — workapi went from 7/18 files judged clean to 17/18, and taskapi from 4/12 to 11/12.

Every multi-package artifact was shipping a dead file. workapi's green artifact contains internal/store/memory.go. It is one line: package store. The spec asked it to hold the goroutine-safe MemStore; the model wrote MemStore in store.go instead, so memory.go had nothing left to declare, the redeclaration stripper emptied it, and it shipped. Go does not care which file in a package holds what, so the build is green. All four multi-package artifacts carry one of these. A green build is not the same as the project that was asked for, and the system had no way to tell the difference. It does now — the check runs when a build goes green and says so.

And the retrieval corpus was teaching the wrong shape to the file that kept failing. Retrieval itself audits clean: 36 verified examples, every file gets two shots, all role-matched. But for internal/store/memory_test.go — a store unit test — the two examples it hands the model are an HTTP router integration test and an HTTP CRUD test. Wrong domain, and the top one demonstrates the shared table with subtests that the spec and the prompt default now explicitly forbid. The corpus had no store-unit-test contract at all, so the model was getting contradictory instructions: the spec said separate focused functions, and the examples showed a table. One new verified example — separate, self-seeding, errors.Is on the sentinels — and every store test now retrieves the shape it is being asked for.

That is six mechanisms in one week that looked like they were working and were not: a harness deleting its own evidence, an error surface that hid the bug from the gate written to fix it, a prompt default teaching the failing pattern, a best-of-N that never selected, a green build hiding a dead file, and a corpus contradicting its own specs. None of them was a model limitation. Every one of them was invisible until something counted it. If there is one lesson from the week, it is that: a system you have not measured is not a system you have.