A stack of masks

The last report left a question hanging that its own numbers could not answer: ten specs are green twice over, but every one of them had been reworked this week. Are the gains the machine getting better, or ten specs slowly memorized? There is one honest way to ask — take a domain the corpus has never seen, write it once, and run it. So I did. It did not answer the overfit question. It answered a better one, and the answer came in layers.

The held-out domain, and an honest correction

ledger: a standard-library-only HTTP backend for double-entry accounting — a money primitive, domain models, a store, a service enforcing the accounting invariant, an HTTP layer, and a graceful-shutdown main. Five packages, nineteen files, nothing like it in the training corpus. The intent was "write it once against the checklist, never tune it, run it once."

It did not stay that way, and pretending otherwise would be the whole mistake. Closing it green took five runs and a series of spec fixes. The value here is not "it passed untouched." It is the stack of distinct failures a fresh domain surfaced, and — the part that matters — who owned each one.

Everything was masked

The structural fact that shaped the whole experiment: a compile error in a package hides every package that imports it. money is imported by models, store, service and api. So for as long as money failed to compile, every bug downstream of it was invisible — the build never type-checked far enough to see them. Each fix peeled one layer and revealed the next. What looked like "one stubborn spec" was a column of independent defects, stacked, each wearing the one above it as a mask.

run 1-2   money.go            compile error   // masks models, store, service, api
run 3     models.go           total.Add()     // revealed once money compiled
run 3     store.go            models.Money    // gate cleared it
run 4     store (runtime)     DEADLOCK        // revealed once it all compiled
run 4     service (runtime)   InsufficientFunds contradiction
run 4/5   store (runtime)     sentinel vs errors.New

Two layers were the machinery's — cleared live

The model kept qualifying the money type as models.Money instead of money.Money — a name-similarity slip in four places in store.go. A gate written weeks ago repaired it in round one, across seven files at once, no model call. And the moment the build compiled far enough, models.go tried to sum postings with an invented method on an int64:

total.Add undefined (type money.Money has no field or method Add)
  widening fix targets to the declaration of type Money (missing method Add)

That second line is the deaf-routing widener — and it fired on the qualified form type money.Money has no method Add. Earlier this same session I'd found that widener was deaf to exactly this shape: its regex only recognised the bare type Store has no method, and missed both *service.Ledger and money.Money. I widened it and wrote a test. Here it was, firing live on a defect it would have slept through a day ago. A gate is only as good as the sentence it listens for — and this one had just learned a new sentence.

panic: boom was a lie at the top of the log

Then the build compiled clean and a test failed at runtime, and the log showed this at the top:

! 2026/07/15 14:18:45 panic: boom
  widening fix targets to package impl in internal/api (persistent runtime failure)

For two fix rounds I read that as the failure — a panic escaping the recovery middleware. It was not the failure. It was a log line. panic: boom is what the Recover middleware prints — log.Printf("panic: %v", err) — when it correctly catches the panic and returns 500. TestRecoverMiddleware passes. A recovered panic doesn't crash; it logs. The real failures were further down, where the log display had truncated them off the top. Only running the tests myself surfaced them. It is the same lesson as a deadlock that reached the model wearing the word "timed out": the loudest line in the output is not the cause of the failure, and a log that shows you the top of a stack is not showing you the bottom.

The deadlock the model could not fix

Underneath panic: boom, a store test hung for the full timeout and Go printed the goroutine dump — which named the bug in two lines:

store.(*MemStore).GetAccount(...)         // takes RLock
store.(*MemStore).CreateTransaction(...)  // already holds Lock

CreateTransaction took the write lock and then, still holding it, called GetAccount, which takes a read lock on the same mutex. sync.RWMutex is not reentrant, so the goroutine wedged forever. The deadlock-surfacing fix from earlier this session worked exactly as designed — Go hit its own timeout before the harness killed the process, so the naming trace reached the log. But surfacing is not fixing: across five rounds the model could not restructure the locking. That earned a spec-level pin — inside the held write lock, check s.accounts[id] directly; never call a method that re-locks. A stochastic concurrency bug the compiler cannot see, made deterministic by naming the discipline the model kept missing.

The one no model could satisfy

And then the deepest layer, the one that had been hiding under every compile error for the entire experiment: a contradiction in the spec itself. The service's Post was shown with an overdraft check — reject any posting that drives a balance below zero:

if balances[p.AccountID]+p.Amount < 0 { return ErrInsufficientFunds }

But this is a double-entry ledger: every balanced transaction has postings that sum to zero, so one side is always negative. That check rejects every balanced transaction that ever existed. And the two tests meant to pin it could only be told apart by account identity:

TestPostBalanced          cash +500, rev -500   → expect SUCCESS   // rev goes negative, fine
TestPostInsufficientFunds cash -100, rev +100   → expect FAILURE   // cash goes negative, not fine

"rev may go negative, cash may not" is a real accounting rule — assets can't overdraw, revenue accounts carry credit balances — but it requires an account type that models.Account{ID, Name} never had. There is no rule on balance and amount alone that passes both tests. No model could satisfy this spec, because the spec asked for something arithmetically impossible without a field it never defined. I wrote that contradiction, in a past session, and it survived a week of green runs — because the code never once ran far enough to reach it. I resolved it by removing the ill-specified feature; the correct alternative, account types, is a larger redesign for another day.

A green suite tells you the tests that exist pass. It cannot tell you the spec is consistent — and a contradiction downstream of a compile error is invisible until the day the compiler lets the runtime reach it.

What was actually the model's

Strip out the machinery's two layers, the deadlock, and the spec contradiction, and what remains that was genuinely the model's own limit was small and of one kind: numeric and mechanical choices the spec described instead of showing. Money.String() dropped the leading "0." and then the sign; Parse lost the sign on negatives; the postings sum used an invented .Add; the store returned errors.New(...) where the tests matched a sentinel with errors.Is. Every one of them fell to the same remedy: stop describing the behavior and show the code — the whole of String() and Parse(), the total += p.Amount operator, the models.ErrNotFound sentinel returned directly. Describe a trap and the model walks into it; show the code and it copies it.

The runtime long tail

There is a second masking structure underneath the first, and it is the one that actually decided how many runs this took. Once the project compiled, it turned out that every run failed at runtime on a different domain method — and always a method the spec had described rather than shown:

run 4   the store returned errors.New(...) where a sentinel was expected   // self-fixed round 2
run 5   CreateTransaction called GetAccount under its own lock → deadlock  // model could not fix
run 6   the service read a balance off the Account, which has no balance   // red

Every one of them is a numeric, stateful, or concurrency method — the exact class the money bug belongs to. The standard parts — the HTTP handlers, the router, the simple delegations — reproduced cleanly every time. So the reliability of a nineteen-file project is not "can the model synthesize nineteen files." It is "how many of its mechanical methods are shown versus merely described" — because at nineteen files, some run will always roll the one you only described. I closed the tail the only way that has ever worked here: I showed each method's code, one at a time, as each run surfaced it — the sentinel returned directly, the lock-free account check, the balance read from the store's map.

The compile layer belongs to the gates, and they clear it. The runtime long tail belongs to the spec author: every method with a trap in it has to be shown, because describing it only sets the odds of which run walks into it.

One of those pins I first put in the wrong place, which is its own small lesson. The lock discipline went into the file the spec says owns the store implementation — but the model consolidates the whole store into a different file and leaves that one empty, so the pin was attached to nothing, and the deadlock came back on the reproduction run. A spec pin is only as good as the file the model actually reads it on, and only as good as whether it shows the code or merely describes it.

Validated before it was spent

One method note, because it is the discipline that kept this honest. Before spending a thirty-minute model run on any fix, I snapshotted the red artifact, applied the fix by hand, and ran go test ./... and go test -race on it. Green on the snapshot proves the spec is satisfiable — that the fix I'm about to ask the model for actually works — before I ask for it. A reference implementation is not the model succeeding; it is the proof that the model can.

Result

The held-out ledger is green, and green twice: two independent runs, build, vet and the race detector across five packages, both converging on the first fix round, both at 84.9% of their own code executed. It took eight runs to earn that — the first green, on run four, was a lucky roll that dodged the deadlock, and the number only became trustworthy once every mechanical method was shown, not described. The overfit question turned out to be the wrong one. A fresh domain is not a single test you pass or fail; it is a stack of masked failures — a compile layer the gates clear and a runtime tail the spec author must close, one shown method at a time. Two of its layers were the machinery's and it cleared them live; one was a deadlock the model couldn't fix and a shown block could; one was a spec contradiction no model could ever satisfy. The genuine model residue was a handful of numeric and stateful traps that a shown line of code dissolves. When the code finally runs, the question is not whether the model overfit. It is whose bug you find — and layer by layer, mostly, it was not the model's.