Research Log

Yesterday I switched off a prompt default because its benefit refused to reproduce. Today I found out why it had ever seemed necessary: it was a patch for a spec that named zero of its test functions and never said where a short code comes from — so the model reached past the API, planted a link, and asked for a code it invented. 404, against handlers that were perfectly correct. Naming the eight tests and showing six lines took the suite from 2 tests to 8 and killed the failure class at the root; with the spec fixed, the default measures an exact null, four arms out of four. Then a second spec refuted the generalisation I was building on — and both of the day's wrong turns came from my own greps, not the model. All three real gains were invisible to green and to coverage, which are the only two numbers this project reports.

I shipped a prompt default in the morning on a clean A/B — 42.4% coverage without it, 75.4% with it, same session, arms isolated — and turned it off by evening. Nothing about the rule changed in between; the baseline did. Chasing that number cost three retractions in one day and produced the measurement this project needed: the server is deterministic within a process and different across processes, so the +33 that bought the rule was measured against a baseline belonging to one process's state, not to the model — and no A/B I have ever compared across sessions was comparable. Along the way best-of-N turned out never to have worked: 2,905 draws, 320 dirty redraws, zero rescues, while the project's own notes marked it done. What survives is worth more than the rule: the model drops tests a spec names, and coverage cannot see it.

I went in to answer one question — were the week's gains the machine generalizing, or ten specs memorized? A brand-new domain, a double-entry ledger the corpus had never seen, answered a better one. It did not fail once; it failed in a stack, each defect masking the one beneath it, because a compile error in a dependency hides every package that imports it. Peeling it took eight runs. Two layers were the machinery's and it cleared them live — a package-qualifier gate, and the deaf-routing widener this same session had taught to hear money.Money. Underneath was a runtime long tail: every run broke a different domain method the spec had described instead of shown — a sentinel, a deadlock, a balance read off the wrong object — and each fell to showing the code. One layer was a deadlock the model could not fix; the deepest was a spec contradiction no model could ever satisfy, invisible for the entire experiment because the code never ran far enough to reach it. Green twice, at 84.9%. When the code finally runs, the question is not whether the model overfit — it is whose bug you find, and mostly it was not the model's.

Yesterday's ten greens were ten single rolls of a stochastic model, and this project's bar is two. Re-running all ten earned the number — and the shape of the result proved two things at once: every spec I had touched gained (+11.2, +13.5, +11.8), every spec I had not came back flat to the decimal (88.8 → 88.8, 88.0 → 88.0). The same measurement that confirmed the gains ruled out a regression in the rest. Then I got this report wrong: I watched two defects vanish from the hardest spec and credited the model — and the log had named two gates on the very next line. An outcome does not name its cause. In a report arguing "measure it, do not believe it", I asserted something I had not measured.

Coverage is not a metric. It is a bug detector. Chasing uncovered blocks across ten Go backends turned up a paginate that panics on any ?offset=-1 and takes the server down; a validator that never trimmed, so a title of three spaces was accepted; a validator that was never called — not by Create, not by Update, dead code behind a status-code mapping with nothing to map; and a router missing POST /projects, so a project could never be created and half the API answered 405. All four shipped. All four sat under a suite that had been green for a week. And all four were hiding in the same place: a branch the spec never named.

I went in to raise coverage in one package. I came out with a live runtime panic that had been shipping green for a week — paginate had no negative-offset clamp, so any request with ?offset=-1 took the process down, and the spec itself said "never panic". The model had written that guard correctly in one run and dropped it in another, and nothing noticed, because nothing ever asked. Then three distinct ways a passing test can be a lie, all of them from the spec: an assertion that cannot fail, one that cannot run, and one that was never asked for. And a fix loop that had spent a week throwing away half of every default this project ever wrote. Six escalating versions of a prompt rule could not stop one bug. Two go/ast gates did — and the deterministic layer doubled, from 5 red projects driven to green with no model involved, to 10.

The coverage sweep the last report promised. Two of three specs went red — and not one of the reds was the model's fault: every one was a test that no correct implementation could have passed. An invalid state made unreachable by the spec's own defaulting rule; a row expecting nil inside a loop that demands an error; and a request replayed after its body was already drained, so a handler that maps ErrExists to 409 perfectly correctly answers 400. That last one had a rule against it already — a rule that named the mechanism and predicted the exact wrong status code — and the model walked into it anyway, because the rule named the body and the model reused the request. A gate is only as good as the sentence it listens for; a default is only as good as the noun the model acts on. Then the ruler itself lied three times, the last time because the line-shifting gates poison their own scoreboard — caught only because the number it produced was arithmetically impossible.

Three specs went red. In all three the gate already existed, and in all three it repaired the file perfectly the moment it was handed it — and never fired, because the compiler had reported the same defect in a sentence it wasn't listening for. From the outside, all three looked exactly like a model that isn't good enough. Then the failure artifacts the harness used to delete become a corpus, and the corpus becomes a test suite: the gate chain now drives five of eight red projects all the way to green on its own, no model invoked — a number that was zero a week ago, and was reported as zero. And a new measurement that is not flattering: the biggest projects are green while running a quarter of their own code.

Every spec in the suite, one run each, no retries: 19 of 22 green. Then the three failures get read, and the last one is not a model failure at all — the fix loop kept dying on a line no model had written. The gates had written it. A gate that inserts an import shifts every line below it, and the next gate, still holding the compiler's stale numbers, repairs the wrong line. True since the day there were two kinds of gate, and it presents as "the loop failed to converge" — which reads exactly like a model that isn't good enough. The chain is split into phases; the property test written to pin the split fails on its first run, three more times. Also: the last hard compile class finally gated (a two-value call in a single-value context, repaired off the enclosing signature), and a green build found to be hiding an empty file the spec had asked for.

Gates get written one at a time, by whatever failure is in front of you — a fine way to start and a poor way to continue. So before writing a sixth, I audited the five against 138 real broken artifacts and ranked what was left. The ranking collapsed: counted by distinct spec, no ungated compile class is hit by more than two. The mechanical surface is covered; stop hunting gates. The audit also caught the harness deleting its own evidence (every run rm -rf'd the artifact a gate was verified against), and caught its own first ranking being wrong. With no gate left worth writing, the leverage moved upstream: a prompt default that had been teaching the model the failing pattern for six specs running, and a spec linter that — replayed against history — catches 4/4 of the runs that actually failed.

One stochastic failure was left standing in the suite, and it was a single shadowed identifier: for _, t := range tasks steals the *testing.T and every t.Fatalf stops compiling. It is the one gate a regex cannot write — the same letter means the Task on one line and the tester three lines down — so it runs on a real go/ast scope walk and refuses the file on anything it cannot resolve. Then we ran it, and it never fired: the compiler was stopping earlier, and the shadow was invisible behind it. Pulling that thread found three defects in the machinery, none of them in the model — a fix loop routing to the wrong file, a root-cause widener that believed the shadow was a genuinely missing method, and a regex that had been quietly truncating symbol names — and a spec that was manufacturing its own failure by demanding a name the model reserves for something else. Fixing the ask greens the spec 2/2. The subtractive probe is recorded as inconclusive.

After two gates and a set of defaults, breadth: point the system at Go the suite never touched. Eight new stdlib-only specs — LRU cache (container/list), min-priority queue (container/heap), generic Set (type parameters), recursive-descent expression evaluator, channel-based pub-sub event bus, word-packed bitset, custom JSON codec, and a crash-recovering write-ahead-log KV store. Seven greened first-generation on the unchanged 7B, including three picked as small-model traps (bit off-by-ones; the MarshalJSON that calls itself forever; a log replay that must survive a reopen). The LRU surfaced a runtime panic the compiler waved through — a hand-built *list.Element double-wrapped by PushFront — an idiom for a prompt default, not a gate. A regression sweep confirms the session's gates and defaults break none of the existing suite.

Report #13 closed the suite by hardening each spec by hand — but a lesson in one spec's YAML only helps that spec. This promotes three recurring authoring rules (field-named struct literals, interface/impl parity, register-the-method-value) into the Builder's permanent, conditional prompt defaults: unit-tested, regression-safe, 171 tests green. Then it tries to prove they're load-bearing by deleting the hardening from a spec — and the probe fails, usefully. It separates what a prompt nudge can carry (idioms) from what it can't (which methods must exist — a completeness failure the model commits by dropping a method from both an interface and its impl, which only a deterministic gate can catch). A negative result that marks exactly where the next lever goes.

Report #12 cleared workapi by making one test's ask explicit. This turns that lever loose on the rest of the suite: the six Go backends the system could never autonomously green — logstats, shortener, kvservice, tasks-api, ratelimit, workapi — are now all green, the 7B completely unchanged. Almost every "model ceiling" was an under-specified ask made explicit (a dropped step, an uncomputed test oracle, a positional literal, a mega-test, a missing method, a mis-wired router, a clock injected after construction); the genuinely mechanical ones became two new deterministic gates. The moat is the system, and it builds real Go backends green for $0.

Report #11 peeled our hardest spec — an eight-package concurrent, bearer-authed backend — to build-clean and vet-clean and stopped at one wall we called the model's own ceiling: the router integration test. It wasn't the model's ceiling. It was the shape of the ask — one long function demanding thirteen cases exceeded the 7B's single-function synthesis budget. Split it into ten small focused per-endpoint tests — better Go practice anyway — and the same unchanged champion writes every one correctly. workapi goes fully green: build, vet, and test-race across all eight packages, model untouched, reproduced from scratch, $0.

Report #10 shut the door on making the model bigger, so we pushed the other lever all the way. On the hardest spec — an eight-package concurrent, bearer-authed backend — six deterministic gates and verified retrieval contracts, model completely unchanged, peeled its walls one by one until it compiled and vet-passed clean across every package, stalling only at the documented 7B integration-test-logic ceiling. The champion's green backends stayed green (regression-checked). Plus three ways a retrieval corpus quietly poisons itself — ranking contamination, name collision, helper leakage — each caught and fixed.

Report #9 named the next frontier: make the hard, large backends pay into the self-distillation set with a longer training budget. We tried it — sequence length 8192 to keep the big cross-package prompts. The budget worked (58 → 74 pairs, workapi included); then the model overfit them and regressed — 27/48 and 23/48 on the unseen bench vs the 33/48 champion — and still failed the spec it trained toward, at the same middleware wall. An honest negative: the champion stands, and the remaining wall is the algorithm's, not the model's.

Eight reports said the moat is the algorithm, not the model — but the goal is the best Go model. So we closed the loop: the Builder's compiler-verified green runs become training data and the model learns from its own output. Turn one regressed (14 pairs → 26/48); with diversity and a longer budget, turn two passed the champion — 33/48 vs 31/48 on the unseen bench, and still 3/3 on the project, converging cleaner. The first net gain from on-policy self-distillation, at $0.

Report #7 blamed the last gap on the model and bet on pretraining. We ran the bet: two DAPT recipes came back net-negative and a four-times-bigger base was rejected — while the exact semantic gap that needed 20 hand edits fell to two verified retrieval examples, model unchanged. All four backends are now autonomous 3/3. The moat is the algorithm.

Same 7B, never retrained — 7→11→13→16 files, one package to six, by turning every recurring failure into a deterministic gate. Three specs fully-autonomous green; the fourth stopped 20 human lines short, and the gap is precisely the model's half of the bargain: semantic expected values. That half is now in (free-GPU) training.

Base vs specialist through the same loop on a real HTTP service — both stall at 2/3. The fix wasn't a bigger model or the fine-tune; it was two verified retrieval examples (→ 3/3). The corpus, not the algorithm, was the bottleneck.

Two trained Go specialists served live ($0, MLX), wired into the Builder with role-routing — one writes the code, another the tests. It builds a verified backend first try (score 3/3). A second spec exposes the real gap: the base model's knowledge, not the scaffolding. Plus two shipped upgrades.

We made the benchmark harder to cut noise. Clean result: a code-specialized 7B beats a general one robustly (+3/+4) — but the win is the base, not our fine-tuning. The lever for code-gen is base choice + the agent loop.

All three Go specialists trained and benchmarked, $0. One clear win (go-test +2), one marginal (go-dev), and one honest task-mismatch (go-review) that taught us to measure each specialist on its own job.

The go-test specialist beats its base at catching planted bugs (8/13 vs 6/13) on a mutation-testing benchmark — a clearer win than code generation, and a lesson in where small-model specialization pays.

Trained a go-developer 7B locally for $0. On a 19-task hidden-test benchmark it beats a general 7B (15/19 vs 12/19) — the thesis validated. Plus an honest negative: the 14B base did worse, not better.