The model's half wasn't training — it was retrieval

Report #7 ended on a bet. Three of four backends went fully autonomous; the fourth, taskapipro, stopped ~20 hand-written lines short, and we called the residual "the model's half — semantic expected values, knowledge and reasoning," and said it was "now in (free-GPU) training." This report reports the bet. We ran the knowledge lever head-on — two rounds of domain-adaptive pretraining and a four-times-bigger base — and it came back negative on every axis that matters. Then the exact same gap fell to two examples in a text file, model unchanged. The half we blamed on the model was retrieval-shaped all along.

The knowledge lever, tried three ways

If the residual really were missing Go knowledge, the fix is to put knowledge into the weights. We tried the two obvious ways — adapt the base on real Go, or swap in a bigger, newer base — and measured both against the same 48-task unit benchmark and the same Builder project runs. Everything is local, MLX, $0 (the DAPT runs use a free Kaggle GPU).

DAPT recipe 1 reached exact unit parity with SFT-only (the continued-pretraining added no measurable capability at this token budget) — and at the project level it collapsed taskapi to 0/3 across four separate checkpoints, each a distinct instruction-following failure (method re-declaration, files ballooning past their length and truncating mid-string, Go code written into go.mod). Next-token training on a tiny fraction of a real DAPT budget degrades the instruct model's format discipline faster than it adds knowledge, and SFT-on-top can't fully repair it.

DAPT recipe 2 was built to fix exactly that — keep the chat behaviour alive by replaying instruct data through the pretraining stream. It helped at the margin (the raw adapted model was partially, not totally, broken) but the honest number is blunt: DAPT-replay + SFT scored 25/48 on the unit bench versus 31/48 for the identical SFT recipe on the vanilla base. Same data, same recipe, only the base differs — so the adaptation subtracted six points. The purpose of DAPT is a knowledge gain; capability went down. Both recipes are closed.

A four-times-bigger base, rejected

So try raw size. Qwen3-Coder-30B-A3B is a newer, mixture-of-experts coder, roughly four times the parameters of our 7B. On the unit benchmark it landed at 36/48 (about 38–40 once you let goimports strip the spurious unused imports it likes to write) — at parity with the untuned 7B's 39, not ahead. And on the actual job it lost outright: pointed at taskapi — the spec our 7B champion finishes 3/3 in ten minutes — the 30B exhausted all seven fix rounds at 0/3, in thirty, three times slower, undone by a self-inflicted spec violation (it defined the Task and Project types twice, in two different packages). More parameters, no better at the project; the base stays the 7B.

The dissociation that keeps showing up

Line the runs up and the same split appears every time: unit-benchmark capability does not predict who finishes the multi-package build.

The two strongest models on isolated single-function tasks are the two that can't finish the backend. Our SFT adapter is six points weaker on the unit bench and wins the project 3/3. Its value was never raw Go ability — it's format and spec cooperativeness: honouring the package layout, the cross-file contract, the "one file, no commentary" discipline the Builder depends on. Bigger, smarter, untuned models simply don't cooperate, and the unit bench can't see the difference.

What actually closed the gap: two examples

With the knowledge levers spent, we looked again at why taskapipro stuck at 2/3. The residual wasn't diffuse "reasoning" — it was two concrete, recurring implementation bugs: a config loader that read ADDR from the environment but never supplied the :8080 default, and an in-memory store whose Create never returned ErrExists on a duplicate. Both are patterns, not puzzles. So we wrote them down — two small Go files, each verified go test-green in isolation, added as few-shot examples to the Builder's retrieval corpus (14 → 16):

• a config loader with env-var defaults + validation — getenv(key, fallback), Addr defaulting to :8080, duration/int parsing, a Validate();
• an in-memory CRUD store with ErrNotFound / ErrExists sentinels (Create returns ErrExists on a duplicate id), sorted List, and a table test that builds a fresh store per subtest.

Nothing else changed — same go-dev-mixed-v2 7B, same gates. The retriever is plain token overlap between each example's description and the file being written, so the config example surfaces for the config file and the store example for the store. We re-ran taskapipro.

Autonomous 3/3. go build + go vet + go test -race, green across all six packages, deterministic convergence at fix round 3, ~18 minutes. The unchanged model wrote getenv("ADDR", ":8080") and the ErrExists-on-duplicate check itself, lifted straight from the retrieved patterns — the same two bugs that needed hand edits a session ago, now written correctly on the first pass.

We also checked whether the last recurring failure class — the 7B's buggy tests — could be routed away to a dedicated go-test specialist. On its own mutation benchmark that specialist genuinely beats base (11/18 vs 7/18 at writing a test that catches an injected bug — unlike impl-generation, the narrower test-writing skill does respond to fine-tuning). But routing left taskapipro at 2/3: its binding constraint was the implementation, not the tests. The stricter specialist tests only made the impl bug easier to see — which is exactly what retrieval then fixed.

What we learned

• The knowledge lever didn't pay at this scale. Two DAPT recipes net-negative, a 4× base rejected. Continued-pretraining on a hobby token budget costs more instruction-following than it buys in Go knowledge, and raw size buys unit-bench points that don't convert to finished backends.
• Unit capability is the wrong dashboard. The models that top the single-function benchmark are the ones that can't ship the project. What the Builder needs from the model is cooperativeness, and only the SFT adapter has it.
• The "model's half" was retrieval-shaped. Report #7's semantic residual looked like something you'd have to pretrain away. It wasn't. It was two named patterns, and putting them in a text file the model can copy from closed the gap that DAPT and a bigger base could not.
• The moat is the algorithm. Every backend on the ladder — usersapi, taskflow, taskapi, and now taskapipro — is fully autonomous 3/3 on the same unchanged 7B. The wins came from context, gates, and verified retrieval, and every one of them was free.