The wheel closes — the model learns from its own verified output
Report #8 ended on a hard-won conclusion: the moat is the algorithm, not the model — retrieval and gates cracked the frontier that pretraining and a 4× base could not. True. But it leaves an itch, because the actual goal is the best Go model, and the model had spent five reports refusing to improve. So we asked a different question: can the model get better from its own output — no teacher, no new human data, no cloud, no dollars? Every time the Builder finishes a spec, it has produced something rare: a prompt paired with a compiler-verified answer. Feed those back and the algorithm that surrounds the model becomes a data pump for the model itself. The first turn of that wheel regressed. The second passed the champion.
The wheel
On-policy self-distillation, built entirely out of parts we already had. The
Builder runs a spec; with tracing on, every file it writes is logged as a
(prompt → file) pair. A harvester keeps only the pairs from runs
that ended green — go build + go vet
+ go test -race all passing across every package — so the
training signal is never "what the model said," only "what the model said
that the compiler and the tests both accepted." A prep step prunes
each prompt down to the training budget (the retrieval examples are dropped on
purpose — the point is to bake the contracts into the weights, not to keep
leaning on the corpus), and the survivors are mixed back into the same SFT
recipe that made the champion. The model trains on its own best, verified,
on-policy work. Cost: $0, local, on an M1 Max.
Turn one regressed — honestly
The first build of the wheel (call it v3) harvested a thin,
low-temperature farm: 14 distilled pairs, oversampled and
trained at a 2048-token sequence length. The number told the story before the
bench did — at greedy temperature the Builder writes nearly the same files
every run, so "more rounds" bought almost no new data. Trained on those 14, the
model memorised a narrow slice and lost ground on everything unseen:
26/48 on the 48-task unit benchmark, five points under the
31/48 champion it was built from. A clean negative, and a clear diagnosis:
distillation with no diversity is just overfitting with extra steps.
Turn two: diversity, and a budget that fits
Two changes, both aimed at the same failure. First, diversity: re-run the farm at sampling temperature 0.35 so the green trajectories actually differ — genuine new pairs instead of byte-identical reruns. Second, a budget that keeps them: train at a 4096-token sequence length so the longer, real Builder prompts survive pruning instead of being dropped. The harvest went from 14 usable pairs to 58 — three times the data, and longer. Same base SFT mixture, same recipe, only the distilled slice changed. We measured the result three ways: the unseen unit bench, a spec the model had trained on, and a spec held out entirely.
| Model | Distilled data | Unit bench (unseen) |
|---|---|---|
| Base 7B, untuned | — | 39/48 |
SFT champion (v2) | none | 31/48 |
Self-distilled v3 | 14 pairs · seq 2048 | 26/48 |
Self-distilled v4 | 58 pairs · seq 4096 | 33/48 |
33/48. The wheel didn't just undo v3's
regression — it put the model two points past the champion it was
distilled from, on a benchmark of tasks that appear nowhere in its
training data. The model taught itself something that generalises, from output
it had generated and the compiler had verified. This is the first time in nine
reports that the model — not the scaffolding around it — measurably
moved the number in the right direction.
Did it keep the job it already had?
A unit-bench gain is worthless if it costs the format and spec cooperativeness
that Report #8 showed is the SFT adapter's whole reason to exist. So we re-ran
the project metric. On taskapi — the five-package backend the
champion finishes 3/3 — v4 went 3/3 as well, and cleaner:
all three runs converged with deterministic fixes only, zero model-fix
rounds, where the champion had needed two.
On taskapi | Unit bench | Project | Convergence |
|---|---|---|---|
SFT champion (v2) | 31/48 | 3/3 | two model-fix rounds |
Self-distilled (v4) | 33/48 | 3/3 | deterministic-only |
And the honest ceiling, unmoved: workapi, the eight-package spec
with concurrency and auth that the champion itself only ever reached green
stochastically, stayed not-green for v4 too — undone by
the same 7B test-writing residuals we have reported for months (a test that
invents a method the interface doesn't have; a worker test that reads its
counter before the workers drain; a middleware chain wired with the wrong
arity). The implementation compiled across most packages; the tests are still
the wall. No regression, no breakthrough — the shared ceiling of both models.
Why it worked the second time, and where it still leans
The mechanism is not mysterious. Verified pairs are a clean signal precisely
because the filter is a compiler, not a preference model — a pair only enters
the data if the code built and the race detector was quiet. What broke
v3 and fixed v4 was never the idea; it was
coverage. Fourteen near-identical pairs teach memorisation;
fifty-eight varied ones teach a distribution. But the same measurement exposes
the next limit, and we would rather name it than hide it: the harvest still
self-selects toward the easy specs. The wheel eagerly produces pairs
for the small, fast backends; the large ones — workapi chief among
them — go green too, but their long cross-package prompts blow the training
budget and get dropped. The diversity we added is real but largely
within a spec, not across new domains. So v4 is a first
net gain, not a solved problem.
What we learned
- A model can bootstrap off its own verified output. No teacher model, no purchased data, no cloud GPU — the Builder's green runs are themselves a training set, and training on them beat the model that generated them on unseen tasks. The compiler is the label.
- Distillation without diversity is overfitting.
v3's 14 greedy-temperature pairs regressed five points;v4's 58 varied ones gained seven back and then some. The lever is coverage — sampling temperature to make the trajectories differ, and a sequence budget long enough to keep them. - The gain kept the job. The self-distilled model is two points stronger on the unseen bench and still 3/3 on the project, converging cleaner than the champion. Capability and cooperativeness moved together, not against each other.
- The next frontier is cross-spec coverage. The wheel favours the specs that are cheap to green. Making the hard, large backends pay into the training set — a longer budget, or harder prompt pruning that keeps on-policy fidelity — is the next turn.