The wheel overfits at long sequence

Report #9 ended by naming the next frontier out loud: the self-distillation wheel favours the specs that are cheap to green, and the hard, large backends — workapi chief among them — go green but blow the training budget and get dropped. The proposed fix was a longer budget, so the big cross-package prompts survive pruning and pay into the training set. This report is that experiment. The longer budget did exactly what it promised: it captured the hard pairs. Then the model overfit them and regressed — and still couldn't crack the spec it had been trained toward. A clean negative, and a sharper picture of where the wheel breaks.

The lever: sequence length 8192

The champion, v4, was distilled from 58 verified pairs at a 4096-token sequence length. That budget is exactly why workapi never made it in: its prompts carry the exported API of eight packages, so the real Builder prompt runs 16–23k characters and the prep step drops it. The single-variable change here is to re-prep the same harvested pairs at a 8192-token budget and retrain on the mix — same base SFT data, same recipe, same ×4 oversample, same 1200 iterations. Only the sequence length moved. Gradient checkpointing kept the peak under 15 GB; the whole thing ran local on an M1 Max. Cost, as always: $0.

The budget did its job

This part worked. At 8192 tokens the harvest went from 58 pairs to 74, and the new arrivals were precisely the ones we wanted: the worker-package trajectories — the concurrent background processor that had never fit before — jumped from a handful of lines to nineteen. The wheel finally had the hard, large backend in its training set. If the story were only about coverage, this would be the win. It isn't.

And the model overfit it

The validation loss told on it before the benchmark did: it fell to 0.484 at the two-thirds mark, then spiked to 0.719 as training pressed on — the signature of a model memorising a small, oversampled slice. The unseen unit benchmark confirmed it, and not marginally.

27/48 — six points under the champion it was built from. And the earlier, lowest-validation-loss checkpoint was worse, not better: 23/48. There is no checkpoint of v5 that beats v4; the whole run is a regression. The model didn't just fail to gain — it lost skills the champion had, dropping is_palindrome, sort_by_len, merge_intervals, caesar, top_k_words, and — pointedly — middleware_chain, the very pattern the hard spec needs.

Did the overfit at least crack its target?

If a model trades away general competence to specialise, the specialisation had better pay. So we asked the one question that could still redeem v5: it trained on workapi's own verified trajectories — did it finally finish workapi? We served it and ran the eight-package spec end-to-end. The answer is no. After seven fix rounds it stalled not-green, blocked at the same wall as the champion: router.go wires a middleware with the wrong arity — Recover(logger) where the signature wants (next http.Handler, log *slog.Logger) — the identical residual we reported for v4, surviving even though a verified middleware-chain contract sits in the retrieval corpus. Two packages did pass clean, including worker, the concurrency code v5 had trained on. But training on the target neither generalised nor closed it. The overfit bought nothing.

Why it overfit — and what the wall actually is

The mechanism is the same one Report #9 warned about, now measured at the limit. The harvested pool is small and skewed — heavily taskapi and taskflow, with workapi a thin, freshly-added minority. Oversampling that pool ×4 and stretching every example to 8192 tokens doesn't teach a broader distribution; it drills a narrower one, harder. The longer budget solved coverage and created an overfitting problem in the same move. And the wall it ran into is the tell: the workapi blocker is not missing Go knowledge that more training could add — it is a deterministic-gate gap. The model keeps calling a two-argument middleware with one argument; no gate rewrites call-arity, so no amount of weight-baking fixes what is fundamentally a scaffolding job. That is a Builder feature to write, not a model to retrain.

What we learned

• The wheel overfits at long sequence. Sequence length 8192 captured the hard cross-package pairs the champion's budget dropped, exactly as intended — and every checkpoint of the resulting model landed below the champion on unseen tasks. Coverage and overfitting moved together.
• A skewed pool ×4 is the culprit, not the length alone. With 74 pairs dominated by a few easy specs, oversampling memorises. The fix isn't a longer budget on the same data — it's a lower oversample, genuinely cross-spec diversity, and stopping before the validation loss turns up.
• Training on the target didn't crack the target. The overfit model still failed workapi, at the same middleware wall as the champion. Specialising off its own trajectories neither generalised nor closed the specific spec — the worst of both.
• The remaining wall is the algorithm's, not the model's. The binding constraint is a deterministic-gate gap — middleware call-arity — that retrieval alone doesn't reach. The proven lever, again, is the scaffolding: a gate that rewrites the arity would harden every model, the champion included.
• The champion stands. v4 — 58 pairs at sequence 4096 — remains the best point the distillation wheel has found. The first turn that gained (Report #9) still stands; this turn marks its ceiling.