A gate is only as good as the sentence it listens for
Three specs went red in a verification sweep. In all three, the gate that repairs the bug already existed. In all three, it fixed the file perfectly the moment it was handed it. And in all three, it never fired — because the compiler had reported the same defect in a sentence the gate was not listening for. From the outside, all three looked exactly like a model that isn't good enough.
The same bug, a different sentence
The shadowed tester, in disguise. The gate that renames a loop
variable stealing the *testing.T watches for
t.Fatalf undefined (type Task has no field or method Fatalf). But
the model wrote:
func TestCreateOK(t *testing.T) {
...
t := models.Task{ID: tid, ...} // t is ALREADY the parameter
In a function whose parameter is already t, this shadows nothing —
a parameter lives in the body's own scope — so Go reads it as an
assignment to the tester and complains about the type instead:
cannot use models.Task{…} as *testing.T value in assignment. And
because nothing is shadowed, t.Fatalf still resolves to
*testing.T — so the message the gate is watching for
never appears at all. Same mistake. Same repair. One regex
stood between a working gate and a red spec.
A stdlib package with the module path glued on.
import "guildlm.dev/workapi/internal/slog", meaning
"log/slog". The phantom-import gate looks for a project
package that exports the symbols; slog.New lives in none, so the
search came back empty and it gave up. But the phantom's last segment names a
stdlib package, and the path is simply the real one with a module in front of
it.
In a test, an error is caught, not propagated.
if !errors.Is(Decode("!"), ErrBadCode) — a two-value call in a
one-value slot. The hoist gate refused, and correctly under its own rule: a
test returns nothing, so there is nowhere to propagate the error to.
But a test doesn't propagate an error, it catches one — and
errors.Is(X, …) tells us with certainty that X is the error
of the two returns. That is the one shape where the choice is not a guess, so
that is the only shape the gate now accepts. Ask it to repair
if Decode("x") != 5, which wants the value, and it still
refuses.
The failures were the corpus all along
Yesterday's audit caught the harness rm -rf-ing its output
directory on every run — quietly deleting the only hard evidence of what the
model actually gets wrong. Failures are now archived. Today that archive stopped
being evidence and became something better.
Re-driving the gate chain over every red artifact the archiver has kept — free, no model server, just the Go toolchain — answers the only question that matters about the gates:
=== driving the gate chain over 8 archived RED artifacts === advanced _fail-ratelimit-… advanced _fail-shortener-… GREEN-BY-GATES _fail-shortener-… GREEN-BY-GATES _fail-taskapipro-… GREEN-BY-GATES _fail-taskapipro-… GREEN-BY-GATES _fail-workapi-… GREEN-BY-GATES _fail-workapi-… advanced _proof-middleware-wall 5 green by the gates alone · 3 advanced · 0 stuck
Five of eight projects the model broke are now driven all the way to
green by the deterministic repairs alone — build, vet and
-race, with the model never invoked. A week ago that number was
zero, and I reported it as zero.
The three that only advance are the honest boundary: two are failing assertions whose specs have since been fixed, and one compiles clean and fails at runtime. That is the semantic layer, correctly out of a gate's reach. And the zero stuck matters as much: there is no archived failure the gates cannot move at all.
So the archive became a test suite. A gate change that breaks the chain passes every unit test — each one checks a single gate against a single fixture — and would otherwise be caught only by a sweep costing hours of GPU. Now the five are a lock, checked in seconds.
A green build is a shallow claim
One more measurement, and it is not a flattering one. Coverage across the suite:
| Kind | Coverage |
|---|---|
| Single-package libraries (bitset, priorityqueue, workerpool…) | 90–100% |
| tasks-api / workapi | 57% / 60% |
| taskapi / taskapipro | 24% / 30% |
The suite's biggest projects are green while running a quarter of their
own code. Two gaps account for most of it. The models package — where
the domain rules live, Task.Validate and Project.Validate,
the checks every other package trusts — had no test file at all,
in any of the three multi-package specs. And the store packages measured exactly
50.9%, which is not a coincidence: the store declares eight methods, four for Task
and four for Project, mirrored — and the tests called only the Task four. Half the
store had never been run, and the mirrored half is precisely where a copy-paste
slip hides.
Both are now specified. But the near-miss while writing them is the whole risk of
this frontier in one line: workapi's models package has Task and
Event, not Project. Asking it for a TestProjectValidate would
have written a test against a type that does not exist, broken the build, and
lowered the green rate in the name of raising coverage. So the next sweep
measures both numbers. A coverage gain paid for with a green loss is not a gain.
The pattern this week has been one thing wearing many hats: mechanisms that look like they work. A harness deleting its evidence, an error surface hiding the bug from the gate written to fix it, a best-of-N that never selected, gates corrupting the files they repaired — and now three gates that were never told their bug had happened. Not one of them was a model limitation. Every one of them was invisible until something counted. The Builder's suite stands at 290 tests, the model has not changed since April, and the bill is still $0.