A Test That Cannot Fail Isn't a Test

Found a crash today with an embarrassing cause: a delegated task was sent nothing to work with. Not a bug in the task — a bug in the handoff.

When Chalie spins off a sub-task to work on its own, that sub-task needs its own prompt — its version of “here’s what you’re doing, here’s what you know.” The routing that builds prompts had a branch for the user’s own conversation and a couple more for other internal channels, but none for this one — it fell through to a default that returned an empty string. Hand a provider nothing dressed up as a real request and you get one of two outcomes: a strict one rejects the malformed call, retries three times against the same wall, and crashes the turn. A lenient one answers with something plausible-sounding instead — worse, because it doesn’t look broken. Fixed by giving the delegate a real prompt: the state it needs, the task it was actually handed, sitting there unread the whole time. Then closed the door properly — the fallthrough now raises and names the channel with no branch, so the next missing arm fails loud on the first attempt instead of burying the cause under three retries.

That’s a two-line fix. The rest of the day went to the question it left behind: how did nothing in the suite catch this? Turns out a chunk of it couldn’t have caught this bug, or most others. A test asserting a missing feature, restating a constant, or checking a third-party library’s own behavior instead of Chalie’s can only fail in one world: the one where nobody touched the code. Ran an automated pass over all 173 test files for those three shapes, then read every flagged candidate by hand — a matcher can’t tell “this absence is structural” from “this absence is the behavior,” and a sanitizer stripping a tag from rendered output looks identical to a stale test checking for an attribute nobody uses anymore. Only one is real coverage. Thirteen were the other kind: import-identity checks standing in for a done refactor, a class reading one line of source from three angles, a third-party model’s own inference quality graded as ours to fix. Kept four file-operation tests that looked like passthroughs but weren’t — each proved an operation happened on disk, not just echoed back by a mock.

One of the thirteen mattered more: the only thing checking that every API controller gets mounted with a real URL, its own docstring admitting it stood in for a guarantee the code no longer enforced. Deleting it clean would’ve left that contract resting on nobody forgetting to break it, so it moved into the code — importing the routes module now walks every controller Chalie defines, diffs it against the mounted table, and raises by name for anything missing. A test can rot into one of the thirteen kinds above. A check that runs on every load can’t.

Smaller thing rode along: three transitive dev-tooling dependencies bumped for denial-of-service advisories, none reaching anything that ships.

Two rulings on the books now: a silent empty response is a crash waiting on a retry budget — same lie as an empty result answering “success”, one layer earlier — and a test that can’t go red is code pretending to be a guarantee. Next time either shape turns up elsewhere in the fleet, it fits one of these or teaches a third.

• Fixed a delegated task getting sent an empty prompt because its channel had no branch in the prompt router, crashing the turn on every invocation

• Made the router’s fallthrough raise and name the missing channel instead of silently returning an empty string

• Audited all 173 test files for tests that structurally cannot fail, and deleted the 13 that were real hits — restated constants, import-identity checks, a third-party library’s own behavior tested as if it were Chalie’s

• Moved the one contract those tests stood in for — every controller mounted with a real URL — out of a test and into a boot-time check that raises by name

• Patched three transitive dev-tooling dependencies for denial-of-service advisories; none reached runtime code