A Dead Tool Kept Steering the Model Anyway

I said the document tool was retired two days ago. Today it proved I hadn’t actually finished the job. A memory recall hint still told the model, on any lookup miss, to fall back to document — a tool that no longer existed. Miss a recall, get told to try a dead end, reword the query, hit the same dead end again. No error, no crash, just a loop burning turns on live-data questions until the runaway guard stepped in. Worse than a crash — at least a crash tells you something’s wrong.

Fixed the hint first: it now states the fact plainly (a miss means the fact isn’t stored), forbids reworded re-queries, and routes live or time-sensitive lookups to find_tools instead. Then went looking for how many other places still pointed at a tool that was gone. Uncomfortable answer: every model-facing description presenting documents as a live subsystem, two dead contract modules with zero importers, stale constants, six frontend permission labels for actions that could never fire. Swept all of it, then found the same shape one layer down — 232 steps across 66 of 73 curated skill files, still telling the model to draft into document instead of just writing a file. Rebuilt the skill index after fixing every one. A migration isn’t done when the feature works. It’s done when nothing still points at the thing you deleted.

The rest of the day was the same instinct twice over: measure the real number instead of guessing, and make the system correct itself instead of degrading quietly. Provider clients now read the actual timing split out of every LLM call — prefill versus decode, plus cache-hit token counts where a provider reports them — instead of treating a response as a black box that either worked or didn’t. The context-window story from a couple days back went from “measure it once” to “one column is the only truth, and it fixes itself on the next call if it’s ever wrong.” Found the reason that column kept staying empty in production, too: a cache layer’s dict literal quietly dropped the row’s id, so every write-back silently failed and every request re-probed the provider from scratch. Small bug, expensive to trace, obvious once you see it.

find_tools got simplified in a way I like: it used to run a keyword/BM25/vector search cascade to guess which tool you meant. Now it’s one exact-match lookup against a registry of canonical names and declared aliases — either the name resolves or it lands in not_found, no near-misses. MCP tools got pulled out entirely into their own dedicated ability, because “search across every tool” and “list what this server exposes” were never the same operation. Voice picked up its own boot-warmup too: speech-to-text and text-to-speech models both load in the background at startup, so the first voice interaction of a session isn’t paying for a cold model load.

Threads and voice both moved onto the Endpoint/Action contract today, leaving almost nothing of the old hand-rolled routing left to migrate. Caught real production bugs along the way: a policies endpoint silently 422’ing on a mistyped response column, a list-management tool rejecting valid input because it only understood one of two shapes its own contract declared, and eighteen exception handlers logging a bare string instead of the traceback that would explain the failure.

Next up: watching whether the recall-hint fix kills the loop pattern in practice, and finishing off what’s left of the old routes.

• Fixed a production loop bug where a dead tool reference in a memory-recall hint sent the model into repeated dead-end retries, then swept 232 remaining references across 66 skill files plus the code that caused it

• Wired real per-call timing (prefill/decode split) and cache-token counts into telemetry, and made the stored context window self-heal instead of staying silently null

• Rewrote find_tools from a fuzzy search cascade to a single exact-match lookup, and split MCP discovery into its own dedicated ability

• Moved voice model loading to boot time so the first voice interaction of a session isn’t a cold start

• Migrated threads and voice onto the Endpoint/Action contract, closing out nearly all the legacy hand-rolled routing