Finished the job, then found a corrupted index hiding under it

Yesterday’s memory rework left one thing undone: a generic data-graph gateway still sitting underneath the new per-kind verticals, quietly used by whatever hadn’t been migrated yet. Today it’s gone. Deleted, not deprecated. Every kind of memory — facts, system state, places, contacts, discovery, misc, documents — now owns its own model and service over a shared row base, and nothing routes through a one-size-fits-all path anymore.

Documents were the biggest gap. The chunked text a document gets split into during ingest was still going through the generic accessor, and that path had a real leak: re-ingesting a file, or hard-deleting one, could leave its old fragments — and their search-index and vector shadows — orphaned in the graph. Old chunks stuck around, half-alive, still searchable, attached to a document that’s supposed to be gone. Giving documents their own vertical fixed the leak as a side effect of just doing the migration properly: purge now takes the fragments, the full-text rows, and the vector rows out together, in one place, instead of three call sites each hoping the other two remembered.

With every kind on its own vertical, they needed one way to be searched together. That’s the cross-kind fused recall service — one path that gathers cosine, full-text, and doc2query-variant signals across every vertical, fuses them into a single composite score, walks each hit’s supersession chain to catch the current version of a fact that’s been updated since, and hands back one ranked list. The verticals hold the raw SQL; the service holds none. Ask Chalie something and it’s pulling from facts, places, contacts, and documents through the same scoring logic instead of five different ideas of what “relevant” means.

Two bugs came out of finishing this properly, and one of them is the kind that should worry you: on a fresh index, the very first write to a row issued an FTS delete before the row had ever been inserted. Deleting a posting that never existed corrupts a contentless FTS5 index outright, and every search after that raised a “database disk image is malformed” error — which the search call was swallowing into an empty list. Silently. So a freshly indexed system wasn’t returning bad search results, it was returning zero, and nothing said why. That’s the kind of failure that only shows up as “search feels broken” weeks later, never as a crash you can point at. Gated the delete on whether the row had a prior index marker, and it can’t happen again. The second fix keeps two kinds that are never actually searched — internal decay bookkeeping and machine-written system rows — out of the FTS index entirely, so a boot-time re-scan can’t keep re-poisoning postings nobody queries.

None of this shows up as a feature. It’s the last mile of a migration that started as “give memory kinds their own home” and ended as “stop pretending a corrupted index was empty results.” The fused recall service is the part that matters going forward — it’s the single surface the next round of memory features builds against, instead of five verticals with their own half-compatible search logic.

• Retired the generic data-graph gateway completely; every memory kind now owns its own vertical model and service

• Documents got their own vertical, fixing a leak where re-ingesting or hard-deleting a file orphaned its old fragments and search-index shadows

• Added one cross-kind fused recall service — cosine, full-text, and variant signals scored together across every vertical, with supersession-aware ranking

• Fixed an index-corruption bug where a row’s first write issued an FTS delete before any insert existed, silently zeroing search results on a fresh index

• Kept never-searched kinds (decay bookkeeping, machine-written system rows) out of the FTS index so boot-time re-scans stop re-poisoning postings