The Memory Bug That OOM-Killed The Host

Chalie’s memory system OOM-killed a host today. Found it ourselves, not a user — small mercy, but let’s not pretend that’s a good day.

Chalie rolls up memories over time: individual episodes consolidate into higher-level summaries, and those summaries consolidate again one level up, so the system can recall what happened last month without re-reading every message from last month. That only works if each level actually compresses the one below it. Ours didn’t. Every level was re-fetching the raw conversation text behind the entire cluster, not just the summaries beneath it — so a second-level rollup was reconstituting a whole subtree back into raw transcript. One call sent as much as 1.4 million tokens to the model, memory spiked to roughly 30GB, and the host killed the process.

First fix was narrow: only fetch raw text at the bottom level, where it belongs, and stop there. Then looked at why the design allowed that in the first place. The code deliberately treated every level identically, no special-casing per level — a fine instinct, applied wrong. Identical code path is good design; identical behavior isn’t, because distillation is supposed to contract, and level two was re-expanding the whole subtree instead. Rewrote it so every level distills only from the gists written by the level below it, and never touches raw text again above the leaf.

That fix exposed how wrong the surrounding design already was. Consolidation ran as a periodic batch job — reduce every episode in a channel’s memory pool with UMAP, cluster what’s left with HDBSCAN, on a schedule, whether anything new happened or not. Rebuilt it to fire the moment a new memory is written: it checks its own local neighborhood, and if enough near neighbors cluster together, that cluster rolls up right there. No batch window, no reprocessing memories that haven’t changed. The old batch path is gone entirely, along with the UMAP/HDBSCAN clustering code and the scikit-learn, scipy, and numba dependencies it needed. Same job, lighter system, closer to the moment it actually matters.

Wrote a real end-to-end test for the new path before trusting it — real database, real encoder logic, with only the two genuinely non-deterministic pieces (the model call and the embedding step) swapped for fakes. Five cases, including the one that matters most under load: fire the same trigger from a dozen threads at once and confirm you get one consolidated memory out, not a dozen half-written ones racing each other.

Then, digging into how the new local-neighborhood search picks its candidates, found a second bug hiding underneath the first one the whole time. The vector search was filtering out deleted and out-of-scope rows after picking its nearest matches, not before — so on a memory store that’s actually grown, real results were getting crowded out by rows that should never have been in the running. Checked it against a live deployment: asked for the 200 nearest matches, got back only 7 of the 24 that actually qualified. Pushed the filtering inside the search itself, so asking for the nearest 20 now means the nearest 20 that count.

All of it — the rebuilt consolidation system and the thread-based conversation work running alongside it — merged into the 1.1.0 release candidate today. That branch is carrying real weight now.

Fixed a consolidation bug that re-hydrated entire raw transcripts at every rollup level, spiking memory to ~30GB and getting the host OOM-killed
Rewrote the rule so every consolidation level distills only from the level below it, never raw transcript, above the leaf
Rebuilt super-episode consolidation from a periodic batch job into one that fires on episode creation, and deleted the old batch path and its UMAP/HDBSCAN dependency stack
Added a real end-to-end feature test for the new consolidation path, including a concurrent-writers case with zero mocks of the code under test
Found and fixed a vector-search bug filtering results after the nearest-neighbor search instead of before it — a live deployment surfaced only 7 of 24 qualifying matches at k=200
Merged the rebuilt consolidation system and the thread-based conversation work into the 1.1.0 release candidate