Rebuilding Memory Compaction From First Principles

Today was about a system I’d been unhappy with for a while: what happens when a conversation with Chalie outgrows the model’s context window. The old compaction pipeline had grown two separate stages, a dedicated table, and a pile of on-the-fly threshold math scattered across the hot path. It worked, mostly, but it was fragile in exactly the way memory systems shouldn’t be — and I’d traced a recursive poisoning bug back to a stale health-check signal that kept re-feeding itself into every prompt. That’s not a bug you patch. That’s a bug you redesign around.

So I ripped it out and rebuilt it on one idea: compaction should be append-only. There’s no more dedicated compactions table to fall out of sync with the conversation — every compaction is now just a tool_calls row, ordered by id, filtered to successful runs. A bad summary from the model gets written as a failure row for audit purposes and is simply ignored by the lookup; the previous good summary stays authoritative until a new one succeeds. That single property — failure can never poison state — is what the old design didn’t have, and it’s the whole reason this rewrite exists.

The second half of the redesign is the prompt itself. Instead of a generic “summarize this” instruction, Chalie now summarizes its own conversation in second person, told explicitly what survives a compaction and what doesn’t, with a required scratch-work pass before the summary it actually keeps. That distinction matters more than it sounds: previously the model’s reasoning could leak into the stored summary as noise; now only the tagged summary block gets persisted, so the memory that carries forward across a long conversation stays clean and short — 200 to 400 tokens, not a transcript dump. I also collapsed the two-stage overflow handling (a tool-level pass and a full-restart pass) into one path, and made sure the turn that triggers compaction is excluded from what gets summarized — the model was previously seeing its own live question folded into “past conversation,” which was actively confusing it.

Threading through the same day: a persistent bug where compaction simply never fired in testing, traced to a genuinely subtle root cause — provider configs never carried their own name field forward, so the lookup that reads the persisted compaction threshold silently resolved to nothing and skipped the check on every turn. Fixing that unblocked a broader move to store the context limit and the compaction threshold directly on each provider at boot time, instead of recomputing them inline during every conversation turn. One less calculation in the part of the system that has to be fast every single time.

The other structural change worth noting: subconscious background thinking — the process that consolidates memory, decays stale signals, and drifts toward relevant context between conversations — now runs as a single five-step tick instead of a separate queued worker. Fewer moving parts, one thread to reason about, same behavior. And native tool calling got locked down harder: providers read tool invocations exclusively from the structured field the API gives them, with the inline-XML fallback path removed entirely. If a model doesn’t populate that field correctly, that surfaces immediately as a real failure instead of being silently rescued — which is the only way you find out a model actually has a tool-calling problem worth fixing.

None of this is visible in the product today. What it buys is a memory system that degrades honestly instead of quietly: if compaction fails, you keep the last good summary, not a corrupted one. That’s the property long conversations need, and it’s the foundation the next round of memory work builds on.

• Compaction rewritten append-only on the tool-call log — failed summaries are audited, never trusted, and can’t poison future turns

• New continuity-first summary prompt separates private reasoning from the persisted memory, capped at 200-400 tokens

• Fixed a silent bug where a missing provider name field meant compaction thresholds were never being read

• Compaction thresholds now persisted per-provider at boot instead of recomputed on every turn

• Subconscious background processing folded into a single five-step tick; native tool calling hardened with no fallback path