Fixing the voice pipeline, four bugs deep

I set out today to make Chalie’s voice output actually sound like something you’d want to listen to. Text-to-speech is one of those features that looks done in a demo and falls apart under real conversation — long replies, markdown-formatted lists, five-chunk answers stitched together. Today was about closing that gap between “works in the demo” and “works every time.”

The first bug was subtle and embarrassing: HTML lists like <ul><li>A</li><li>B</li></ul> were having their tags stripped for speech synthesis, but the words on either side of each tag ran together into gibberish — optionsFirst, detailSecond. The phonemizer silently dropped these malformed tokens, so whole list items just vanished from what you heard. The fix was mechanical once diagnosed: insert a space at every block-tag boundary before stripping. Four regression tests pin the cases — minified lists, adjacent paragraphs, headers, line breaks.

Second bug, worse in practice: the speech model leaves half a second of trailing silence at the end of every synthesis call. Stitch together an eight-sentence reply in five-or-so chunks and you get audible dead air every three to four seconds — I captured the actual waveform on a live reply and found eight silence gaps sitting exactly at the chunk boundaries. Trimming leading/trailing silence per chunk before concatenation, while keeping a few dozen milliseconds so consonants don’t clip, turns that into a continuous, natural-sounding read.

Third: a concurrency bug that could kill synthesis outright. The phonemizer backend caches mutable state across calls, and multiple chunks phonemized in parallel threads would clobber each other’s internal counters, throwing a hard “number of lines in input and output must be equal” error that took the whole reply down with it. The fix splits phonemizing (now serialized behind a lock, since it’s fast) from audio inference (kept parallel, since it’s slow) — same throughput, no more race. Each chunk also now gets one retry if it errors or comes back near-silent, falling back to a hair of silence rather than aborting its neighbours — one bad chunk degrades gracefully instead of taking the whole reply with it.

Underneath all of this was a bigger rebuild from earlier the same day: a pool of independent synthesis instances running in parallel instead of one shared instance queued behind a lock, an explicit error instead of silent truncation when a chunk runs over the phoneme budget, and markdown stripped unconditionally so *bold* doesn’t get read aloud as “asterisk bold asterisk.” Long replies that used to take minutes now synthesize in seconds, and errors surface instead of disappearing.

Alongside the voice work, I ran a security and code-quality sweep: hardened three regexes with backtracking-based denial-of-service exposure (voice cleanup, message processing, link-preview parsing), added a path-traversal guard on data file endpoints, fixed a couple of real return-type bugs a static analyzer flagged, and cleared a batch of lint smells. Separately, a memory bug got closed where every recall — even the silent ones run automatically to seed context — was appending a document-search result the model never asked for; that’s now gated to fire only on recall the model actually requested. And a routing tweak moved tool discovery to the very front of what the model always sees, since “I don’t have a tool for that” was showing up when the model just hadn’t thought to look.

None of this is glamorous work, but it’s the kind that decides whether voice mode feels trustworthy or flaky the tenth time you use it. Next up is making sure that reliability holds as reply length and formatting complexity keep growing.

• Fixed four independent voice-pipeline bugs: dropped list items from tag-stripping, accumulating dead-air gaps between chunks, a parallel-synthesis race condition, and hard failures on bad chunks — each with dedicated regression tests

• Rebuilt TTS around a pool of parallel synthesis instances with explicit overflow errors (no silent truncation) and unconditional markdown stripping

• Hardened three regexes against backtracking-based denial-of-service and added a path-traversal guard on data endpoints

• Fixed a memory bug where silent, auto-triggered recalls were leaking an unrequested document-search result into the model’s context on every turn

• Moved tool discovery to the front of the model’s always-available toolset so named capabilities stop getting incorrectly refused