Rebuilding Voice on Two Single-Purpose ONNX Models

Today was about pulling the voice stack apart and rebuilding it on components I actually trust. The combo package we’d been using bundled TTS and STT behind one interface, which sounded convenient right up until something inside it broke and I had no lever to pull. I split it into two single-purpose ONNX libraries — kokoro-onnx for speech synthesis, moonshine-onnx for transcription — and bundled the model weights directly into the image so a fresh install never stalls waiting on a cold download.

The bigger unlock is what that split enabled: espeak-ng, the phonemizer TTS depends on, now ships as a wheel via espeakng-loader. No system package, no distro-specific install branch, no “did apt or apk grab the right version” guesswork. Kokoro v1.0 handles synthesis with the af_heart voice; Moonshine base handles transcription, chunked into 60-second windows so a 10-minute voice note doesn’t trip the model’s internal length assertion. TTS collapsed back to a single WAV blob response — no NDJSON, no per-sentence streaming — which is a simpler contract for the frontend to reason about, even if it means the client waits for the full clip. I’d rather have a boring, predictable wire format right now than a clever one I have to keep patching.

That simplicity didn’t survive first contact untouched. Two bugs surfaced fast once real audio started flowing through the new path. First, Moonshine’s own transcribe() call wraps its input with an extra batch dimension internally — and since I was already handing it a batched array, the shapes collided and every transcription came back as a 500. The fix was to strip the leading axis before handing off, so both the short-clip and chunked paths stay in 1-D until the model re-wraps them itself. Second, the way we flatten HTML lists into plaintext was collapsing <li> items into one unbroken run of words — a three-item list was read aloud as one breathless sentence instead of three. A period injected before each closing </li> (when the item doesn’t already end in punctuation) gives espeak the beat it needs to pause. Small fix, but it’s the difference between voice output sounding like a person and sounding like a fax machine.

I also went back and cleaned up a couple of rough edges from the sentence-segmentation work that shipped the previous cycle: URLs are now spoken as their hostname — “google dot com” instead of being silently dropped — and the TTS error sentinel no longer carries a stray done: true, which was letting the frontend mistake a failure for a successful finish. Neither of these are headline changes, but they’re exactly the kind of paper cut that erodes trust in voice mode if left alone. Alongside the voice work, I trimmed the find_tools summary from 191 characters down to 78 — that string gets sent on every single turn since the tool is always available, so shaving it down is a direct, permanent cut to per-turn token cost.

None of this changes what Chalie can do yet — it changes how honestly the voice stack can tell me when something’s wrong, and how cheaply every turn runs while it does. That’s the boring infrastructure work that makes the next round of voice UX improvements possible without fighting the plumbing first.

• Voice stack rebuilt on kokoro-onnx (TTS) and moonshine-onnx (STT), replacing the combined package
• espeak-ng now ships as a wheel via espeakng-loader — no system package dependency
• Model weights bundled into the image so first request skips the network fetch
• Fixed a shape-mismatch bug causing 500s on transcription, and missing pauses between spoken list items
• URLs now read aloud as hostnames; TTS error sentinel no longer falsely signals success
• find_tools summary cut from 191 to 78 characters, lowering per-turn token cost