How We're Building Chalie: An Agent Harness That Punches Way Above the Model Weights

From day one, Chalie was built on a bet: that you could drop a smaller model into a better harness and have it beat a bigger model in a worse one.

That bet is why memory came before features, and why every prompt and every tool is hand-rolled rather than pulled off a shelf. A model is a fixed quantity — you get whatever it was trained to be. The harness is the part you actually control: what the model sees, what it can reach for, what it is forbidden from doing, and what it remembers from the last time it tried.

Roughly eight months of active development later, the bet has paid. Chalie holds its own against frontier harnesses, closed and open-source alike, running on hardware you could buy secondhand.

This post is about how, and about the parts that are still broken.

Chalie was never meant to be a coding assistant

This matters for everything that follows, so let’s be clear up front.

Chalie is a companion. The goal has always been something that remembers you and helps with the ordinary machinery of a life — triaging email, holding your calendar, nudging you about the thing you said you’d do, keeping track of who’s who, running the lights, and eventually the harder domestic stuff like money and food. The current release branch carries forty-three separate ability modules, and the ones that matter most are the boring, shipped ones: mail, calendar, contacts, reminders, documents, places, home. The more personal territory — budgeting, meal planning — currently lives a layer up, as declarative playbooks that compose those general abilities rather than as bespoke features.

Version 1.2.0 will bring a coding agent. It’s built and running on the release-candidate branch, not yet tagged or released. And it arrived last, not first — it exists because we needed it, not because Chalie was ever pointed at developers.

The clearest evidence of that ordering is structural: the coding agent is built on exactly the same delegate mechanism as the ability that handles email and calendar. It’s one more instance of a pattern the harness already had for daily-life work, not a special architecture bolted on for programmers.

That distinction has a consequence. A coding assistant can specialise ruthlessly; it only ever does one kind of work. A companion cannot. The toolset has to be broad and individually tuned, which is a much harder shape to build. So we ended up building harnesses around our own development lifecycle to keep up — not skills, not off-the-shelf integrations, but full agent loops purpose-built to help develop the thing.

First, the evidence that the bet was real

Early in building the local tooling below, we rebuilt its memory system and ran a controlled test on the result. Three passes, one weak local model held constant throughout, the same three questions asked verbatim each time, pointed at the Chalie codebase. The only variable was the memory underneath.

The questions were the sort a new engineer asks on day one: how many abilities does Chalie have? Why do we have a key healer? What are rich media cards and how are they generated?

Pass 1 didn’t just score badly, it failed in a specific and recognisable way: one question inflated the context to 180,000 tokens and then produced no answer at all. Another gave up empty. That is what a capable model looks like when the harness feeds it badly — not stupid, just drowning.

Pass 3 is the interesting one. Nothing was retrained and nothing was rebuilt. A human corrected a single fact, and because facts are anchored to a specific symbol in a specific file — staleness measured as content-hash drift rather than elapsed time — the corrected fact superseded the old one instead of sitting beside it as a duplicate. One correction, permanently, across every future answer. Same model. Eighty-nine percent fewer tokens. Ten out of ten.

One table, one variable. Hold the model completely still, change only what surrounds it, and you move from 3.7 to 10 while spending a ninth of the tokens. Whatever the ceiling of that model was, it was nowhere near the thing we were actually hitting.

So we did it again, one level down

If the trick works for Chalie, it works for the agents that build Chalie. Same principle, smaller scale: don’t reach for a bigger brain, build a tighter harness around a small one.

The orchestrator is Claude Code, running a frontier model. Its job — its only job — is to reason about the problem and dispatch. Everything below it runs locally.

Five specialisations, one model. The orchestrator never picks the tools and never writes the code. It decides what needs doing and hands down an instruction small enough that a 35B model cannot misread it:

“Find every caller of this function.” “Rewire it to take the config object instead.” “Assert that the settings modal opens when the gear button is clicked.”

Thin instruction, thick tool

This is the part people usually get backwards. The instinct when a small model underperforms is to write it a longer prompt. The opposite works better: shorten the instruction until it is unambiguous, and push all the precision into the tool.

None of the middle column is the model guessing. Finding callers is a language server answering a query — the same machinery an IDE uses, driven over the same protocol, with real language servers for Python, TypeScript, PHP, HTML and CSS. Stepping through a failure is an actual debugger session with breakpoints, stepping, and live frame inspection — state that persists between tool calls, so the agent can look around a stack frame rather than reason about what it might contain. Profiling is real profiling: sampled hotspots, memory snapshots, execution traces with exact call counts and per-line hit counts.

A small model asked to guess which functions are hot will produce a confident, plausible, wrong answer. A small model handed a profile will read it correctly. Almost all of the perceived intelligence gap is really a grounding gap.

What each mode is forbidden from doing

The other half of the trick is subtraction. Every mode is defined as much by what it cannot reach as by what it can — the mode is chosen at launch, and the tools listed for it are the complete toolset. There is no tool discovery. Nothing else is ever callable.

Every one of those prohibitions was earned. The performance mode has no edit tools because an agent that can measure and fix will start fixing before it has finished measuring — so the rule is measure first, never guess, never optimise here. The test mode cannot write files because an agent that can edit the code it is testing will eventually edit the test instead of the bug. If a fix is needed it says so and hands the finding back up; the orchestrator dispatches a separate code run.

Constraints like these are cheap to state and enormously effective on a small model. You are not asking it to have good judgement. You are removing the option to have bad judgement.

The prompts are the product

The whole system prompt for the coding agent is about forty lines, and almost none of it describes capabilities. It’s nearly all behavioural correction, written one clause at a time after watching something go wrong:

Ground claims in tool results; if you have not looked, look before answering.

When a tool call errors, fix the specific problem named in the error before retrying; never resend identical arguments. If the same call fails twice, change approach or tell the user.

Tool outputs from earlier turns are pruned from your context; restate load-bearing paths, values, and excerpts in your replies so they survive.

That last one exists because context compaction silently eats the evidence an agent is relying on, and the agent doesn’t notice. So the harness now tells it to write down anything load-bearing where it will survive.

The failures that produce these lines are not subtle. One measured turn arrived carrying 2,555 tool calls, of which 2,500 were byte-identical — a model stuck in a loop, cheerfully re-sending the same failing edit two and a half thousand times. The fix wasn’t a better prompt; it was a hard cap in the harness that blocks a repeating call, plus a corrective nudge injected as a user message specifically so it survives compaction, plus a synthesised “this round was abandoned” result so the model can see the wall it hit.

You cannot prompt your way out of that. You have to build the wall.

Problem one: how do the agents stay in sync?

They don’t. That’s the design.

Only the orchestrator gets the task board and the real memory system. Every local agent gets a small, private, per-project memory of its own — a single file next to the project it is working on. It is deliberately narrower than Chalie’s: no sharing between projects, no board, no human review workflow, no curation.

That per-project memory is still the most interesting piece of the local harness. Facts are anchored to a path, a symbol, and the content hash of that symbol at the moment the fact was learned, so a fact goes stale when the code it describes changes rather than when a timer expires. Decisions live in a small typed graph where one entry can explicitly supersede another. At the start of a task the agent gets a cheap, derived map of the project plus a handful of anchored facts — a fixed budget, no model call required to assemble it.

But it is a notebook, not an institution. The orchestrator is the commander; the agents follow the instruction to the letter and remember only what helps them do the next one. That asymmetry is doing real work: five agents sharing one memory would spend most of their tokens negotiating with each other.

Problem two: who maintains the harnesses?

This is the honest part.

If the tools are the thing that makes a small model work, then a broken tool silently degrades everything downstream — and the only entity that ever sees the breakage is the agent holding the broken tool.

So the agents were given a way to complain. One tool call, written from inside the failure, appending to a log that belongs to the harness rather than to whichever project happened to be open at the time — because a harness defect is a property of the harness, not of the codebase it was pointed at. The prompt is blunt about the distinction: this is for your tools misbehaving, not for the task being hard. An edit reports success but the file is unchanged. A result comes back empty when it shouldn’t. A message names a tool you don’t have.

Then a repair agent — a frontier model, working on the harness itself — picks the report up and fixes it. And here’s the part that makes it work:

The repair agent doesn’t get to declare victory. The agent that filed the complaint does.

Concretely: the QA agent reports that its browser automation has stopped working. The repair agent picks up the issue, makes a fix, and then goes back to the QA agent with something like “try that again — based on the failure you hit, it should be working now; confirm.” The QA agent runs it for real, in its own harness, with its own tools, and says yes or no. If it says no, round two. The loop closes when the agent that was actually blocked is satisfied — not when the model that wrote the patch is confident.

It’s a small inversion and it removes a whole failure class. A model marking its own homework will pass itself, every time.

The honest bit

This is new, and the first version of it did not work at all.

The reporting channel sat in the harness for weeks and fired exactly zero times across roughly four thousand tool calls in twenty-five sessions. The log file it was supposed to write never came into existence. The agents were hitting broken tools — we know they were — and instead of reporting them, they described the problem in their final answer and moved on. Describing a failure feels like reporting it. It isn’t.

Worse, we had an automated check that was supposed to catch exactly this, and it was reporting a pass. It parsed the report log to verify the contract held, using a library that was never declared as a dependency — so on a machine without that library the check quietly skipped every single one of its own assertions and reported green. A test that cannot fail isn’t a test. It’s decoration that costs you the thing it was protecting.

Both halves are now fixed. The missing dependency is declared and a skipped assertion is a hard failure rather than a silent pass. And the trigger moved: instead of a line in the system prompt hoping the model remembers to call the tool, the harness now appends the instruction directly onto the specific error results where a defect is most likely — at the exact moment of failure, in the tool output the model is already reading.

It fires now. We proved it by breaking a tool on purpose and watching the report land. But it has still never fired organically in real use, and nothing reads the log automatically yet — a human still opens it. Calling this a closed self-healing loop would be a lie. It’s a channel that works, wired to a process we still run by hand, and we’re actively working on the rest.

We’re leaving that in because the alternative — describing the version that exists in our heads — is exactly the failure mode we just spent a paragraph fixing.

All of it, on one graphics card

Every local agent above runs against a single RTX 3090 serving a 35B mixture-of-experts quantised to four bits, with a 131,072-token context window and prefix caching on, handling several agents concurrently.

The single-card part was measured, not assumed. There is a second, smaller card in the machine, and splitting the model across both makes it slower — the single card generates roughly 45% more tokens per second per stream, because the second card sits behind a much narrower link than the first, and every token has to cross it. One card, fully loaded, beats two cards poorly connected. The serving engine was chosen the same way: under concurrent load some engines get slower in aggregate as you add streams, which is precisely backwards for a fleet of agents working in parallel.

That is the whole inference budget. One consumer card, bought secondhand, doing the work of five specialists.

What actually transfers

If you take one thing from this, take the ordering.

Spend on memory before you spend on model. The three-pass table above is one model, unchanged, going from mostly wrong to fully right while its token bill dropped by nearly nine-tenths — because of what it was handed, not what it was. Then give it real tools rather than asking it to simulate them — a language server, a debugger, a profiler, a browser — because grounding beats reasoning at this size, every time. Then take tools away, aggressively, so each mode has exactly one job and no way to invent a second. Then make the instructions small enough to be unambiguous, and put the precision in the tool.

And when the harness breaks, let the thing that got hurt be the thing that decides it’s fixed.

The model is the part you can’t control. Everything else is engineering — and the engineering is where the wins are.