modeld: the local inference daemon

modeld was Contenox’s dedicated local inference daemon: a separate process that ran on your machine (or a remote GPU box) and owned hardware resources on behalf of every frontend — the CLI, the VS Code extension, Beam, and any ACP editor — talking to the pure-Go runtime over gRPC. It served GGUF models through llama.cpp and OpenVINO GenAI models through a shared session contract, so the runtime itself never linked native inference code. A single modeld binary registered as a remote backend served a laptop and a dedicated GPU box through the exact same protocol, and later builds added native vision (VLM) support through llama.cpp’s multimodal stack, with capability truth reported per model rather than inferred from its name.

The seam was deliberately owned by the consumer: the runtime declared the session interface and modeld implemented it, so the runtime never imported the daemon. It was not a chat API and not a bare token stream. A session was opened against a model and then driven with EnsurePrefix — keep the stable prompt prefix’s KV hot — PrefillSuffix for only the part that changed, and Decode, plus Snapshot/Restore, Describe, Embed, and an ExplainContext report. Every call answered with real numbers: tokens reused, dropped, prefilled and resident, remaining capacity, and a manifest digest. A raw token-level boundary was built first, as an in-memory implementation stress-checked against both real backends, and then rejected — both engines sit at a higher altitude than raw tokens, and OpenVINO GenAI in particular holds the tokenizer and chat template internally and caches a string prefix, which a token-only daemon could never have honoured. Reuse was therefore keyed on a context manifest, never on byte equality: profile id, backend version, model digest, prompt-template digest, a runtime digest covering context size, batch, GPU layers, tensor split, flash attention and KV type, BOS policy, and per-segment byte and token ranges with their hashes. Two gates had to pass — runtime compatibility first, then a token-level longest-common-prefix match — and a suffix was only accepted if tokenizing the whole prompt cold produced exactly the warm path’s stable tokens plus the suffix tokens. A tokenizer merge across the split boundary was rejected as a manifest mismatch rather than papered over.

Its most distinctive piece was the capacity planner. On opening a session it read the model’s own metadata — layer count, KV head count, head dimension, KV dtype, architecture, and the declared sliding-window size cross-checked against what the loaded model actually reported — took a live free-memory snapshot of the target device, subtracted weights, runtime overhead, a policy headroom and a floor reserved for every other process on the card, then inverted the remaining budget through GQA-correct KV arithmetic sized from KV heads rather than attention heads. What came out was not one number but six, deliberately never collapsed: the model’s own ceiling, what was requested, what raw memory allowed, the dense window actually served, the physically hot KV window, and the planner’s logical window once a host-RAM cold budget was counted — each carrying a machine-readable reason when something was the binding constraint. Quantized KV was costed at its real byte width. GPU layer offload was resolved against the requested context rather than maximised blindly, because a fully offloaded model with an unusable context window is a worse answer than a partially offloaded one that works. Beyond the hot window, residency was shared policy: one pure-Go planner decided what to keep and what to park, and a small per-backend executor carried it out — llama.cpp by native KV surgery, removing a range and sliding the tail’s positions down, and OpenVINO, whose KV is not range-addressable, by exporting the block to a cold store and re-prefilling the survivors. Same interface, two genuinely different mechanisms, with each backend reporting its real capabilities — tail removal, middle removal, position shift, sparse attention and its window, cold store, recompute — rather than claiming a parity it did not have.

A cooperative, cross-process TTL lease file gave one daemon per data root a single owner: race-free cold acquisition through exclusive create, expired takeover through atomic overwrite and read-back, and the owner’s instance UUID as a fencing token checked on every cross-process call — fencing, explicitly, not authentication. The lease record also carried the daemon’s endpoint, so discovery was one atomic file read, and probing it alongside the binary’s location yielded four distinct states — not installed, not running, stale, running — each with its own typed error and its own remedy. Inside the daemon a monotonic slot generation was bumped on every load, unload, model switch, owner takeover and fatal backend reset, so a handle minted before any of those failed cleanly instead of addressing the wrong model; one operation at a time held the slot, with a whole turn counting as a single protected operation; and an idle reaper returned memory to the system after inactivity while keeping implicit sessions resident, so a warm model outlived the CLI process that created it. Errors were typed rather than generic — busy, not active, switch required, insufficient memory, stale generation, backend mismatch — and anything that made resident-token bookkeeping untrustworthy closed the session as fatal and evicted the slot instead of continuing on numbers it could no longer prove.

What it proved

A live-hardware capacity planner beats static config. Computing effective context from an actual memory snapshot plus real KV math — GQA-correct, sliding-window aware, quantization aware — solved the “why did this just OOM” problem instead of hand-waving it.
Six context numbers are more honest than one. Model ceiling, request, memory fit, served window, hot KV and planner window mean different things; collapsing them is how one session ends up reporting three mutually contradictory answers to the same question, which is exactly what happened until every capacity answer was made to come from the resident session’s own open-time resolution.
Manifest identity, not byte equality, is what makes prefix reuse safe — and it paid. A ~9,000-token stable prefix cost 1.77 s to prefill cold and 0.03 s warm on a 6 GB laptop GPU, reusing all 9,350 tokens and prefilling none, turning a 1.97 s turn into a 0.18 s one. On CPU the same shape ran 7.61 s cold against 73 ms warm, and editing the stable prefix changed its hash and correctly sent the next turn straight back to a full prefill. The stable-prefix hash was a reliable predictor of a cache hit.
Lease-based ownership lets many frontends share one local model safely. CLI, editor, and browser sessions could all point at the same resident model without a central server process owning everything — one backend session was verified serving four CLI invocations spanning two conversations.
A stateful session boundary, not a stateless provider interface, is the right shape for local inference. Warm KV, prefix reuse, and durable snapshots only make sense once the daemon owns session lifetime — a snapshot round-trip through a fresh session reproduced a greedy continuation exactly, but only at f16 KV precision; the default 8-bit cache was measurably lossy.
Streaming eviction cannot be planned from a manifest. Freshly streamed tokens have no manifest blocks yet, so a plan-based evictor sees nothing to free; the working design pinned the attention sink at the head and the recent tail, parked the middle to cold storage in fixed 256-token blocks, and ran independently of the manifest — which is what let a conversation carry roughly 7,500 tokens of history through a ~3,000-token hot window with no overflow at all.
Capability truth has to be reported, not inferred. Metadata parsing, loader support, pipeline support, device support, memory fit and certified quality are six independent questions, and only their intersection is a usable model — a GGUF can declare an architecture the linked runtime cannot load while its numeric metadata still parses perfectly, so the capacity output looks valid right up until nothing serves it. Device enumeration is likewise not device support.
A benchmark row is only worth what its label says. Every measurement was tagged by the path that produced it — the shipped product path, a raw backend control, or a microbenchmark — and a run that produced zero tokens because of a launcher failure, a missing library, or a context rejection was recorded as a failed run rather than as throughput.
Refuse-don’t-spill beats silent degradation. An explicit, typed context-overflow refusal proved more useful to an agent than a quietly truncated or CPU-spilled response — and when the refusal itself quoted a stale context number, the fix was to make the error carry the live one.

Where this lives now

Local inference now rides Ollama and vLLM directly; the capacity-aware, session-first thinking modeld proved out still shapes how the runtime treats any local backend.