Troubleshooting & recovery

Contenox keeps its state in one local database, so almost nothing is lost when a process dies. What it does not have is a daemon that notices. Recovery is something you run, and this page is the list of what to run.

Start here for anything that is not obviously a code problem:

contenox doctor

doctor is not a read-only probe. On its text output it also reclaims missions whose host process is gone, finishing them abandoned and printing how many. It says so on the line it prints. contenox doctor --json does not sweep — the JSON payload has no field to report the count in, and a mutation a diagnostic cannot mention stays silent.

doctor leads with the one question you ran it to answer:

Ready: no — No default model is set. Internal chat and chains using {{var:model}} need it.
Next:  contenox setup

The Next: line is the single command that moves you closest to yes. When everything is fine it reads Ready: yes.

Chat won’t answer: no model configured

Symptom. contenox chat, contenox new, or a chain run refuses to resolve a model. doctor reports Ready: no.

Cause. One of the defaults is unset, or the default names something the reachable backends do not serve. doctor distinguishes these:

Fix. Take the Next: command. For the guided path:

contenox setup

Or set them directly:

contenox config set default-provider ollama
contenox config set default-model qwen3:8b
contenox config get default-model

Provider unreachable, or a bad key

Symptom. Backends are registered, but doctor reports every one of them in error.

Cause. doctor separates the failure kinds rather than lumping them into “unreachable”:

Fix. Confirm what is registered and where it points, then repair the one that failed:

contenox backend list

For a local Ollama, doctor will probe your Ollama URL (OLLAMA_HOST, or http://127.0.0.1:11434) when no Ollama backend is ready and print the exact commands to pull a model and register the backend. For a hosted provider, see the provider integration pages for the key it expects.

If you just fixed a key, rerun doctor without --skip-cycle so the backend cycle actually re-runs.

doctor says a HITL policy preset is stale

Symptom. A line naming a policy file by full path, saying it predates one or more toolsets.

Cause. The policy file on disk carries no rule for a toolset the same-named preset in this build rules on — the envelope is older than the tools. This is invisible from the inside: it is not an error, it is calls to those tools quietly falling through to the file’s default_action. With a fail-closed default that means an approval card per read; with default_action: allow it means those calls run unreviewed.

Fix.

contenox init --refresh-policies

This rewrites only the hitl-policy-*.json presets, in ~/.contenox and in any workspace .contenox copy that shadows it. Chains, config, and sessions are untouched — but your own edits to those policy files are replaced. If you hand-authored a preset, copy it out first, or add the missing rules yourself: the notice stops the moment the file gains them, by refresh or by your own hand.

A policy never gates which tools the model can see — that is the chain’s tool allowlist. It gates what happens when one is called.

doctor says a trusted-binary declaration stopped matching

Symptom. A line reporting trusted-binary drift on a policy file.

Cause. A trusted_binaries block pins a command name to an absolute path and a SHA256. The binary changed — usually a legitimate upgrade. The allow is withdrawn and the call now asks a human, which is correct but gives no clue that a binary moved underneath it.

Fix. Look at the declarations first, then re-read them:

contenox hitl trust --list      # every declaration and its state on this host
contenox hitl trust --refresh   # re-read every declaration and rewrite its hash

--refresh is the upgrade path: it re-resolves and re-hashes what is already declared, and adds nothing new. See Trusted binaries.

A chain or policy file fails to load

Symptom. A run fails at load time, or an envelope name is refused.

Cause. A structural defect in the file — a dangling goto, a handler signature mismatch, an unknown field, a rule pattern that can never match.

Fix. Lint before anything runs it:

contenox vet                 # every .json in the workspace .contenox/
contenox vet --all           # plus ~/.contenox/
contenox vet ./my-chain.json # one file

Unlike doctor, vet is a pass/fail gate: it exits non-zero when any vetted file fails, so it belongs in CI. A WARN line is not a failure — it means a field parses and is accepted but is not enforced as strongly as it reads, and it names what to rely on instead.

Files are classified by content: a "tasks" array is a chain, a "rules" array (or a hitl-policy-*.json name) is an envelope; anything else is skipped. If your file is being skipped, that is why.

Recovering after a crash or a restart

There is no contenox resume command. Resume is not a verb — it is a side effect of the two commands you would run anyway.

contenox approvals respond <id> … records the verdict and, when a checkpoint exists under that ask, resumes the suspended run in the responding process. Ordering matters here and is deliberate: for a checkpointed run, the process proves it can build an engine before anything is recorded. A process with no usable model configuration is refused outright and the ask stays pending, answerable from a terminal that can reach your models — because a checkpointed run’s verdict is one-shot and must not be spent by a process that cannot act on it.

contenox approvals list is the reconciling read. Every run of it does three things:

1. Applies expired asks’ on_timeout verdicts.
2. Finds stranded checkpoints — an answered ask whose run was never claimed, or whose claim is older than the 10-minute resume staleness bound — and carries them to completion in this process. It only builds an engine when a strand actually exists.
3. Lists what is still pending.

If this process cannot build an engine, it says so and leaves the strands for the next capable one, rather than failing them.

A resume that itself fails is not lost: its checkpoint is retained with the failure recorded and becomes reclaimable again once its claim goes stale. And a resume that died after the approved tool call ran does not re-run it — the claiming resumer records the gate call’s result on the checkpoint the moment it completes, so the retry replays that result instead of executing the side effect twice.

What survives what

A mission is stuck in open and nothing is running it

Symptom. contenox mission list shows an open mission, its heartbeat is hours old, and no process is driving it.

Cause. A mission unit is a child subprocess of the host that fired it. The host is gone; the row is not.

Fix — wait, or close it yourself. The runtime reclaims it automatically, but on a deliberately generous bound: six hours of heartbeat silence, widened further when the mission has an ask parked on it whose own wait window is longer than that. Reaping live work is unrecoverable; reaping late only delays a row you were already ignoring.

The sweep is lazy, not scheduled. It runs on a host coming up, on contenox mission list, on contenox mission show, and on contenox doctor (text output). With none of those happening, the row simply waits.

A reclaim is never silent — it finishes the mission abandoned with the reason reclaimed: host process gone plus the measured silence, and files a blocker report explaining it. That reads differently from a mission you stopped yourself (stopped by operator), so the two are never confused.

If you do not want to wait out the six hours:

contenox mission stop <mission-id> --reason "host is gone"

This abandons it now, closes its pending asks, and reaps any live unit through its host. Note what a reclaim does not touch: any run the mission checkpointed is untouched, and any ask it filed is left pending on its own expiry. Those are contenox approvals list’s business, not the mission sweep’s.

See Missions for the full lifecycle.

Filing a bug: doctor --bundle

When something is wrong in a way this page does not cover, produce the attachment a maintainer will ask for:

contenox doctor --bundle
contenox doctor --bundle --bundle-out /tmp/report.zip
contenox doctor --json --bundle          # bundle notes go to stderr; stdout stays parseable JSON

It prints:

Bundle: /home/you/contenox-doctor-20260805-142230.zip
  Contents: doctor.json, build.txt, and any telemetry.log / beam.log found.
  Redacted: 3 credential-shaped value(s). Review the file before sharing it.
  Report:   https://github.com/contenox/contenox/issues/new?title=…&body=…

What is in the zip:

What is redacted. Every member is passed through the same credential scrubber on the way in, including doctor.json, whose backend URLs can carry a key in the query string. It matches named assignments (api_key=, token:, authorization=, …), URL userinfo (scheme://user:secret@host), bearer tokens, and recognizable provider key shapes — OpenAI, Anthropic, Google, GitHub, AWS access key ids, Google OAuth tokens. The field names and punctuation survive so a redacted log stays greppable; the value becomes [REDACTED].

It errs toward over-redaction on purpose: a false positive costs one diagnostic line, a false negative costs a key. It is still a heuristic. Review the file before sharing it — the printed redaction count is there so you can sanity-check it rather than trust it silently.

The issue URL is pre-filled with the environment facts a maintainer asks for first — version, platform, ready verdict, provider/model, backend counts — and names the bundle as the attachment. It carries no log content: only you decide what leaves your machine.

Next

• Missions — the mission lifecycle, states, and reclaim in full
• HITL policies — the policy format and the shipped presets
• Trusted binaries — what a pin protects and what it does not
• Quickstart — first-run setup, if doctor says nothing is configured
• contenox doctor reference · contenox approvals reference