Documentation for v0.36.0

The full reference. New here? Start with Getting started.

Model router

Point any OpenAI Chat Completions client at http://127.0.0.1:4141/v1 with "model": "auto", and delegation-router sends each request to the cheapest profile in your pool that is strong enough for it. A decider you choose rates the request; the rating sets a minimum score; the cheapest pool profile that reaches it answers, and the response says which model answered and why.

• It runs on your machine and listens on loopback only.
• You name the decider, the pool and the default; it chooses only inside the pool and discloses the chosen model in every response.
• It never retries, falls back or widens the pool. It keeps a decision only for a conversation the client names, in memory (Conversations).
• No public instance, access tokens or multi-tenant limits.
• serve prints one line at start, naming any profile set aside as unreachable, then nothing: no request or error log, no header, no key. Opt-in receipts are the only record.

The decider sees an excerpt of each request (all of it with input: full) unless it runs locally. Only the chosen profile gets the request, with its own key, never a key or header from the caller.

Endpoints

delegation-router check
delegation-router serve
# another shell:
curl -si 127.0.0.1:4141/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"Rename foo to bar in utils.py"}]}'
curl -s 127.0.0.1:4141/v1/models        # the pool, plus "auto"
curl -s 127.0.0.1:4141/v1/health

Use base_url="http://127.0.0.1:4141/v1" and any API key; the router ignores it, because each profile uses its own. Claude Code (Anthropic Messages) and Codex 0.163 (custom providers only on the Responses API) cannot send requests here, but both can be pool profiles.

Naming a pool profile as model skips the decider and still discloses; the context and cost rules still apply. An agent loop that sends X-Router-Conversation keeps its first route (Conversations). doctor.sh’s “Model router” section shows the decider, the catalog in effect and check’s warnings.

Commands

Only serve dispatches a pool profile; starting it and sending auto is you directing the router. Running a decide recommendation is a separate delegation-run you ask for.

decide --lane ROLE recommends a profile for a task you are about to delegate. Candidates are the pool profiles that declare the role and whose adapter supports it (the check delegation-run makes); the rest go under blocked. The recommendation carries the adapter’s capabilities (text-patch, sandboxed-worktree, …). A decision that lands on a default without the role, or an unreachable one, recommends nothing and says so. A role no reachable pool profile takes is an error.

Configuration

A router block in the personal configuration turns the router on; without it the router is off and check says so.

{
  "schema_version": 2,
  "review_policy": "optional",
  "profiles": {
    "flash": {"adapter": "deepseek-api", "model": "deepseek-flash", "roles": ["clerk"]},
    "qwen":  {"adapter": "token-plan-openai", "model": "qwen3.8-max", "roles": ["builder"]},
    "sol":   {"adapter": "openai-compatible", "model": "gpt-6.1-sol", "roles": ["judgement"],
              "base_url": "https://api.openai.com/v1", "credential_env": "OPENAI_API_KEY"}
  },
  "router": {
    "decider": {"kind": "llm", "profile": "flash"},
    "pool": ["flash", "qwen", "sol"],
    "default": "qwen"
  }
}

Profiles may declare routing fields; a declaration wins, the catalog fills the rest, and check shows each tier’s tier_source.

Where the tiers come from

Artificial Analysis publishes an Intelligence Index (0–100, a composite of public evaluations), coding and math sub-indexes, and list prices for a few hundred models and reasoning settings. One published rule, index-bands-v1, makes tiers of it: frontier within 8 points of the day’s best, mid within 20, small below; the catalog records the bands and the date. delegation-router catalog show prints the rule, the best, and every model’s tier and source; each row carries its evidence (measured name, index per reasoning setting, date), or null when placed by hand.

Neither catalog refreshes itself:

• Shipped: config/catalog.json, refreshed before each release.
• Local: catalog refresh writes $DELEGATION_DATA_HOME/config/catalog.local.json, which wins while at least as recent. It needs AA_API_KEY (Keys and environment); a free account issues one. Uninstall keeps key and catalog; reinstall restores them.

refresh matches by the board’s slug (claude-opus-5-5, -xhigh, …), else by measured name through the row’s aa_name, never partially, and names what it could not place. A profile whose model differs from a catalog id only in case or punctuation still matches, by measured name. The data is Artificial Analysis’s.

The index measures general capability, not your tasks; eval on your golden set is the local proof.

How a request is routed

1. Features: estimated tokens, turns, tools, images, max_tokens, JSON output requested, arithmetic-looking numbers.
2. Set aside, without a model call: profiles that cannot take the request go under blocked (table below).
3. Decide: the decider rates the request, unless skipped.
4. Policy floor-v1 picks one profile.
5. Forward one request to that profile’s endpoint with its key. The router body field is dropped, model rewritten, stream_options.include_usage set when streaming. Connecting is bounded at 20 s; the profile’s parameters.timeout bounds the exchange (504 timeout). Without it a non-stream answer is awaited indefinitely and a stream is cut after 30 s without an event.
6. Disclose.

The default is never replaced: an auto request that lands on a blocked or unreachable default is refused 400 no-usable-candidate, with a reason.

The decider is skipped when it cannot change the outcome: nothing is eligible; only the default is; or the decider has a price, no eligible profile scores above the default, and the cheapest saves less than 10× the decision’s cost (450 input tokens at the decider’s price).

The excerpt (at most 4,000 characters): turn count, whether there is a system prompt, tool names, the first 1,000 characters of the system prompt, and the last user turn trimmed to 2,048; never images. Before the trim, the blocks coding harnesses add to a user turn are dropped: <system-reminder> (OpenCode; Claude Code through translators such as claude-code-router), <environment_context> (Codex) and <environment_details> (Roo Code). They describe the session, not the task, and their dates and versions do not count as arithmetic. A user turn that is only such blocks is passed over for the one before it, and kept when no turn says more. input: full sends every message’s text as it is, up to 32,000 characters.

The question: how much capability this needs (0 trivial … 4 frontier, in half points), and does it involve code, math or rare knowledge. long (over 20,000 estimated tokens) and tools are set deterministically. An out-of-schema answer, HTTP error, timeout or missing key is undecidable: the default answers, with a warning. Out of schema includes any probability or confidence that is not a finite number in [0, 1], a difficulty outside its scale (a classifier’s outside 0..1) and a distribution that does not sum to 1 (within 0.005 a level); one such value voids the whole answer, and no value is clamped into range. Text in the request cannot steer the route: the decider never treats it as instructions and cannot name a model.

Policy floor-v1

No usable signal, or confidence below 0.55: the default. Otherwise the rating becomes a minimum score on the Artificial Analysis index:

• A yes/no “needs a strong model” sets the frontier band from 0.7, the mid band from 0.3; the higher minimum wins.
• Tags count from 0.6. math, rare and tools raise the minimum to the mid band, long to the frontier band. code narrows the candidates to code-capable profiles (any, if none of those reaches the minimum).
• Every eligible profile at or above the minimum is a candidate, whatever its tier; the cheapest by price.in × est_tokens + price.out × max_tokens (1024 when unset) wins. If none reaches it, the strongest eligible answers and the reasons say so.
• A profile’s score is the current-v5 index at its exact measured effort. An unmeasured named effort has no score. A hand-placed model or a declared tier counts as the bottom of its band.
• Effort, on deepseek-api and token-plan-openai only: high from a rating of 3.6 (a floor; a declared xhigh or max is kept), low below 1.5 on a small-tier profile.

Why this model and not that one

The rule is Levanto Sage’s: the cheapest model that will get it right. The decider judges the task; your configuration judges the models.

For a configured pool of Luna/max, Terra/high and Opus/max, using the 2026-10-28 v5 snapshot:

These are published benchmark scores and per-token cost estimates for the example, not task-success guarantees or local CLI invoices. Model-level unnamed measurements cannot establish a floor for a named profile effort.

Your levers: tier and price per profile, the pool, the default, the ceiling. The reasons give the minimum, the chosen score and each candidate’s estimated cost.

Native profiles in the pool

A native profile has no chat-completions endpoint, so the router goes through delegation-run. The conversation becomes a prompt file (a lone user turn verbatim, otherwise a [system]/[user]/[assistant] transcript), delegation-run --profile <id> --lane <role> runs it in an empty temporary workdir, and the answer returns as one chat completion.

• Role: the first read-only role the profile declares and its adapter supports; a build role only if nothing else. check and router.delegation_run.lane name it.
• Limits stay: the profile’s timeout, the identity check (502 provider-identity-mismatch), OAuth locks (a second concurrent Kimi, Muse Code or Grok request gets 429 provider-rate-limited), sandboxes.
• Receipt: review_policy, review_status, status under router.delegation_run.
• Streaming: one event with the whole answer, then the router chunk.
• Cost: null when the CLI reports no usage.
• Nothing outlives the request: the dispatch runs in its own process group. When the client hangs up or serve stops, the whole group — dispatcher, runner and the CLI it started — gets SIGTERM, and SIGKILL 10 s later.

Claude models join through claude-code (claude-zai runs GLM through the Claude Code CLI).

Conversations

Without help, the router decides every request on its own. An agent loop sends one request per step, so each step pays a decision and can land on another model: a step with tools needs the mid band, and once the conversation passes 20,000 estimated tokens long asks for the frontier. The model changes halfway through the task, and the provider’s prompt cache is lost with it.

A client that sends X-Router-Conversation: <id> (1–128 letters, digits, ., _, :, -) on its auto requests keeps the first route:

• The first request is decided as usual and pins the conversation to the profile it dispatched (in shadow mode, the default) at that request’s effort. X-Router-Pin: new.
• Every later request goes to that profile without asking the decider, as if the client had named it: mode pinned, X-Router-Pin: held, and router.pin with the profile, since and the request count. The context, cost and JSON-mode rules still apply; a request the pinned profile cannot take is refused 400 no-usable-candidate (or cost-ceiling) with pinned naming it, never sent elsewhere. Start a new id to decide again.
• Memory only: 1,024 conversations, the least recently used forgotten first, and a pin lapses after 12 hours without a request. A restart forgets them all. The id is never written to receipts or echoed back.

The pin is set when the first request is routed, whatever the provider answers, so a client’s retry reaches the same profile. Naming a pool profile as model ignores the header. Without the header nothing changes.

Deciders

Every decider reports the same signals: a 0–4 difficulty (expectation, distribution, confidence), a “needs a strong model” probability, and tag probabilities. A service decider uses its own key (credential_env), never a pool profile’s; llm uses its profile’s (Keys and environment). Each decision is priced from price and disclosed.

Fields: kind; profile (llm); for services, url (HTTPS or loopback HTTP), credential_env, model (Jev), price ({in, out} USD per Mtok); input (excerpt or full); timeout (1–600 s). Sage and Jev URLs may point at your own proxy; Jev through OpenRouter is not wired. Measured answers, latencies and costs: Decider measurements.

Pin the decider you measured. jev-latest follows TypeSafe’s newest Jev, so an eval gate passed last month may describe a model that no longer answers. Set model to a fixed Jev version from TypeSafe’s list and run eval again before you change it. Every decision names the version that answered (X-Router-Decider, router.decider.version), and so does the eval report.

What a response carries

Headers: X-Router-Profile, X-Router-Model, X-Router-Tier, X-Router-Effort, X-Router-Decider (id/version), X-Router-Policy, X-Router-Difficulty, X-Router-Confidence, X-Router-Undecidable, X-Router-Latency-Ms (the decision’s), X-Router-Mode (route, shadow, explicit, pinned), X-Router-Pin (new or held, with X-Router-Conversation), and X-Router-Model-Mismatch: 1 when the provider names a model other than the profile’s or an alias.

A historical 2026-10-21 router object next to choices, shown for its response shape (reasons state difficulty on 0–1, so a rating of 1 reads 0.25). Its scores and bands predate v5; current responses also disclose index_version:

"router": {
  "version": "1", "mode": "route",
  "decider": {"id": "llm", "version": "deepseek-flash+46002df5", "input": "excerpt", "reason": "routine rename"},
  "policy": {"id": "floor-v1", "version": "2026-10-19"},
  "signals": {"difficulty": 1, "scale": [0, 4], "tags": {"code": 1.0, "math": 0.0, "long": 0.0, "rare": 0.0, "tools": 0.0},
              "p_needs_strong": null, "confidence": 0.8},
  "chosen": {"profile": "flash", "model": "deepseek-flash", "tier": "mid", "tier_source": "artificialanalysis:2026-10-21", "score": 39.5, "reasoning_effort": null},
  "minimum_score": 0.0,
  "default": "qwen", "undecidable": false,
  "reasons": ["difficulty 0.25", "tag code 1.00", "minimum score 0.0 (mid from 37.6, frontier from 49.6)",
              "code → cheapest code-capable profile scoring ≥ 0.0", "flash scores 39.5, mid (artificialanalysis:2026-10-21)",
              "candidates: flash $0.0013, qwen $0.0068, sol $0.0109 → flash"],
  "blocked": [], "warnings": [],
  "features": {"est_tokens": 310, "turns": 1, "tools": false, "images": false},
  "decision_latency_ms": 1171, "decision_cost_usd": 0.00012, "catalog_version": "2026-10-21",
  "usage": {"prompt_tokens": 300, "completion_tokens": 20}, "cost_usd": 0.000114,
  "provider_reported_model": "deepseek-flash", "mismatch": false,
  "latency_ms": 1830, "upstream_status": 200
}

A request with X-Router-Conversation also carries "pin": {"state": "new"|"held", "profile", "since", "requests"}.

Streaming: headers first, provider events unchanged, then a chunk with "choices": [] carrying router before data: [DONE]. If the stream was cut, closed early, idle over 30 s or sent an event over 64 KiB, that chunk carries error.code stream-interrupted, stream-idle, stream-limit or timeout (or a native profile’s error code), and upstream_status 502.

A mismatch on an HTTP profile is disclosed, not fatal; delegation-run fails on it (exit 70), so a native profile fails 502 provider-identity-mismatch.

Errors

{"error": {"code", "retryable", "profile"?, …}} plus X-Router-Error. A refusal before the body is read closes the connection. Provider failures keep their meaning and lose their payload: only provider-rejected-request relays a message, cut to 500 characters with key-shaped tokens redacted. Rows down to busy never reach a provider; “native” means a delegation-run exit code.

Measuring a decider

delegation-router eval --items ~/work/evals/router/items.jsonl --decider sage --output ~/work/evals/router/sage

The golden set lives outside the repository; its JSONL format is the contract (no importer for any particular corpus):

{"id": "easy-rename", "prompt": "Rename …", "paraphrases": ["Please rename …"],
 "outcomes": {"flash": {"correct": true, "cost_usd": 0.0011}, "qwen": {"correct": true, "cost_usd": 0.0093},
              "sol": {"correct": true, "cost_usd": 0.041}},
 "oracle": {"cheapest_correct": "flash"}}

outcomes is what each profile did when you ran it; the harness never dispatches a pool profile, it asks the decider once per prompt and paraphrase. For the decider and the rules-only baseline it reports accuracy, cost (outcomes plus decisions), optimal-selection ratio, robustness (same tier across paraphrases), undecidable rate, decision latency p50/p95, and a deferral curve sweeping the frontier cut 0.1–0.9 without new decider calls.

--output writes report.json, gate.json and decisions.jsonl: one line per prompt and paraphrase with the profile chosen, the signals, the reasons and the decider’s own answer (as decide --raw shows it; never the prompt). The gate is advisory; serve never reads it.

Intelligence Index v5 evidence

The shipped catalog identifies Intelligence Index 5.0. Its 8/20-point bands use only current-suite measurements. An effort without an exact measurement has no routing score, including named efforts for a model measured at a single unnamed setting. GLM-5.3-Flash and Mistral Medium 3.5 have no current v5 index; their old scores are not reused. Explicit profile tiers remain user declarations.

A newer local catalog with a different or missing index version is ignored. Refresh it to v5 before using it as an overlay; partial compatible overlays retain row-level version checks. The user’s router pool, default and decider are not changed automatically. A default may still be used on an undecidable request; its selection is disclosed and is not a benchmark qualification.

AA task cost and provider token prices are separate. Legacy coding/math fields are not relabeled as new v5 component scores. Model refresh uses the catalog’s explicit AA slug where available, preserving the DeepSeek 1017 identity.