Skip to main content

Model gateway

Every model call in Kaizen goes through one gateway. Nothing — not the agent runtime, not a worker, not a packaged tool — holds a model credential or talks to a provider directly.

What the gateway owns​

ConcernWhat it does
RoutingSelects the deployment for the requested role: chat, embedding, reranking
PolicyApplies provider capability boundaries — a model is used only for what it is declared able to do
BudgetsEnforces per-turn and per-member ceilings before the call is made
EffortTranslates the turn's effort level into provider parameters
RetriesBounded, with the turn's idempotency preserved
NormalizationReturns one response shape regardless of provider differences
AuditRecords the call through the same append-only funnel as everything else

Because all of that lives in one place, adding a provider does not mean re-implementing budget or policy logic, and a provider quirk cannot leak into agent behavior.

Roles, not model names​

Kaizen assigns models to roles rather than wiring a model name into a feature. Roles include chat, embedding, and reranking, alongside internal roles for query planning, rewriting, and filter detection.

Several settings default to platform-selected, meaning Kaizen takes whatever the platform has assigned to that role rather than pinning a name that would drift out of date.

Provider profiles carry declared capabilities, and profile verification is cached (MODEL_PROFILE_VERIFICATION_TTL_HOURS, default 168) so capability checks do not cost a round trip per turn.

Effort​

Effort is a per-turn reasoning budget, defaulting to LLM_DEFAULT_EFFORT = medium. A member can set their own default and override it per conversation.

Effort is applied by the gateway, so it behaves identically for interactive chat, delegated sub-agents, prompt chains, and scheduled task runs.

For Pi routes, effort becomes a provider token field only when authenticated platform metadata names the exact reviewed parameter or identifies a supported serving engine. Python reserves answer capacity, binds the parameter and count into the run manifest, and the sidecar rejects any alternate field. Model names and member input never select provider sampling parameters.

Budgets​

SettingDefaultBound
LLM_PER_USER_DAILY_TOKENS0 (unlimited)Optional per-member daily cap when set to a positive number
LLM_LATENCY_BUDGET_SECONDS5.0Assist-path latency ceiling
AGENT_LOOP_BUDGET_MS120000Maximum inactivity between healthy model or tool events
AGENT_LOOP_MAX_CYCLES10General tool-bearing model rounds; a staged creation reserves up to 52 additional bounded protocol rounds
CHAT_CONTEXT_DEFAULT_WINDOW_TOKENS8192Assumed window when unknown
CHAT_CONTEXT_RESERVED_OUTPUT_TOKENS1024Held back for the answer
CHAT_CONTEXT_SAFETY_MARGIN_TOKENS512Headroom against tokenizer drift
CHAT_CONTEXT_RETRIEVAL_TOKEN_SHARE0.5Share of the window retrieval may claim
PI_MODEL_MAX_OUTPUT_TOKENS65536Operator provider-output ceiling
PI_MODEL_ANSWER_RESERVE_TOKENS4096Pi output capacity protected from reasoning

Exhausting the step budget does not fail the run. The engine grants one forced synthesis step so the member gets a cited answer from evidence already gathered. See Agent runtime.

Two embedding budgets​

Embedding runs under two deliberately separate budgets:

  • Interactive — SEARCH_EMBED_LATENCY_BUDGET_SECONDS (5.0), for query embedding on the member API path. Exceeding it degrades to lexical branches rather than making a member wait.
  • Bulk — INGEST_EMBED_LATENCY_BUDGET_SECONDS (120.0), for the worker's ingestion path. The packaged topology further caps uploaded-document batches at four inputs per physical Kamiwaza request so the platform request deadline is not mistaken for provider unavailability.

Keeping them separate is what stops a large reindex from making interactive search feel slow.

Prompt caching​

PROMPT_CACHE_ENABLED is off by default and gates both the prefix annotation and composer-time warming; SPECULATIVE_PROMPT_CACHE_ENABLED only narrows what happens once the master switch is on. Caching is applied by the gateway, so its correctness properties — never caching across members, never caching authority — hold uniformly rather than per feature.

Failure behavior​

  • A provider error is normalized into a safe category, including an error frame carried inside an HTTP-200 stream. Raw provider errors never reach a member response, an audit payload, or a task trace, and the LLM ledger records the call as an error rather than a transport success. Where the failure arrives in-band — an error frame inside an accepted stream, or a bridge retry that does not return a completion — the provider's own error type and prose are written to the server log instead, so an operator can name the cause without a packet capture. A request the platform rejects outright, before any stream is accepted, is reported by status rather than by body. See Observability.
  • A provider-confirmed HTTP 402 becomes the bounded provider_credit_exhausted category with member guidance to correct the balance, limit, response size, or model selection. The gateway never derives that label from untrusted provider prose.
  • A capability mismatch — asking a model for something its profile does not declare — is refused before the call rather than discovered in a malformed response.
  • A budget exhaustion is a policy outcome, not an exception: the turn ends with a bounded, honest result.
  • A Pi response that consumes its effective output limit without text or a tool call receives one same-effort semantic continuation. A second such response is classified as output exhaustion and is not transport-retried.
  • A Pi response that is structurally empty without reaching that limit — no text, no tool call, transport successful — is retried as a pre-commit provider failure, per request payload. Two bounds end the turn instead of retrying forever: the per-payload budget latches once spent, and a turn-wide count of consecutive content-free responses catches the case where the payload drifts between empty replies. Either produces the bounded provider_empty_response_exhausted category, which is not transport-retried. Any response carrying content resets both.
  • An unavailable role fails closed. Kaizen does not silently substitute a different deployment.

An OpenAI-compatible provider can place a sanitized pre-output error inside an otherwise successful HTTP event stream. Before any output is admitted, the gateway may replay that same trusted request once through the non-streaming transport to recover an authoritative status or bridge a healthy completion. It never replays after output begins, never changes the authenticated target, and never forwards the raw fallback response.

The replay is reserved for failures a second attempt could actually change. The replayed request is the same trusted request with streaming turned off, so a rejection that survives that one difference — a malformed request, an unsupported parameter combination, any 4xx other than 408, 425, and 429 — is answered from the first attempt rather than retried. A rejection that names streaming itself is the exception, and still earns the replay, because turning streaming off is precisely what the replay changes. The gateway reads the status out of the in-band error frame, since the streaming transport reports HTTP 200 even when the provider refused the call. A frame reporting a timeout, a 5xx, a transport failure, or no status at all is still treated as transient and still earns the one replay. The replay is driven by what the frame says: a socket that drops without sending one ends the turn rather than triggering it.

A status the gateway had to read out of the provider's own prose steers that retry decision but nothing a member sees. Only a status the platform supplied as a structured field can select a member-visible category, which is what keeps the provider_credit_exhausted rule below true: a provider cannot talk its way into a billing message by writing one into an error string. Where prose alone names a status that would carry member guidance, the replay is kept rather than skipped — that is how the gateway obtains the real HTTP response the guidance is allowed to come from, instead of answering a billing failure with an invitation to try again.

A turn that runs under delegated authority — a scheduled task, or any server-side run — opts out of the replay entirely and takes the direct stream. Its upstream failures are logged the same way, but no second attempt is made.

That last one has a practical consequence worth knowing: pointing the embedding role at a deployment that does not serve an embeddings endpoint produces ingestion failures and "hybrid unavailable" search degradation rather than a quiet fallback. See Models & embeddings.

Provider wire-shape normalization​

Every model call an agent makes crosses one seam: the run-scoped proxy at /internal/pi/model/v1/chat/completions. Two differences between otherwise healthy OpenAI-compatible deployments are corrected there, so a deployment that behaves correctly by its own contract does not degrade agent behavior.

Cumulative text. The streaming contract is delta-only — each chunk carries only the new characters. Some servers instead resend the whole accumulated content or reasoning_content in every chunk. A delta-only consumer appends those snapshots, and the answer visibly restarts and replays. The proxy detects snapshot mode from the stream itself and forwards only the unseen tail. The decision needs real evidence: a repeated fragment shorter than 24 accumulated characters is treated as ordinary language, not as a snapshot.

Tool calls rendered into text. Some chat templates emit the call as markup inside content and leave tool_calls empty. A consumer reading only the wire field sees a plain answer, and the tool the model asked for never runs. The proxy withholds completed call markup from the answer and re-emits it as a tool_calls delta. Both the JSON form inside <tool_call> and the nested <function=…><parameter=…> XML form are recovered, and XML values that are really JSON keep their types rather than arriving as strings.

Both corrections are driven by what the response contains. Neither inspects the model's name, an unclosed block is never dispatched — truncated arguments must not reach a tool — and a provider that sends real tool_calls is never second guessed.

Capability without a cataloged name​

The agent runtime resolves model behavior from Pi's generated catalog, keyed by model identity. A Kamiwaza deployment is free to carry any name, and an externally registered one carries no repository identity to fall back on, so that lookup routinely misses. Everything keyed off it would then degrade at once, silently.

Kaizen closes that with what the platform itself declares. The real context window is resolved Kaizen-side and always wins over the catalog. The reasoning declaration is read from the platform's authenticated capability vocabulary and carried on the run manifest, so a member's effort selection still reaches a deployment the catalog has never heard of. Reviewed thinking-budget parameters come from the same place.

Authenticated deployment metadata may also declare max_output_tokens, max_completion_tokens, or max_tokens. Kaizen persists that value with the accepted input and intersects it with the operator ceiling and one-token-less than the authenticated context window. A generated catalog profile supplies provider compatibility but does not override this concrete deployment authority. Missing output metadata therefore stays bounded by the operator and context limits rather than being guessed from a display name.

The rule is that a capability is either declared by authenticated platform metadata or it is absent. A cataloged profile still wins where one exists, and a model's name never grants a capability — that is what keeps an arbitrarily named deployment from inheriting behavior it does not have.

note

Modality is the one place a bounded identity fallback survives, so that older inventories exposing no capability markers keep vision. A deployment that declares its capabilities is never subject to it.

Sampling parameters a deployment refuses​

Some reasoning deployments — OpenAI's gpt-5 and o-series families, however they are named on the platform — accept only the provider's own default sampling temperature and reject any explicit value outright. Kamiwaza returns that refusal as 400 Check model_id and provider-specific request parameters, which fails the whole call rather than degrading it.

The gateway therefore omits temperature entirely for those families, keyed on the deployment's wire model id, and sends it for everything else. It is omitted rather than rewritten to the default, because the provider owns what its default is and a rewrite would quietly change the sampling of a call whose caller believed it had asked for something narrower.

This is the one place a model's name changes what Kaizen sends, and it is withholding a parameter rather than granting a capability — the rule above still holds. It matters operationally because every server-side gateway call shares this body: with such a deployment selected, manual /compact and the orchestration planner failed while interactive chat kept working, since the agent runtime builds its own request body and sends no temperature at all.

What the gateway will not do​

  • Accept a model choice from a request body.
  • Let a tool, package, or remote agent bypass it.
  • Return raw provider output into model context without normalization.
  • Carry a member's credentials to a provider.