Configuration reference
Kaizen carries roughly 400 settings. This page organizes the ones an operator actually reaches for, by domain. The extension manifest is the authoritative list; validation checks it on every deploy.
Settings are read from the environment and, where the config store is enabled, from the operator Settings screen. Secrets are marked as such in the manifest and are injected and redacted rather than stored in plain configuration.
Where to look first
| I want to… | Section |
|---|---|
| Turn a capability on or off | Feature flags |
| Choose models | Models & embeddings |
| Change a bound | Limits & quotas |
| Change scheduler behavior | Task scheduler operations |
| Change sandbox capacity | Sandbox pool sizing |
| Change logging | Below |
| Find out what a deployment must supply | Below |
What a deployment must supply
Only two settings are required unconditionally, and a deployment that supplies just those two starts:
| Setting | Purpose |
|---|---|
DATABASE_URL | Application database connection |
KAIZEN_FIELD_KEY | Field-encryption key; the Secrets section below covers the App Garden equivalent |
DATABASE_URL carries the password, and in the bundled Compose topology it is
built from POSTGRES_PASSWORD rather than restated. Set POSTGRES_PASSWORD
and both Postgres and every service that connects to it move together; an
installer that generates the database password for the Postgres service alone
used to leave the API authenticating with the development default and
crash-looping on password authentication failed for user "postgres". A
deployment with a managed database still overrides the whole connection string
through KAIZEN_DOCKER_DATABASE_URL, which wins over the derived default.
MULTIPART_PART_DB_CONNECTION_BUDGET defaults to 2 and accepts 1–32.
It limits multipart part PUTs that can hold a PostgreSQL transaction while
waiting for object storage, across all API replicas. The bundled Compose API
passes this setting through from .env; configure the same value on every
replica. It is a starting safety cap, not a measured capacity target. Size it
against PostgreSQL connection headroom for other requests and workers; see
Capacity and Limits & quotas.
Everything else either carries a default or is required only once you turn on the feature it belongs to. In particular, the Slack and Linear settings below are conditionally required — a Kaizen with both integrations off needs none of them, and asking for a Linear webhook secret to boot a deployment that has ticketing disabled would be a defect, not a policy.
| Setting | Required when | Enforced by |
|---|---|---|
SLACK_BOT_TOKEN | A connector of kind slack is configured | Connector preflight, at build time |
LINEAR_API_KEY or LINEAR_OAUTH_TOKEN or LINEAR_OAUTH_CLIENT_ID + LINEAR_OAUTH_CLIENT_SECRET | TICKETING_PROVIDER=linear | Settings validation, at startup |
LINEAR_TRIAGE_TEAM_ID | TICKETING_PROVIDER=linear, before a ticket is filed | Linear adapter, at first use — or at startup, if LINEAR_REOPEN_STATE_ID is a state name rather than a UUID |
LINEAR_REOPEN_STATE_ID | TICKETING_PROVIDER=linear | Startup probe, which resolves the value against the workspace |
LINEAR_WEBHOOK_SECRET | TICKETING_PROVIDER=linear | Webhook route construction, at startup |
PI_SIDECAR_URL | Any of MEMBER_AGENTS_ENABLED, AGENT_ORCHESTRATION_ENABLED, MEMBER_PROMPT_CHAINS_ENABLED, TASKS_ENABLED is on — the first three default to on; TASKS_ENABLED defaults to off | Settings validation, at startup, and again when one of the first three is turned on live |
PI_SIDECAR_URL is the base URL of the Pi agent-runtime sidecar, which the
bundled Compose topology supplies. Because the Pi-backed entry points default
to on, a deployment that assembles its own environment has to set it or refuse
to start; a blank value used to start cleanly and then fail every agent, chain,
and task turn at request time with sidecar_error, because the runtime was
retrying a path with no HTTP authority in front of it. The value must be an
absolute http/https URL with no credentials, query, or fragment. Only a
deployment with all four entry points off may omit it.
TICKETING_PROVIDER defaults to disabled, and the Slack data connector is
opt-in, so the default posture requires none of the above. Turning either on
without its configuration still fails loudly rather than degrading silently,
though the two fail in different places, and the Slack one fails wider than you
might expect.
Three of the Linear settings refuse at startup, so the deployment does not come
up: the credential rule refuses the settings object, the webhook route refuses
to mount without a signing secret rather than verifying deliveries against an
empty key every caller could guess, and the startup probe refuses when it cannot
resolve LINEAR_REOPEN_STATE_ID against the workspace.
LINEAR_TRIAGE_TEAM_ID is the one whose timing depends on another setting. Set
LINEAR_REOPEN_STATE_ID to a state UUID and the startup probe accepts it
without consulting the triage team, so a blank team surfaces later — at first
use, when the bot tries to file an issue. Set it to a state name like Todo
and the probe has to look the state up in the triage team's workflow, which
makes a blank team a startup refusal like the other three.
A Slack connector without its token refuses when connectors are built, and the
inventory is built in one pass: the first factory that refuses aborts the whole
build, discarding the connectors already constructed. So one blank
SLACK_BOT_TOKEN stops every configured connector's ingest, not just
Slack's. Nothing crashes — the scheduler and the ingest worker each catch the
failure, log it, and carry on with no connectors, and the config-store
reconciler logs and returns before it can schedule any. A deployment in this
state looks healthy and ingests nothing.
If ingestion stops for every connector at once, suspect a
credential on one of them. Search the logs for skip_misconfigured, which
matches both docs_ingest_cron.skip_misconfigured (scheduler) and
worker.docs_ingest.skip_misconfigured (queue worker); the reconciler logs
reconcile.build_failed. Each carries the refusing connector's error text.
Catalog installs never saw this distinction, because the platform supplies blank values for the four settings it passes; the Linear API credential is not passed at all, and a catalog install that leaves ticketing off never needs it. It matters for Compose, Helm, and raw runs of the published image, where nothing fills them in.
Service identity and logging
| Setting | Default | Purpose |
|---|---|---|
SERVICE_NAME | kaizen | Static service label |
SERVICE_VERSION | installed version | Override per rollout |
SERVICE_ENVIRONMENT | development | development / staging / production |
LOG_LEVEL | INFO | DEBUG … CRITICAL |
LOG_FORMAT | auto | json, console, auto (TTY → console, pipe → json) |
LOG_ACCESS_LOG_DROP_PREFIXES | /healthz,/health/readiness,/health/liveness | Silences health-probe access-log spam |
OTEL_METRICS_ENABLED | false | Mirrors every canonical metric to OTLP/HTTP when enabled |
OTEL_METRIC_EXPORT_INTERVAL_MILLIS | 60000 | OTLP periodic export cadence |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | SDK default | Metric-specific Collector endpoint |
OTEL_EXPORTER_OTLP_ENDPOINT | SDK default | Base Collector endpoint fallback |
OTEL_EXPORTER_OTLP_HEADERS | unset | Secret Collector authentication headers |
Leave LOG_FORMAT=auto. Without the drop-prefixes default, health probes are
the overwhelming majority of log volume on an idle deployment.
The OTLP exporter also honors the standard OpenTelemetry timeout, certificate, client-certificate, and compression variables. Configure those as deployment environment values; they are deliberately not runtime-editable settings.
Image-owned tokenizer cache
TIKTOKEN_CACHE_DIR is not an operator setting. The API-family image pins it
to /opt/kaizen/tiktoken-cache, bakes and integrity-checks every tokenizer
encoding referenced by the application, and verifies the cache again before
the API or a worker starts. Do not set, clear, or redirect this variable in an
App Garden request, Compose .env, Helm values, or raw-process environment.
An override fails startup rather than permitting a public tokenizer download.
Sign-in sessions
| Setting | Default | Purpose |
|---|---|---|
MEMBER_SESSION_IDLE_TIMEOUT_MINUTES | unset (None) | Optional browser inactivity timer, from 1 to 10,080 minutes |
MEMBER_BASE_URL | platform app URL | Trusted login/logout return root |
The inactivity setting is live in Admin area → Configuration → People & access. Leave it blank to disable Kaizen's timer and follow Kamiwaza's session expiry. A positive value adds an earlier browser deadline and invokes the same Kamiwaza logout flow as manual sign-out. It is not a backend authorization ceiling: a modified client can ignore the timer until Kamiwaza expires or revokes its session. Members receive a warning before expiry, and only browser interaction resets the deadline — polling and streaming do not.
Keep MEMBER_BASE_URL aligned with the public deployed app root, including its
/runtime/apps/<name> prefix. Kaizen validates every login and logout return
target against that hostname and base path. Include a port when an operator
must pin one explicitly. When the trusted root omits a port, Kaizen preserves a
non-default port from the external browser authority selected by the platform
ingress and carried through Kaizen's own web proxy; an authority naming a
different hostname, or a port that conflicts with an explicit configured port,
cannot replace the trusted root. If no trusted root is configured, Kaizen ignores caller-supplied deep
links and falls back to the request root; configure this value explicitly for
production deployments.
Agent-builder and sandbox streaming
| Setting | Default | Purpose |
|---|---|---|
MEMBER_CHAT_STREAM_HEARTBEAT_SECONDS | 15 | How often an otherwise silent agent-builder or sandbox chat response emits a keepalive, from 1 to 60 seconds |
Two surfaces stream a turn straight from the agent runtime to the browser: the agent builder's try-it-out chat and the Studio sandbox chat. Those responses stay open for as long as the agent runs and are legitimately silent whenever it is thinking rather than producing — a slow first token, a long tool call, or a write parked on the member's approval. This setting is how often such a response emits a comment frame during those gaps, so no proxy on the path mistakes a working turn for an abandoned one. Browsers discard the frame.
This does not govern the member's conversation event
stream. A member's conversation streams over GET /conversations/{id}/events,
which has its own keepalive on SHARED_CHAT_PRESENCE_HEARTBEAT_SECONDS
(default 10 seconds). The runtime keepalive is interleaved in the shared
runtime path, so it also traverses the internal orchestration relay — which is
why the relay's frame consumers skip comment frames rather than parsing them.
If ordinary chat is being cut off by a proxy, that is the setting to look at.
Change this only to fit a proxy you cannot reconfigure. The value must stay
comfortably below the shortest idle timeout anywhere between the member's
browser and api-backend — see Deployment
topology. Raising it
past your shortest idle timeout re-creates the problem it exists to prevent;
lowering it costs little beyond noise.
The keepalive holds the connection open. It does not make a dropped stream recoverable — see Streaming & event contracts for what a reconnect does and does not restore.
Retrieval and search
| Setting | Default | Purpose |
|---|---|---|
HYBRID_RETRIEVAL_ENABLED | on | Lexical + semantic |
SEARCH_DEFAULT_LIMIT | 10 | Results returned |
SEARCH_POOL_SIZE | 50 | Candidate pool before fusion |
SEARCH_MAX_PER_DOCUMENT | 3 | Caps one document dominating |
SEARCH_SNIPPET_CHARS | 280 | Snippet length |
SEARCH_EXPANSION_ENABLED | off | Query variants |
SEARCH_FUZZY_FALLBACK_ENABLED | on | Near-miss fallback |
SEARCH_EMBED_LATENCY_BUDGET_SECONDS | 5.0 | Interactive query embedding; the effective wait is capped at half of SEARCH_BRANCH_LATENCY_BUDGET_SECONDS (4.0 s by default), so raising it past that has no effect |
SEARCH_BRANCH_LATENCY_BUDGET_SECONDS | 8.0 | Per branch; half of it also caps the query-embedding wait |
HNSW_EF_SEARCH | 80 | Vector recall vs latency |
SEARCH_DEFAULT_COLLECTION_IDS | empty | Default scope |
QUERY_PLANNER_MODEL | platform-selected | Planning role |
Ingestion
| Setting | Default | Purpose |
|---|---|---|
INGEST_MINI_CHUNKS_ENABLED | on | Fine-grained chunks |
INGEST_MINI_CHUNK_TARGET_TOKENS | 64 | Mini-chunk size |
INGEST_MINI_CHUNK_OVERLAP_TOKENS | 16 | Overlap |
INGEST_EMBED_BATCH_SIZE | 16 | Batch size |
INGEST_EMBED_MAX_BATCH_TOKENS | 4096 | Token ceiling per batch |
INGEST_EMBED_MAX_INPUT_TOKENS | 240 | Per input |
KAMIWAZA_CONNECTOR_SYNC_INGEST_CONCURRENCY | 2 | Interactive connector embedding lanes in the packaged topology |
DOCUMENT_DENSE_INDEX_BATCH_SIZE | 4 | Uploaded-document inputs per packaged Kamiwaza request |
INGEST_EMBED_LATENCY_BUDGET_SECONDS | 120.0 | Bulk budget |
INGEST_CONTEXTUAL_RAG_ENABLED | off | Generated summary per chunk — costs a model call each |
DOCUMENT_HTML_STRUCTURED_DATA_MAX_BYTES | 67108864 | Valid embedded JSON admitted from one HTML document |
DOCUMENT_EMBEDDED_FIGURE_MAX_COUNT | 20 | Pictures read inside one document — 0 turns figure extraction off |
DOCUMENT_EMBEDDED_FIGURE_TEXT_MAX_CHARS | 20000 | Text recovered from those pictures per document |
EMBEDDING_CACHE_TTL_SECONDS | 604800 | Query-embedding cache |
EMBEDDING_CACHE_MAX_ROWS | 100000 | Cache bound |
Models and budgets
| Setting | Default | Purpose |
|---|---|---|
MODEL_ROLE_ASSIGNMENTS | — | Role → deployment mapping |
MODEL_PROVIDER_PROFILES | — | Declared provider capabilities |
MODEL_PROFILE_VERIFICATION_TTL_HOURS | 168 | Capability re-check interval |
LLM_DEFAULT_EFFORT | medium | Default reasoning budget |
PI_MODEL_MAX_OUTPUT_TOKENS | 65,536 | Operator output ceiling for every Pi model request |
PI_MODEL_MAX_OUTPUT_CONTEXT_SHARE | 0.25 | Maximum share of the resolved context window used for Pi output |
PI_MODEL_ANSWER_RESERVE_TOKENS | 4,096 | Output capacity protected from reviewed reasoning budgets |
KAIZEN_MODEL_CONTEXT_WINDOW_OVERRIDES | {} | Exact canonical model ID → reviewed context window when inventory omits it |
KAIZEN_MODEL_REASONING_BUDGET_OVERRIDES | {} | Exact canonical model ID → reviewed Pi reasoning-budget parameter when inventory omits it |
KAIZEN_MODEL_REASONING_TRANSPORT_OVERRIDES | {"Qwen3.5-9B":"qwen_chat_template"} | Exact canonical model ID → reviewed Pi reasoning wire shape when inventory omits it |
KAIZEN_MODEL_DEVELOPER_ROLE_OVERRIDES | {} | Exact canonical model ID → supported or unsupported for the OpenAI developer role when inventory omits it |
LLM_PER_USER_DAILY_TOKENS | 0 (unlimited) | Optional per-member daily cap; set a positive number to enforce it across API and background task workers |
LLM_LATENCY_BUDGET_SECONDS | 5.0 | Assist path |
LLM_GATEWAY_URL / LLM_GATEWAY_TOKEN | unset | Gateway endpoint; token is a secret |
PROMPT_CACHE_ENABLED | off | Prompt caching; master switch for both rows |
SPECULATIVE_PROMPT_CACHE_ENABLED | on | Speculative caching; inert while the switch above is off |
Agent loop and context
| Setting | Default | Purpose |
|---|---|---|
AGENT_LOOP_MAX_CYCLES | 10 | General tool-bearing model rounds per turn. Staged creation has a separate 52-round conservative recovery allowance; per-run append capacity derives from the trusted model-output ceiling and the artifact total remains 400,000 characters |
AGENT_LOOP_BUDGET_MS | 120000 | Maximum inactivity between healthy model or tool events; active long-running streams keep the turn alive |
AGENT_LOOP_MAX_PARALLEL_TOOLS | 1 | Concurrent tool calls |
AGENT_LOOP_FINAL_SYNTHESIS_ENABLED | on | Forced-synthesis grant on budget exhaustion |
AGENT_STREAM_CORRECTION_RULES | [] | Bounded JSON list of response patterns and fixed reminders |
AGENT_STREAM_CORRECTION_MAX_ATTEMPTS | 1 | Pre-commit correction attempts; 0–3 |
AGENT_TOOL_DISCOVERY_DIRECT_LIMIT | 10 | Deterministic directly advertised tool count |
CHAT_CONTEXT_BUDGET_ENABLED | on | Context budgeting |
CHAT_CONTEXT_DEFAULT_WINDOW_TOKENS | 8192 | Assumed window |
CHAT_CONTEXT_RESERVED_OUTPUT_TOKENS | 1024 | Reserved for the answer |
CHAT_CONTEXT_SAFETY_MARGIN_TOKENS | 512 | Headroom |
CHAT_CONTEXT_RETRIEVAL_TOKEN_SHARE | 0.5 | Retrieval's share |
CONTEXT_COMPACTION_MODE | eager | When to compact |
CONTEXT_COMPACTION_SOFT_RATIO | 0.60 | Start considering |
CONTEXT_COMPACTION_EAGER_RATIO | 0.78 | Compact now |
CONTEXT_COMPACTION_TARGET_RATIO | 0.50 | Compact down to |
CONTEXT_COMPACTION_WARM_CANDIDATES_ENABLED | on | Prepare off the critical path |
Each stream-correction entry is {id, pattern, reminder}. IDs must be unique
lowercase identifiers; patterns are limited to 256 characters and exclude
lookaround constructs; reminders are limited to 2,000 characters. The runtime
buffers response delivery only when at least one valid rule is active. A match
consumes another model step and the correction-attempt budget.
PI_MODEL_MAX_OUTPUT_TOKENS is live under Admin area → Configuration →
Models & roles. Pi uses the lower of this value, an output limit declared by
the authenticated concrete deployment, and the live
PI_MODEL_MAX_OUTPUT_CONTEXT_SHARE. The share is also editable under Models
& roles and defaults to 0.25. It treats advertised output capacity as a
maximum rather than a request size, preserving prompt space consistently for
cataloged and custom deployments. Keep the absolute ceiling within the
concrete deployments' served limits; raising it above a provider's accepted
maximum can make a deployment without complete metadata reject a turn before
producing output.
The earlier PI_UNKNOWN_MODEL_MAX_OUTPUT_TOKENS name remains an upgrade alias.
Kaizen preserves its configured value as PI_MODEL_MAX_OUTPUT_TOKENS, but the
meaning is broader: the ceiling now applies to every Pi model request. If both
names are present, their values must match
or startup fails. Replace the old name after reviewing whether its value is a
safe deployment-wide ceiling.
PI_MODEL_ANSWER_RESERVE_TOKENS is also live under Models & roles. It does
not add a sampling parameter to unknown deployments. Authenticated platform
metadata is authoritative for context and reasoning behavior. When that
inventory is incomplete, the two KAIZEN_MODEL_*_OVERRIDES maps provide an
exact-ID operator fallback; they never match display names or model families.
The reasoning map accepts only thinking_budget_tokens and
thinking_token_budget. Kaizen then caps that field so at least the configured
reserve remains inside the effective output ceiling. Restart the affected API
workload after changing either environment-only map.
Collaborative chat execution
| Setting | Default | Purpose |
|---|---|---|
SHARED_CHAT_INPUT_LEASE_SECONDS | 180 | Fenced input ownership window; active runtime calls renew it |
The runner renews an active input lease at most every 15 seconds and once more
when the runtime returns. A heartbeat can extend only an unexpired running
input owned by the same runner and lease token. Database failure, expiry, or
ownership drift cancels the runtime and leaves recovery to the journal instead
of allowing a stale terminal publication.
Current Kext installs use the application defaults for all
SHARED_CHAT_* limits, including this lease. The platform accepts at most
128 explicit service environment entries. Local Compose deployments can
override these values through .env; Kext operators cannot override them
until that platform limit changes.
Native Desktop input authority
| Setting | Default | Purpose |
|---|---|---|
DESKTOP_INTERACTIVE_AUTHORITY_ENABLED | false (off) | Native Desktop input admission and process-local authority for input claims |
This deployment-only setting requires the native Desktop bridge at the fixed
origin http://desktop-bridge:9000. KAMIWAZA_API_URL must resolve to that
origin or http://desktop-bridge:9000/api. Enable it only in a topology that
supplies the bridge's private versioned bind, refresh, and release protocol;
the setting is not a live Admin Configuration toggle. Private authority calls
use /_kzd/authority/v1/{bind,refresh,release} on that fixed origin and never
follow redirects to another endpoint.
Native admission applies to authenticated POST input submission and explicit
input-resume requests. It requires a desktop-local workroom principal,
one valid x-kamiwaza-desktop-admission proof, and one UUID
x-kamiwaza-desktop-admission-id. The native bridge binds the admission to the
accepted conversation, input, and canonical run. Ordinary server or PAT
sessions do not activate this private protocol. Invalid admission, an
unavailable bridge, or an invalid native reply fails closed.
The proof and renewed credentials remain in memory. Each renewal checks current conversation write access and the same active actor, input, run, audience version, and unexpired database lease before contacting the bridge. Revocation, cancellation, archiving, expiry, or unavailable current access prevents renewal. Release closes local authority even if the bridge is offline.
With the flag on, an input claim also needs authorization held by that process. A restart or another replica cannot reconstruct it from a durable input row. Use authenticated native re-admission through the input-resume path to restore eligible work, or authenticated owner cancellation to clear pending work. Recovery gives an orphaned Desktop run a 330-second grace window after its lease expires, then may reconcile terminal evidence; it never invents renewal credentials or resumes execution without locally held authority. Turning the flag off does not remove the authority requirement from inputs already marked as Desktop inputs. See Security guardrails.
Agent pipeline selector
| Setting | Default | Purpose |
|---|---|---|
AGENT_PIPELINE_SELECTOR_ENABLED | off | Enables pipeline choices only for allowlisted Kaizen administrators |
The pipeline pane at the left of Chat's model and run settings shows disabled Tomo, which remains the baseline for normal conversation creation. Enabling the flag does not grant access by itself. The caller must also have live Kaizen Admin authority and match the deployment-bundled opaque selector allowlist. Keep the flag off outside a reviewed comparison deployment.
Platform capability tools
The deployment-wide dynamic tool allow-list is stored under
KAIZEN_KAMIWAZA_TOOL_SELECTION and managed from Admin → Tools rather than
the general settings form. Its empty default is fail-closed for ordinary dynamic Core
and MCP tools. The closed SDK adapters remain default-on for compatibility but
honor an explicit kamiwaza_source:sdk or exact-adapter disable in the same
map. Exact generated tool names and Core-tag, SDK-source, or MCP-server
selectors share the same optimistic versioned setting; the current member's role, workroom, agent grant, and
conversation selection are still intersected at run time.
Public-web search uses a separate live approval map,
GENERAL_PUBLIC_WEB_BINDINGS, under Admin → Configuration → Agents & tools.
An empty map grants no reviewed public-search adapter. Each map key is a
generated MCP tool name;
its reviewed value must include extension_name, server_id, operation,
schema_sha256 (the exact input-schema SHA-256), api_base, workroom_id, and
contract: "query-organic-v1". General also requires a literal
"general_enabled": true in that binding. A seven-field binding from an older
deployment remains valid for Data but defaults to not approved for General,
even if the operation was selected globally. Add the flag only after reviewing
the provider, exact operation, schema, and workroom. An explicitly approved
binding lets General call only that operation in its workroom without selecting
the raw MCP tool globally. Data still needs global selection and can send only
a query grounded in the member's
own message. The exact bound raw MCP operation is reserved from Personal,
Data, default, and other ordinary assistants even when general_enabled is
false or absent. An unbound Serper operation retains the ordinary global
selection behavior for Personal and default assistants; Data still blocks its
unreviewed raw query path. An explicit false at the Kamiwaza root, MCP source,
MCP server, or exact tool takes precedence over the General binding. Member, workroom,
conversation, and current-schema checks still apply to every call.
Admin → Tools shows the global checkbox separately from each General
approval. Existing bindings lacking the flag appear as Needs General approval;
edit that binding's JSON under Admin → Configuration → Agents & tools to add
"general_enabled": true. Setting it to false revokes General access during
an active run. Block this operation for all profiles writes an exact
false with the current tool-policy version; it immediately revokes General's
approval while retaining the exact bound operation's raw-MCP reservation. Removing
a binding in Configuration also revokes General, but first check for broader
global selectors: without the binding, that raw operation may reappear for
ordinary assistants. The binding and policy are rechecked during an active run.
An administrator in another workroom may see the operation as unavailable from
their local discovery view. The exact deny remains available for a stored,
valid binding; enabling that operation still requires local discovery.
General-only recovery
After Block this operation for all profiles, recover General alone through
the existing authenticated configuration API. Turning the global checkbox on
writes an exact true: it also enables Data's reviewed public-search path,
subject to Data's member, workroom, conversation, and member-grounded query
checks. Removing the binding can expose the raw MCP operation to other
assistants; retain the reviewed binding instead.
- As a Kaizen App Admin with the config store enabled, call
GET /api/ops/effective-config. FindKAIZEN_KAMIWAZA_TOOL_SELECTIONand retain its entireconfigured_valuemap andversion. This hidden tool policy is not an editable field in the Configuration form. A default-source row hasversion: null; useexpected_version: 0for its first write. - Confirm the selected generated tool name has literal
false, then remove only that exact key from a copy of the map. Preserve every other key and value, including denials. Keep its complete reviewedGENERAL_PUBLIC_WEB_BINDINGSentry with literal"general_enabled": true. A remainingfalseatkamiwaza,kamiwaza_source:mcp, or the operation'smcp_server:<server-id>still blocks General; removing the exact denial does not override those parents. Review any such ceiling separately. - Send
PUT /api/ops/settings/KAIZEN_KAMIWAZA_TOOL_SELECTIONwith{"value": <entire preserved map minus that exact false key>, "expected_version": <version from GET, or 0 for default>}. For example,{"kz_reviewed_lookup": false, "unrelated_allowed": true, "unrelated_denied": false}becomes{"unrelated_allowed": true, "unrelated_denied": false}. This replaces the full map; sending only a partial map would discard unrelated policy. On409 version_conflict, stop and re-read effective config. Reconcile the concurrent administrator edit before preparing a new full-map request; never replay the stale map or blindly retry the write. - Re-read effective config and Admin → Tools to verify the saved version,
preserved map, and General approval. General-only access also requires
no other matching positive global grant for this operation: an exact
tool or MCP-server
truecan enable Data even with the exact key absent. Root and MCP-sourcetruevalues are ceilings, not grants by themselves. Preserve unrelated grants. Current binding/schema and ordinary member authorization still apply; successful configuration is not proof of a live search.
| Setting | Default | Purpose |
|---|---|---|
KAMIWAZA_TOOL_BROKER_ENABLED | on | Master ceiling for generic Core and MCP discovery |
GENERAL_PUBLIC_WEB_BINDINGS | empty ({}) | Exact reviewed public-search bindings for Data and General; independent tool and member authorization still required |
KAMIWAZA_SDK_TOOLS_ENABLED | on | Independent ceiling for the closed direct-SDK family |
KAMIWAZA_EXTERNAL_CAPABILITIES_ENABLED | off | Member source registration and external schema tools |
KAMIWAZA_EXTERNAL_CROSS_ORIGIN_SOURCES_ENABLED | off | Allow an explicitly marked schema to describe an API on another origin |
KAMIWAZA_EXTERNAL_QUERY_AUTH_ENABLED | off | Discover external operations that put credentials in URL query parameters |
KAMIWAZA_CREDENTIAL_REMEMBER_ENABLED | off | Explicit encrypted DataHub/Catalog Remember action |
KAMIWAZA_CAPABILITY_SIGNING_KEY | unset | Restart-required secret for actor/workroom/schema-bound plans; use at least 32 bytes |
These remain the operator-facing names. During extension deployment, Kaizen
maps them to KAIZEN_* runtime aliases because the SDK reserves
KAMIWAZA_* container environment keys for platform-owned ConfigMap values.
Both names resolve to the same settings in the process; do not configure the
runtime aliases directly.
Trusted Artifact renderer packs use one deployment-owned bootstrap allow-list:
| Setting | Default | Purpose |
|---|---|---|
ARTIFACT_RENDERER_PACKS_ENABLED | empty | Environment-only activation of image-installed packs by exact <pack-id>@sha256:<digest> reference |
Installed bytes remain inert while the list is empty. See Artifact renderer packs before changing it; API and web image receipts must remain byte-identical. The Admin Configuration database cannot override this trusted-code boundary.
External capabilities also require SANDBOX_EGRESS_ENABLED, a signing key and
proxy URL for that subsystem, and an egress mode/host/CIDR/port policy that
admits the intended origins. The external switch does not widen egress. Keep
cross-origin sources and query-string authentication off unless their narrower
risks have been reviewed; changing either live setting affects new discovery
and revokes incompatible outstanding credential forms. Keep
Remember off unless the deployment's DataHub/Catalog secret service is ready
for member-owned encrypted bundles. Use once never writes there regardless of
the Remember setting.
Data and General public-search approvals
GENERAL_PUBLIC_WEB_BINDINGS defaults to an empty JSON map, so it approves no
public search. With the config store enabled, it is live under Admin →
Configuration → Agents & tools → Public web search approvals. A binding is
keyed by the exact discovered MCP callable name and requires seven
nonempty string fields: extension_name, server_id, operation,
schema_sha256, api_base, workroom_id, and contract. The supported
versioned contract is query-organic-v1. General additionally requires the
literal boolean "general_enabled": true; an omitted or false flag grants no
General access. Legacy seven-field bindings remain valid for Data under its
ordinary allow-list and member-selection checks. Missing required fields,
unrecognized fields, a non-boolean approval flag, another contract, or a
malformed digest grant nothing.
For example, a reviewed installation named approved-install whose MCP
serverInfo.name is public produces server ID approved-install_public.
Its operation lookup has callable name kz_approved-install_public_lookup.
If its complete discovered parameter schema were:
{
"type": "object",
"properties": {
"query": { "type": "string" },
"category": { "type": "string", "default": "search" }
},
"required": ["query"]
}
the illustrative binding would be:
{
"kz_approved-install_public_lookup": {
"extension_name": "approved-install",
"server_id": "approved-install_public",
"operation": "lookup",
"schema_sha256": "21288dc51beab5721251966f417cba9c0050c29e3ff1242f69937a18b363f131",
"api_base": "https://platform.example",
"workroom_id": "room",
"contract": "query-organic-v1",
"general_enabled": true
}
}
Use the exact reviewed extension name returned by the platform inventory for
extension_name, including a deployment hash suffix when present. server_id
is a separate stable identity: discovery strips that suffix before combining
the installation name with the MCP server name, so an unchanged server_id
does not establish an exact extension_name match.
Replace these illustrative identities with the reviewed installation and the
member's effective platform scope. api_base pins that Kamiwaza platform
context; it does not select an arbitrary search endpoint. The digest is the
lowercase SHA-256 of the complete schema serialized as UTF-8 JSON with sorted
keys, compact separators, unescaped Unicode, and no non-finite numbers. Review
and update the binding when the schema changes rather than retaining an old
approval.
The contract requires a string query as its only required field. It permits
only the optional fields category, page, gl, location, and tbs;
category, if present, must default to search. The adapter exposes and sends
only a bounded query, and accepts public organic links and snippets. In Data,
the query must use the member's own words rather than terms introduced by
private sources. A remote description or read-only hint is not public-search
approval.
Admission independently requires the tool broker, actor authorization, the
matching installed extension/server/operation, and the exact current schema
and platform workroom scope. Data additionally requires the ordinary current
Admin → Tools allow-list and member/conversation selection. General uses its
separate explicit binding approval without requiring global selection of the
raw MCP tool; explicit root, MCP source, server, or exact-tool denials still win.
Admin → Tools keeps General approval separate from the global checkbox. Its
versioned Block this operation for all profiles action revokes General while
retaining the bound operation's raw-MCP reservation. Current binding and
administrator policy are read again before execution, including calls to an
already advertised tool. Removing or changing the binding, setting
general_enabled to false, or applying an administrator deny prevents
subsequent General calls. An unreadable enabled policy store fails closed
rather than using stale approval.
General remains public-only; approval grants no private data, executable tools,
or arbitrary MCP operations. These settings describe admission requirements,
not evidence that a deployment has activated search.
Federated clusters
FEDERATION_SHARED_IDP is a live JSON setting in Admin → Configuration
(Connectors / credentials). It lists the Kamiwaza federations members may call
from chat, and for each one the shared identity realm members link to:
{
"ORION": {
"issuer": "https://idp.example/realms/coalition",
"client_id": "kaizen-federation"
}
}
- The key is the federation name Core's mesh proxy accepts in
/api/mesh/<federation>/…. issueris the shared realm's public issuer URL. It must use HTTPS, and the realm must be one this platform lists in its trusted shared issuers.client_idis a public client registered in that realm for Kaizen. It needs the OAuth device grant enabled, theoffline_accessscope available, and the same attribute claims (for exampleclearance,country) the receiving cluster's gates read. Refresh-token rotation in the realm is supported: Kaizen stores each replacement and refreshes a member's link one call at a time. Like a pending link, that serialization holds within one api-backend replica.- Federations that share an
issuermust share itsclient_id. A member holds one link per realm, and a refresh token works only for the client that obtained it. An entry that names another client for an issuer already listed is ignored.
The tools also need the live flag FEDERATION_TOOLS_ENABLED (Admin →
Configuration, Feature flags), which ships off. With the flag off nothing is
wired whatever this setting holds; with it on, an empty setting, the default,
still wires no federation tools. A malformed entry is ignored.
Changes apply to new agent turns. Kaizen reaches each issuer with the platform
trust bundle (SSL_CERT_FILE), so a privately signed issuer certificate must be
in that bundle. See Tools for how the tools use the
linked credential.
Workers
| Setting | Default | Purpose |
|---|---|---|
WORKER_QUEUE_ENABLED | on | Distributed queue |
WORKER_QUEUE_REDIS_URL | redis://valkey:6379/1 | Queue backend |
WORKER_QUEUE_NAMESPACE | tomo:arq | Retained durable queue namespace, so an upgrade can see already-enqueued work |
WORKER_ROLE | all | Role of this worker process |
WORKER_CONCURRENCY | 4 | Jobs per process |
WORKER_JOB_TIMEOUT_SECONDS | 43200 | Job ceiling |
WORKER_METRICS_PORT | 9100 | Metrics |
WORKER_HEALTH_CHECK_INTERVAL_SECONDS | 30 | Health cadence |
DOCUMENT_WORKER_SEPARATE_QUEUE_ENABLED | off | Isolate document work |
Tasks
Tasks ship off (TASKS_ENABLED=false): this release holds
them for a later qualified release. An administrator may turn them on, but
that is not a qualified configuration of this release, and an upgrade keeps a
value the deployment already carries — see
Task scheduler operations before changing it
either way. When on, the flag requests rollout but does not grant
authority. Member, scheduler, dispatch, repair, worker, model, and resource
boundaries open only from Core's complete delegated-workload readiness.
TASKS_BACKGROUND_AUTH_MODE=core is the only production mode.
The previously shipped local value is accepted only so an upgraded install
can boot and be reconfigured; it never satisfies authority or scheduler
readiness.
| Setting | Default | Purpose |
|---|---|---|
TASKS_ENABLED | off | Reversible rollout intent, held off in this release; cannot override platform readiness |
TASKS_BACKGROUND_AUTH_MODE | core | Generic platform adapter; legacy local parses but remains unavailable |
KAMIWAZA_ORIGIN | injected by the platform | Public origin of the platform hosting Kaizen. Server-side platform authority derives from it, so Kaizen names no ingress controller and works the same on either one. Must be an absolute http(s) origin carrying no userinfo, query, or fragment; a routing prefix is supported. Anything else leaves Tasks unavailable and logs tasks.platform_origin_unusable at startup — the rest of Kaizen is unaffected |
KAMIWAZA_PLATFORM_GATEWAY_URL | injected by the operator when configured | Standard internal callback transport origin derived from immutable platform policy. It changes where bytes travel, not the registered path, Host, TLS authority, or authorization context |
KAMIWAZA_VERIFY_SSL | unset, which verifies | Certificate verification for outbound platform calls, server-side model and embedding calls to the platform's own hosts included; external model providers are always verified. The platform injects this only to turn verification off, so leave it unset unless you mean to disable it |
KAMIWAZA_INTERNAL_API_URL | injected by Core's extension runtime | Core's in-cluster Service URL. Member-authorized embedding goes here instead of the public origin when the host is a Kubernetes Service (<service>.<namespace>.svc...) |
KAMIWAZA_CA_BUNDLE | the platform's SSL_CERT_FILE when that names a readable file | PEM trust bundle for the platform calls the extension library makes, model discovery included. The platform mounts its own trust bundle and names it in SSL_CERT_FILE; Kaizen bridges that at startup, so a platform behind a private authority verifies without any further setting. Set it only to override the platform's bundle |
TASKS_BACKGROUND_AUTH_URL | KAMIWAZA_ORIGIN + /api/v1/delegated-workloads | Generic delegated-workload API base URL. Override only when the platform serves that contract somewhere else |
TASKS_PLATFORM_GATEWAY_URL | KAMIWAZA_PLATFORM_GATEWAY_URL, then KAMIWAZA_ORIGIN | Override for the server-side platform transport origin. A direct Core service is not a gateway route. Trusted model and platform URLs still supply the path, Host, TLS authority, and authorization context |
TASKS_BACKGROUND_AUTH_AUDIENCE | empty | Legacy task-runtime audience; projected workload identity does not require it |
TASKS_CONSENT_RETURN_BASE_URL | empty | Optional explicit override for the public Task consent return base |
KAMIWAZA_APP_URL | injected by platform | Exact public extension URL, including runtime path; queued chat turns use it for Core consent returns because their immediate request origin is internal |
TASKS_MAX_ACTIVE_AGENT_LOOPS_PER_OWNER | 25 | Maximum non-deleted server-created Agent loops for one owner |
TASKS_AGENT_LOOP_MAX_LIFETIME_SECONDS | 604800 | Maximum loop lifetime accepted from chat (seven days) |
TASKS_AGENT_LOOP_MAX_OCCURRENCES | 100 | Server cap applied to a loop's explicit occurrence limit |
KAMIWAZA_SERVICE_CLIENT_ID | empty | This deployment's own platform service identity. Optional; see below |
KAMIWAZA_SERVICE_CLIENT_SECRET | unset | Client secret for that identity — secret |
KAMIWAZA_SERVICE_TOKEN_ENDPOINT | empty | Token endpoint the identity is minted from |
TASKS_PLATFORM_GATEWAY_URL resolves through three members in order — this
setting, the operator-injected KAMIWAZA_PLATFORM_GATEWAY_URL (its origin
only), and finally KAMIWAZA_ORIGIN — so leaving it empty does not mean "no
gateway": it means the deployment's own public origin, which a pod normally
cannot reach, and in a cluster that is the configuration that makes model calls
hang until they fail as model_unavailable. Kaizen logs
platform_model_route_is_public_origin at boot on every service when the chain
resolves to the public origin inside Kubernetes.
Current Kext installs use the application defaults for
TASKS_MAX_ACTIVE_AGENT_LOOPS_PER_OWNER,
TASKS_AGENT_LOOP_MAX_LIFETIME_SECONDS, and
TASKS_AGENT_LOOP_MAX_OCCURRENCES. Local Compose deployments can override
them through .env; the current Kext environment limit does not expose them.
The service identity is how a run calls the platform as this
deployment rather than as the member, and only connector sync and model
traffic forward it. Workload attestation authorizes scheduled runs and
needs no secret here, but it does not reach a member's safe test. With this
identity a safe test calls the model as the deployment. Without it, the
member's own session is sealed into a single-use
draft test model grant
for that one test run; a test that reaches the worker without a usable grant
reports draft_test_model_grant_unavailable and offers Test again. Only a
deployment that can neither sign nor seal reports
platform_authority_unavailable, whose summary names an administrator.
The worker also logs worker.platform_service_identity_absent at startup.
Task traffic carries this deployment's workload attestation. If
the resolved platform origin is not HTTPS, or KAMIWAZA_VERIFY_SSL is off,
Kaizen logs platform_transport_unauthenticated at startup and the
assertion is readable and forgeable by anything on the network path. A
loopback origin is exempt, because a port-forward terminates locally.
Both api-backend and background-worker emit that warning at startup. The
worker is the process that actually presents the attestation, so its log is the
one worth reading.
Task traffic used to run without verifying the platform's
certificate — a defect, now fixed — and at the same time moved from an
in-cluster address to the platform origin over TLS. A deployment whose
platform certificate comes from a private or self-signed authority must
either carry KAMIWAZA_VERIFY_SSL=false, which the platform injects where
verification is deliberately off, or supply a trust bundle. The bundle the
platform mounts and names in SSL_CERT_FILE counts: Kaizen bridges it to
KAMIWAZA_CA_BUNDLE at startup, so model discovery and Task traffic verify
against the same authority. Otherwise the connection fails and Tasks stop
working. This is not reported at startup:
establishing it would require making the call.
With no platform origin injected and neither URL set explicitly,
Tasks are unavailable. That is deliberate — Kaizen will not guess an ingress
controller's service name — but it is otherwise indistinguishable from Tasks
being switched off, so both processes log tasks.platform_origin_absent at
startup naming the settings that would resolve it.
The extension manifest, rather than an environment variable, declares the
trusted workload shape. api-backend has the task-control role for consent
intents and run reservation/observation. background-worker has the narrower
task-executor role for claims, heartbeats, cancellation observation, fenced
run/effect transitions, credential use, and readiness. Both use the fixed
kamiwaza-workload-sts audience, a maximum 600-second assertion lifetime, and
the ordered TokenReview then offline projected-token profile set.
The platform's signing and nonce material belongs to the Kamiwaza cluster, not to this extension and not to an individual node. Node replacement must project the same preserved Core authority Secret. FIPS-regulated installations must use externally provisioned P-256 material generated in their validated provider; Kaizen never receives that private key.
Kaizen consumes one short-lived server-derived readiness report. Availability is
the logical AND of TASKS_ENABLED, the exact 20-family v1 capability set, all
required platform components, a healthy registered attestation profile,
compatible kamiwaza.model and kamiwaza.dataset v1 guards, report freshness,
Core authority configuration, and scheduler prerequisites. Projected workload
assertions and sender-constrained proofs replace legacy service-client bearer
credentials for normal runs. There is no setting that converts partial
readiness into a pass.
| Setting | Default | Purpose |
|---|---|---|
TASKS_CORPUS_MAX_SOURCES | 100000 | Sources per run |
TASKS_CORPUS_MAX_SOURCE_BYTES | 50 GB | Aggregate source bytes |
TASKS_CORPUS_MAX_SOURCE_ROWS | 100000000 | Aggregate rows/chunks |
TASKS_CORPUS_PARTITION_BYTES | 256 KiB | Bytes per partition |
TASKS_CORPUS_PARTITION_ROWS | 1000 | Rows/chunks per partition |
TASKS_CORPUS_PARTITION_TOKENS | 16000 | Tokens per partition |
TASKS_CORPUS_MAX_PARTITIONS | 250000 | Partitions per run |
TASKS_CORPUS_MAX_CONCURRENCY | 1 | Deterministic workers per run |
TASKS_CORPUS_MAX_PARTITION_ATTEMPTS | 3 | Attempts per partition |
TASKS_CORPUS_MAX_MODEL_CALLS | 250000 | Model calls per run |
TASKS_CORPUS_MAX_INPUT_TOKENS | 500000000 | Admitted source tokens |
TASKS_CORPUS_MAX_OUTPUT_RECORDS | 10000000 | Canonical output records |
TASKS_CORPUS_MAX_OUTPUT_BYTES | 100 MB | Canonical model output |
TASKS_CORPUS_MAX_STORAGE_BYTES | 500 MB | Combined exports |
TASKS_CORPUS_MAX_ELAPSED_SECONDS | 86400 | Wall-clock run budget |
These are hard operator ceilings. A Task may request a smaller value; preview stores and displays the effective capped definition. Missing or invalid policy blocks corpus work rather than selecting an unlimited default.
Sandbox
| Setting | Default | Purpose |
|---|---|---|
SANDBOX_ENABLED | on | Master |
SANDBOX_BACKEND | kubernetes | Backend |
SANDBOX_BROWSER_SESSIONS_ENABLED | off | Embedded browser sessions; turn on only with a configured TURN relay |
BROWSER_TURN_URLS | empty | JSON array of turn:/turns: relay URLs; required for browser sessions |
BROWSER_TURN_SHARED_SECRET | derived from KAIZEN_FIELD_KEY | Secret; TURN REST shared secret used only to mint per-lease credentials. Set it only to match a relay whose secret you do not control |
BROWSER_TURN_CREDENTIAL_TTL_SECONDS | 900 | Per-session TURN credential; 60–3600 seconds |
SANDBOX_CONTROLLER_URL | http://sandbox-controller:8085 | Controller |
SANDBOX_RESOURCE_CPU_REQUEST | 50m | Worker CPU reservation; set on the controller. Must not exceed SANDBOX_RESOURCE_CPU_LIMIT |
SANDBOX_RESOURCE_CPU_LIMIT | 2 | Worker CPU ceiling; set on the controller |
SANDBOX_RESOURCE_MEMORY_REQUEST | 256Mi | Worker memory reservation; set on the controller. Must not exceed SANDBOX_RESOURCE_MEMORY_LIMIT |
SANDBOX_RESOURCE_MEMORY_LIMIT | 4Gi | Worker memory ceiling; set on the controller. Browser sessions need at least 1.5 GB |
SANDBOX_STORAGE_SIZE | 1Gi | Worker workspace (/tmp/work) size; set on the controller. A scratch volume sharing the worker's 2 GiB ephemeral-storage limit, or the claim size when SANDBOX_PERSISTENCE is on |
SANDBOX_LEASE_BACKEND | redis | Shared allocator; required in controller mode |
SANDBOX_REDIS_URL | redis://valkey:6379/2 | Dynamic pool coordination |
SANDBOX_WARM_POOL_MIN | 3 | Warm instances |
SANDBOX_POOL_TARGET_IDLE | 1 | Idle reserve |
SANDBOX_POOL_MAX_ALLOCATED | 32 | Hard ceiling |
SANDBOX_MAX_CONCURRENT_CREATES | 3 | Create concurrency |
SANDBOX_ACQUIRE_WAIT_SECONDS | 30 | Lease wait |
SANDBOX_ACQUIRE_QUEUE_MAX | 256 | Queue depth |
SANDBOX_MAX_CONCURRENT_PER_MEMBER | 2 | Per-member fairness |
SANDBOX_MAX_WALL_SECONDS | 30 | Execution ceiling |
SANDBOX_MAX_OUTPUT_BYTES | 1 MiB | Output ceiling |
SANDBOX_MAX_ARTIFACT_BYTES | 10 MiB | Artifact ceiling |
SANDBOX_WORKSPACE_MAX_BYTES | 8 MiB | Workspace ceiling |
SANDBOX_HASHLINE_STATE_MAX_BYTES | 64 MiB | Trusted retained snapshot/register ceiling |
SANDBOX_HASHLINE_ENFORCE_SEEN_LINES | on | Refuse anchors not displayed under the tagged snapshot. Local Compose .env only: Kext installs always use the default, because the platform accepts at most 128 service environment entries |
SANDBOX_LEASE_TTL_SECONDS | 900 | Lease lifetime |
SANDBOX_LEASE_HEARTBEAT_SECONDS | 60 | Renewal cadence |
SANDBOX_WORKSPACE_PERSIST | on | Persist snapshots |
SANDBOX_WORKSPACE_BACKEND | object_store | Snapshot store |
SANDBOX_EGRESS_ENABLED | on | Outbound network; applies without a restart |
SANDBOX_EGRESS_MODE | public_internet | isolated, public_internet, private_network, or host_allowlist; applies without a restart. Any other value resolves to isolated, not to the default |
SANDBOX_EGRESS_HOST_ALLOWLIST | empty | Exact hosts for allowlist mode |
SANDBOX_EGRESS_PRIVATE_CIDRS | RFC1918 + IPv6 ULA | Reachable ranges in private/allowlist mode |
SANDBOX_EGRESS_ALLOWED_PORTS | 80, 443 | Destination ports |
SANDBOX_EGRESS_PROXY_URL | http://sandbox-egress-proxy:8000 | Proxy |
SANDBOX_EGRESS_PUBLIC_KEY_URL | in-cluster Kaizen API | Where the proxy reads its verifying key. No secret: signing keys are derived from the field key and never leave the backend |
SANDBOX_EGRESS_TOKEN_TTL_SECONDS | 600 | Per-run token; 60–14400 seconds |
SANDBOX_PACKAGE_REGISTRIES | PyPI and npm | Admin-owned registry map per runtime (index_url plus exact allowed_hosts). Packages baked into the sandbox image never use it. Set {} on an air-gapped install (an unbaked import then fails at once, naming the baked stack) or point it at an internal mirror |
SANDBOX_PACKAGE_TOKEN_TTL_SECONDS | 300 | Package broker token |
SANDBOX_PACKAGE_MAX_INDEX_BYTES | 8 MiB | Per-index response ceiling; 1–64 MiB |
SANDBOX_PACKAGE_MAX_FILE_BYTES | 100 MB | Per-file ceiling |
SANDBOX_WORKER_TOKEN | generated per deployment | Secret; the bearer token the worker API requires |
PKG_RUNNER_OFFLINE | off | package-tool-runner only. On, a package the runner image does not bundle fails immediately instead of trying npm install. Bundled agent packages always run from their pre-installed trees. Recommended on air-gapped installs |
On an App Garden install, SANDBOX_WORKER_TOKEN needs no provisioning: Kamiwaza
generates it for each deployment, and the sandbox controller and the API receive
the same value from the deployment Secret. The controller refuses a worker
request that carries any other token, so the value must reach both from one
source. On Kubernetes with Kamiwaza 1.3.3 or later, each sandbox worker Pod
reads the token from a Secret through valueFrom.secretKeyRef, so the value
never appears in the worker Pod spec. A direct Compose install generates it into
.env with the command in .env.example. If the API starts with no value at
all, it derives one from the field key so that it never runs the worker
unauthenticated. The controller cannot derive a token, so the two then disagree
and every worker creation is refused. A sudden run of refused worker requests
after a manual configuration change points here.
It is worth knowing what it protects: the worker's HTTP API is what runs code and drives browser sessions, and it is reachable from inside the worker container — which is where untrusted member code runs. A deployment that reaches the worker with no token at all leaves that API open, which is why the value is derived rather than simply optional.
Warm minimum, pool maximum, target idle, burst cooldown, create concurrency, acquire wait, queue maximum, and per-member cap are live Admin settings. Kaizen validates their combined bounds and publishes one shared Valkey policy version; changing them does not restart the API or sandbox controller.
Egress enabled state, mode, host allowlist, private CIDRs, and allowed ports are available in Admin as restart-applied settings. Mode changes never bypass the master switch or missing signing key. Existing installations therefore remain closed until an operator deliberately enables the policy.
Controller mode with platform deployment identity rejects memory lease state
or an empty SANDBOX_REDIS_URL. This is a startup error, not a degraded local
fallback. Static-worker focused development without deployment identity retains
the in-memory compatibility seam.
A deployment that sets nothing gets no browser: the code default is off, and
in this release kamiwaza.json and docker-compose.yml ship it off too, because
media is relay-only and Kaizen does not ship a TURN relay. The image still
carries Chromium and the shipped worker ceiling still clears its launch floor,
so turning browser sessions on is a configuration change, not a rebuild.
To turn them on, configure the relay first and then set
SANDBOX_BROWSER_SESSIONS_ENABLED=true, from Kaizen Admin or the deployment
configuration. Browser sessions need four operator conditions while the flag
reads on:
- the sandbox egress policy can mint its normal signed browsing grant;
- workers have at least 1.5 GB of memory (the shipped ceiling is 4Gi);
BROWSER_TURN_URLSincludes at least one browser-reachable endpoint and one plainturn:URL withtransport=tcp; the platform sandbox NetworkPolicy must allow that exact authority and port; andBROWSER_TURN_SHARED_SECRETmatches that TURN service's REST-auth secret — or is left unset, in which case Kaizen derives it fromKAIZEN_FIELD_KEYand the relay must be configured with that derived value.
Only the address is genuinely operator-supplied. Leaving the secret unset is the
recommended path: the derivation is deterministic, deployment-unique, identical
on every replica without coordination, and rotates with the field key, so no
credential has to be generated, stored, or kept in sync — an installer computes
the same value for the relay's static-auth-secret. It uses HKDF-SHA256 over the
field key with salt kaizen/sandbox/turn/v1 and info
browser-turn-shared-secret, urlsafe-base64 without padding.
When the flag reads on and BROWSER_TURN_URLS is empty, startup logs
browser_sessions.turn_unusable at error. That pairing means the browser tools are
offered to every agent and fail when a member asks for one, so it is reported at
boot rather than discovered by a member; the member-facing failure says the
deployment is not configured for browser sessions and names no environment
variable. The shipped default avoids that pairing; it arises only when an
operator turns the flag on without a relay. After enabling, run the
administrator relay check (GET /api/ops/browser-sessions/relay) to confirm the
relay accepts this deployment's credentials before a member asks for a page.
Kaizen refuses a session before contacting the media adapter when TURN configuration is missing or malformed. It never sends the shared secret to the member or worker. Instead it derives a username from the lease expiry and a hash of the session and actor, signs that username using the TURN REST HMAC contract, and sends the short-lived result. The browser media client is relay-only, so a STUN-only list is not a fallback.
Use regional managed TURN pools for hosted deployments. A customer-managed coturn service is compatible when it enables the same shared-secret authentication. Dynamic sandbox Services remain ClusterIP-only in both cases; do not add per-session NodePorts or public pod addresses.
The sandbox controller's worker memory ceiling applies to every worker in the
shared pool, not only workers currently showing a page. A worker below the
browser minimum refuses startup and names SANDBOX_RESOURCE_MEMORY_LIMIT
rather than failing later under an OOM kill.
A worker reserves 50m CPU and 256 MiB and may burst to 2 CPUs and 4 GiB: the
scheduler counts only the reservation, and the ceiling leaves room for an agent
working a moderately hard job. All four values reach the sandbox controller
explicitly, and the controller refuses to start when a reservation exceeds its
ceiling, with a SANDBOX_RESOURCE_* error such as
requests.cpu (50m) exceeds limits.cpu (25m). So to lower a ceiling below the
shipped reservation, lower the matching reservation as well — for example
SANDBOX_RESOURCE_CPU_LIMIT=25m together with
SANDBOX_RESOURCE_CPU_REQUEST=25m. A zero or negative ceiling is refused the
same way, because Kubernetes reads a zero limit as no ceiling at all. The four
values are deployment environment, not Admin settings: they are rendered into
the controller and into the extension's sandbox declaration
(spec.sandbox.resources) when Kaizen is deployed, so changing them means a
redeploy. The controller applies them to the worker pods it creates. Under the
platform operator, workers are claimed from a platform-owned pool, and whether
that pool follows the declaration is the platform's decision; the controller
still refuses a contradictory pair at startup there.
Under the platform operator, each claimed worker has a fixed deadline that the platform sets when it creates the claim. Kaizen's lease heartbeat confirms the worker still exists but does not extend that deadline, so a browser session or long agent job still running when it passes loses its worker. Check the platform's claim lifetime before relying on long-lived sessions there.
Two bounds are fixed in the product rather than configurable. A session is idle-reaped after 15 minutes, and the streamed viewport is 1280×800. Neither is exposed as a setting, so do not plan capacity on being able to lengthen the first — a member who leaves a tab open still loses the session, and its logged-in profile, at the quarter hour.
Sessions and code runs draw on one pool. A deployment that
enables browser sessions on a pool sized only for run_code will see
busy refusals on both, because a watched page holds its worker for as long
as the member keeps it open — not for the 30 seconds an execution takes. Size
for concurrent readers before enabling. See
Sandbox pool sizing.
The Kubernetes default does not require gVisor. The Kamiwaza operator supplies
the controller, namespace, and worker lifecycle under its existing authority;
an extension installer does not need cluster-admin access. Leave
SANDBOX_NAMESPACE unset to use the platform-owned namespace and controller
permissions. An explicit namespace override requires an externally provisioned
namespace and its controller permissions. If the organization
requires gVisor, a cluster administrator must install it on eligible nodes and
configure the worker RuntimeClass. Package Creator inherits that deployment
choice and never selects or provisions a runtime itself.
Secrets
Settings carrying secrets are declared with password metadata in the manifest and must never appear in task rows, queue payloads, logs, audit payloads, or member responses.
Kaizen 0.4 rejects removed pre-0.4 product-prefixed names found in either the
process environment or .env instead of silently falling back to defaults.
The startup diagnostic lists names only, never their values. The temporary
exceptions are KAIZEN_APP_PASSWORD, KAIZEN_DOCKER_DATABASE_URL,
KAIZEN_AUTO_MIGRATE, TOMO_FIELD_KEY, and
TOMO_ALLOW_INSECURE_LEGACY_FIELD_KEY; no other retired environment name is
accepted. The first two are Compose interpolation inputs, while the remaining
three are narrow runtime migration aliases.
The Pi sidecar enforces the same hard cut independently because it does not
load the backend Settings object. Remove PI_KAIZEN_CALLBACK_URL and
PI_KAIZEN_CALLBACK_TIMEOUT_MS; their replacements are
PI_KAIZEN_CALLBACK_URL and PI_KAIZEN_CALLBACK_TIMEOUT_MS. Compose forwards
either non-empty retired name to agent-runtime only so that process can fail
closed instead of silently using its default. When both canonical and retired AUTO_MIGRATE aliases are present,
their boolean values must agree across process environment and .env; Kaizen
refuses an ambiguous migration instruction before applying a schema change.
Kaizen ships no shared default for any service credential. Kamiwaza generates
the Pi runtime token (PI_SIDECAR_TOKEN), the package runner token
(PKG_RUNNER_TOKEN), the sandbox worker token (SANDBOX_WORKER_TOKEN) and the
object store's access and secret keys (SEAWEEDFS_S3_ACCESS_KEY,
SEAWEEDFS_S3_SECRET_KEY) for each deployment. It stores them in the
deployment's Kubernetes Secret, and every service that needs one receives the
same value. This requires Kamiwaza 1.3.3 or later; an earlier platform cannot
generate the worker token or the object-store keys. Each of these is held only
in memory or rewritten on every start, so a fresh value is safe on a redeploy.
After a redeploy, Kaizen resets every sandbox worker it finds from before the
restart. A worker still holding the previous token fails that reset and is
replaced, so expect brief worker churn after the rollout.
The Postgres role passwords still carry a development default. They are applied
only when the database volume is first initialized, so a new value would lock
an existing installation out. Override POSTGRES_PASSWORD for any installation
reachable beyond its own pods.
Kamiwaza generates a deployment-scoped SECRET_ENCRYPTION_KEY when Kaizen is
launched from App Garden, stores it in the deployment's Kubernetes Secret, and
keeps it stable across restarts and image-only upgrades. Kaizen consumes that
value as its field-encryption key; no launch-time value needs to be entered.
Direct Compose installations may continue to set KAIZEN_FIELD_KEY explicitly.
It must decode to exactly 32 bytes from standard or URL-safe base64, and Kaizen
refuses startup when an arbitrary or malformed value is supplied. Keep an
explicit key stable for the lifetime of its database because changing it makes
previously encrypted credentials unreadable. Back up and restore the
deployment Secret together with the database; restoring only the database into
a deployment with a newly generated key makes those credentials unreadable.
For remote repository-wrapper kz-ext dev runs, a supplied key must match the durable
.kz-ext/dev-field-key; the wrapper refuses a mismatch instead of replacing
the key and orphaning encrypted development data.
For an App Garden restore or migration that must read existing ciphertext,
provide the old material under the canonical SECRET_ENCRYPTION_KEY deployment
secret before creating the deployment. The deprecated TOMO_FIELD_KEY alias is
for direct Compose or raw-process configuration only. If an App Garden request
supplies only that earlier name, Kamiwaza does not classify it as the generated
deployment secret and creates a separate SECRET_ENCRYPTION_KEY; Kaizen then
fails closed rather than silently choosing one key.
The repository wrapper and frozen SDK carry the durable field key through
remote kz-ext dev deployment to the API backend, background worker, and
document worker. The platform stores the canonical key in a deployment-owned
Secret. No manual key patch is needed after deployment. Reuse the existing
key when updating or restoring encrypted data; an explicit SDK checkout
override must support the same deployment contract.
Older direct-Compose installations that never configured a key used the
former public all-zero default
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=. Preserve the database before
upgrading. For a recovery-only boot, set that old KAIZEN_FIELD_KEY explicitly
and set
KAIZEN_ALLOW_INSECURE_LEGACY_FIELD_KEY=true; Kaizen emits a startup warning but
allows the old data to be read and exported. This flag does not rotate data in
place. After recovery or migration, shut down, configure a newly generated key,
remove the flag, and verify the migrated data before deleting the backup.
Never place browser bearer tokens, projected service-account assertions, DPoP keys or proofs, run/effect capabilities, broker handles, connector credentials, action tokens, approval decisions, MFA state, or copied member roles in configuration. There is no supported local production task-authority mode.
KAIZEN_APPROVAL_DEFAULT_MODE is the live workspace starting mode under
Admin area → Configuration → Agents & tools. It defaults to
never_ask (Never ask) and accepts default (Ask when needed),
ask_writes, read_only, or never_ask. Members may inherit it or choose a personal, per-agent,
per-conversation, or task mode. An existing member choice wins when the
workspace default changes.
This setting is a default, not an administrator security ceiling. Administrator policy, tool grants, MFA, resource ACLs, and sandbox controls remain separate deployment-owned floors and cannot be bypassed by any mode.
Organization theme
THEME_DIR points at a trusted, non-member-writable directory holding the
organization's theme.json, logo, and favicon. Leave it unset and the
deployment serves the stock Kaizen brand; nothing is baked into the image or
the bundle either way.
The distinction worth knowing is between a missing mount and a broken theme:
THEME_DIRunset, or set to a path with no readable directory behind it — the deployment starts on the stock brand and logstheme.mount_absentnaming the path. An orchestrator that injects the variable without its volume therefore cannot take the API down, and an intended white-label that silently did not arrive is still visible in the log.- A directory that is mounted but holds a malformed
theme.json, an oversized asset, a file whose bytes contradict its extension, or a symlink escaping the directory — startup fails loudly and names the offending file. Theme content is authored, so a broken one is a deployment error rather than something to paper over.
Below-AA color contrast is a startup warning, not a failure.
Inspecting effective configuration
The operator surface can report effective configuration and diff it, which is the right way to answer "what is this deployment actually set to?" — reading environment variables off a running container misses config-store overrides.