Feature flags & rollout
Flags are Kaizen's rollback mechanism. A database downgrade is not — schema and data cleanup is always a separate, reviewed operation.
How flags behave
- Every flag is resolved server-side and re-checked by its backend route.
- A hidden navigation item is a convenience, never the access control.
- Turning a flag off stops new behavior. It does not delete retained data, history, or audit evidence.
- Invalid dependency combinations fail configuration validation rather than starting in an inconsistent state.
MEMBER_AGENTS_ENABLED,AGENT_ORCHESTRATION_ENABLED, andMEMBER_PROMPT_CHAINS_ENABLEDreach the Pi agent-runtime sidecar, so turning one on live is rejected whilePI_SIDECAR_URLis unset, and a stored override in that state is dropped rather than applied on reload.TASKS_ENABLEDcarries the same startup requirement. See Configuration.
The one dependency chain that matters
Collaboration, Projects, and history search must be enabled in this order:
1. SHARED_CHAT_COLLABORATION_ENABLED
2. PROJECT_WORKSPACES_ENABLED
3. CHAT_HISTORY_SEARCH_ENABLED
4. CHAT_HISTORY_INDEXED_RETRIEVAL_ENABLED (opt-in optimization)
All four ship enabled by default in the packaged topology except indexed retrieval, which remains opt-in.
Blast radius
The Projects flag is deliberately narrow. Its blast radius is the Project API and UI plus the Project-derived access path. Owner and private chat behavior, and eligible direct chat sharing, remain available when it is off.
Rollback, in reverse
- Disable indexed retrieval first. This retains strict exact authorized history search — the correctness oracle — rather than dropping straight to nothing.
- Then disable history search. This removes chat and Project branches. It does not restore whole-transcript scanning.
- Before disabling Projects, resolve any viewer whose only current access is Project-derived. Turning the flag off removes that path, and a member with no other path loses access.
Rollback preserves Projects, conversations, journals, direct grants, resources, and audit events.
Flags by domain
Core member surfaces
| Flag | Default | Gates |
|---|---|---|
MEMBER_API_ENABLED | on | The member API itself |
MEMBER_CHAT_STREAMING_ENABLED | on | Streamed answers |
MEMBER_CHAT_DELETE_ENABLED | off | Member-initiated chat deletion |
MEMBER_CHAT_FEEDBACK_ENABLED | on | Answer feedback |
MEMBER_DOCUMENT_PREVIEW_ENABLED | on | Source preview |
MEMBER_COLLECTIONS_ENABLED | on | Collections |
MEMBER_PROMPTS_ENABLED | on | Prompts |
MEMBER_PROMPT_CHAINS_ENABLED | on | Prompt chains |
MEMBER_AGENTS_ENABLED | on | Agents library |
DOCUMENT_IMPORT_ENABLED | off | Uploads |
DOCUMENT_EXPORT_ENABLED | on | Exports |
DESKTOP_INTERACTIVE_AUTHORITY_ENABLED | false (off) | Native Desktop admission and process-local input-claim authorization |
Enable Desktop interactive authority only in a native topology with the fixed
http://desktop-bridge:9000 bridge and matching KAMIWAZA_API_URL (that origin
or its /api base). This is a deployment setting, not a live Admin toggle.
Native input submission/resume needs a valid admission proof and UUID for a
desktop-local principal; ordinary server/PAT sessions cannot activate it.
Current input access, database lease, and native renewal checks fail closed.
Restarted processes and other replicas cannot recover memory-only authority
from stored inputs. Native re-admission or owner cancellation handles pending
work; recovery may reconcile expired terminal evidence without recreating
authority. Turning the flag off does not bypass checks on an existing Desktop
input. See Native Desktop input authority.
Agents and authoring
| Flag | Default | Gates |
|---|---|---|
AGENT_STUDIO_ENABLED | on | The guided builder |
AGENT_PACKAGES_ENABLED | on | Packages |
AGENT_BUNDLES_ENABLED | on | Shipped agent bundles |
AGENT_ROUTING_ENABLED | on | Automatic agent selection |
AGENT_ORCHESTRATION_ENABLED | on | Delegation |
AGENT_ORCHESTRATION_PARALLEL_ENABLED | on | Parallel sub-agents |
AGENT_ORCHESTRATION_EVALUATOR_OPTIMIZER_ENABLED | off | Evaluator-optimizer pattern |
AGENT_TOOL_DISCOVERY_ENABLED | on | On-demand tool descriptors |
AGENT_PIPELINE_SELECTOR_ENABLED | off | Pipeline choices for allowlisted Kaizen administrators; Chat's model and run settings otherwise show disabled Tomo |
FEDERATION_TOOLS_ENABLED | off | Federated cluster tools (link_federation_identity, federated_read, federated_write); also requires the FEDERATION_SHARED_IDP setting |
Delegation being enabled does not opt any member in — sub-agent selection is per member and per agent, and defaults to none.
Pipeline selection has two independent gates. AGENT_PIPELINE_SELECTOR_ENABLED
must be on, and the caller must have live Kaizen Admin authority with an opaque
scope in the deployment-bundled allowlist. The flag alone grants no selector
access. Turning it off removes selectable options and makes the protected
selection routes return 404. Every Chat composer still shows the disabled
Tomo baseline in the pipeline pane at the left of model and run settings.
Execution
| Flag | Default | Gates |
|---|---|---|
SANDBOX_ENABLED | on | run_code |
ARTIFACT_RUN_ENABLED | on | Running an artifact |
SANDBOX_EGRESS_ENABLED | on | Outbound network from sandboxes, public addresses only, applied without a restart. Setup shows the scope actually in force, which is isolation whenever the configured mode cannot be enforced |
SANDBOX_BROWSER_SESSIONS_ENABLED | off | Shared browser sessions; also requires egress, workers with at least 1.5 GB of memory (4Gi ships), and configured relay-only TURN |
TASKS_ENABLED | off | Scheduling and unattended work, only while platform-ready. Held for a later qualified release |
WORKER_QUEUE_ENABLED | on | Distributed queue |
DOCUMENT_WORKER_SEPARATE_QUEUE_ENABLED | off | Isolate document work |
Trusted Artifact renderers use the separate deployment-owned bootstrap
ARTIFACT_RENDERER_PACKS_ENABLED exact-digest allow-list rather than a boolean
flag. An installed pack remains inert until its complete id and OCI digest are
listed in the API-family environment. It cannot be changed by an Admin
Configuration database override. Disable it by removing that exact reference
before restoring the base images. See
Artifact renderer packs.
TASKS_ENABLED ships off in this release, which holds Tasks for a later
qualified release. With the flag off, members see no Tasks entry in the rail or
the mobile tab bar, direct Tasks links return to Home, Library pickers offer no
tasks, chat offers no task tools, the task API answers 404, and the scheduler
and task worker job are not registered. The setup checklist attributes the
closed surface to this deployment's own switch, not to a platform hold.
An administrator may turn Tasks on. That is a deliberate operator choice, not a qualified configuration of this release: the release's acceptance testing does not cover the enabled Tasks workflows. Turning the flag back off is the reversible rollout and emergency-stop control, but it does not cancel work already in flight — see Turning Tasks off with work in flight.
TASKS_ENABLED is necessary but not sufficient. The server computes one
decision from the flag and the exact 20-family delegated-workload v1 set,
every required component, a healthy registered attestation profile, compatible
model and dataset resource guards, and a fresh readiness report. The browser
receives only that decision and safe reason categories; it cannot infer or
combine platform state itself.
If static scheduler registration is incomplete, Kaizen reports
tasks_scheduler: held on /healthz with HTTP 200 so unrelated surfaces stay
ready and workers do not restart-loop. The health probe never contacts Core.
At request time, any false release term still keeps the rail, direct routes,
chat tools, scheduler wake, queue handoff, dispatch, worker claim, model calls,
credential use, and protected resource effects closed. Turning the flag off or
losing a dependency never deletes retained definitions, runs, notifications,
approvals, or audit evidence.
SANDBOX_BROWSER_SESSIONS_ENABLED ships off in this release. Browser media
is relay-only: both the member's browser and the sandbox worker must reach an
operator-provided TURN relay, Kaizen does not ship one, and BROWSER_TURN_URLS
ships empty. Shipped on, every agent would be offered browser tools that fail
the moment a member asks for a page. Turn it on only after configuring the
relay, as described in Configuration.
The flag gates the capability on the server rather than cosmetically. With it off, agents are wired no browser tools, and every browser-session route answers 404 before it touches a session or the sandbox pool, so a modified client has nothing to reach. The routes stay registered when the configuration store is on, so turning the flag on from Kaizen Admin takes effect without a restart. The chat pane neither looks for a session nor opens one from a link, and the administrator relay check reports the surface switched off rather than a relay fault.
It also has prerequisites it cannot satisfy on its own. Each of them fails closed rather than degrading to something weaker:
| Missing | Result |
|---|---|
| An egress policy that can mint a signed grant | Opening is refused as unavailable. There is no unmediated-network fallback |
| Chromium in the worker image, or a browser that will not launch | Refused as unavailable — the member sees a failure, not an empty pane |
| A free worker | Refused as busy. Sessions lease the same pool as run_code and do not jump its queue |
| A worker memory ceiling of at least 1.5 GB | Refused, naming the setting to raise. The shipped 4Gi clears it; a deployment that has lowered it does not |
That last row exists because the capability is gated by capacity as well as by
the flag, and capacity fails in a way a flag does not. Chromium needs about
1.4 GB, so the worker reads its own cgroup ceiling before launching and refuses
below 1.5 GB rather than letting the kernel take the browser mid-session.
SANDBOX_RESOURCE_MEMORY_LIMIT ships at 4Gi, well clear of that floor — if
you have tuned it below 1.5 GB, raise it before enabling the flag.
It is worth being precise about what that setting is, because the two are easy
to conflate. SANDBOX_RESOURCE_MEMORY_LIMIT is a ceiling, not a
reservation: it becomes the container's resources.limits.memory. The
reservation is the separate SANDBOX_RESOURCE_MEMORY_REQUEST, which ships at
256Mi. So 4Gi does not reserve 4 GB per worker and does not change what the
scheduler counts — a warm pool of eight reserves about 2 GB in total, not
32 GB.
The consequence is overcommit — sixteen to one on memory, forty to one on
CPU — and it is
deliberate: a worker running run_code rarely approaches the ceiling, so reserving for the worst case would waste most
of a node. A worker running a browser sits near it for as long as the member
keeps the page open, and so can an agent working a demanding build or test
job. Size the node for the number of concurrent heavy workers — browser
sessions and demanding agent jobs — you expect rather than the pool size, and
remember that when a node runs short of memory, the kubelet evicts the pods
using most beyond their reservation — a member's live browser or a busy agent
job, not an idle worker.
The flag is read when a session is opened. Turning it off stops new sessions immediately but does not tear down sessions already running; those end when the member who opened one closes it, or when the 15-minute idle reaper collects it. Plan a rollback around that window rather than assuming the capability is gone the moment the flag flips.
There is no data-cleanup step in that rollback, because a session holds no durable row. The browser profile — cookies, storage, whatever the member logged in to — is destroyed when the session ends, while the audit record of what was opened, navigated, and acted on survives independently.
Tool brokers
| Flag | Default | Gates |
|---|---|---|
KAMIWAZA_TOOL_BROKER_ENABLED | on | Generic Kamiwaza Core and MCP tools |
KAMIWAZA_SDK_TOOLS_ENABLED | on | Closed direct-SDK tools |
KAMIWAZA_EXTERNAL_CAPABILITIES_ENABLED | off | External source routes, management tool, schema discovery, and execution |
KAMIWAZA_EXTERNAL_CROSS_ORIGIN_SOURCES_ENABLED | off | Sources whose schema and API origins differ, with per-source consent still required |
KAMIWAZA_EXTERNAL_QUERY_AUTH_ENABLED | off | Operations whose authentication is injected into a query string |
KAMIWAZA_CREDENTIAL_REMEMBER_ENABLED | off | Separate explicit DataHub/Catalog credential persistence |
Public search has a separate JSON approval map, GENERAL_PUBLIC_WEB_BINDINGS,
which defaults to empty ({}). It is live under Admin → Configuration →
Agents & tools → Public web search approvals when the config store is enabled.
It approves only exact reviewed installation, operation, schema, platform, and
workroom bindings for query-organic-v1; current administrator tool selection,
member selection, and actor authorization remain independent requirements.
It does not widen General's public-only policy. Remove a binding or revoke its
Admin → Tools selection to deny subsequent calls, including calls to an already
advertised adapter. Current-policy read failure denies execution rather than
retaining stale approval. See the binding fields and example
before activation; neither a default nor an example confirms live availability.
External capabilities also fail closed unless the sandbox-egress proxy and the capability-plan signing key are configured. Disable Remember first, then the external capability switch, then egress when rolling back. Disabling either external capabilities or egress leaves registered source metadata in place but prevents new external calls; it does not delete sources or remembered DataHub objects. Disabling either narrower risk flag removes the affected operations from new tool discovery and rejects an already displayed credential form at submission.
Efficiency
| Flag | Default | Gates |
|---|---|---|
MEMBER_COMPRESSION_ENABLED | on | The member settings surface |
MEMBER_TOOL_RESULT_COMPRESSION_ENABLED | on | Tool-result compression availability |
MEMBER_PLAIN_SPEAK_COMPRESSION_ENABLED | on | Plain-speak availability |
MEMBER_PROMPT_FLUFF_COMPRESSION_ENABLED | on | Prompt trim availability |
CHAT_CONTEXT_BUDGET_ENABLED | on | Context budgeting |
PROMPT_CACHE_ENABLED | off | Prompt cache; master switch for the row below |
SPECULATIVE_PROMPT_CACHE_ENABLED | on | Speculative cache; inert while the prompt cache is off |
CONTEXT_COMPACTION_WARM_CANDIDATES_ENABLED | on | Warm compaction candidates |
Availability is not opt-in: all three compression tools default to off per member.
Retrieval
| Flag | Default | Gates |
|---|---|---|
HYBRID_RETRIEVAL_ENABLED | on | Lexical + semantic |
SEARCH_EXPANSION_ENABLED | off | Query variants |
SEARCH_FUZZY_FALLBACK_ENABLED | on | Near-miss fallback |
INGEST_CONTEXTUAL_RAG_ENABLED | off | Per-chunk generated summaries |
INGEST_MINI_CHUNKS_ENABLED | on | Fine-grained chunks |
INGEST_CONTEXTUAL_RAG_ENABLED calls a model per chunk. Measure the ingest
cost on a sample before enabling it across a large corpus.
Interop
| Flag | Default | Gates |
|---|---|---|
KAIZEN_A2A_POC_ENABLED | off | The A2A proof of concept |
KAIZEN_A2A_DISCOVERY_ENABLED | off | Serving agent cards |
KAIZEN_A2A_INBOUND_ENABLED | off | Accepting external tasks |
KAIZEN_A2A_OUTBOUND_ENABLED | off | Registering remote agents |
KAIZEN_A2A_SEND_ENABLED | off | Dispatching to them |
All five ship off, so a deployment that sets none of them peers with nobody and
serves no agent card. Turning the feature on takes the master switch plus each
stage you want: KAIZEN_A2A_POC_ENABLED alone still leaves discovery, inbound,
outbound, and send closed.
The stages are separate, but they are not independent: the Agent Card is served only when discovery and inbound are both on, so discovery alone exposes nothing. Do not enable inbound A2A on a deployment reachable from an untrusted network without an independent authentication layer.
The A2A screen saves each field against that field's current stored version, and it reads those versions back from the setup status after every save. Consecutive saves of the public address, the published agent mappings, or the capability secret therefore succeed, while a save made from a stale screen is refused with a conflict instead of overwriting the other administrator's change. Reload the screen and repeat the change when that happens.
During the Kaizen 0.4 identity cut, the migration deletes old A2A flag overrides and all prior-release peer credentials. Because the canonical flags above default to off, leaving them alone keeps A2A closed after migration. Set them explicitly only once new Kaizen credentials have been issued and every peer has been re-registered.
Turn engine
| Flag | Default | Gates |
|---|---|---|
TURN_ENGINE_ENABLED | on | The resilient turn engine |
AGENT_LOOP_FINAL_SYNTHESIS_ENABLED | on | Forced-synthesis grant |
PI_SESSIONS_ENABLED | on | Runtime sessions |
CONVERSATION_MEMORY_ENABLED | off | Cross-conversation memory |
Turning TURN_ENGINE_ENABLED off selects the low-level Pi Agent compatibility
path. It does not weaken model routing, tool policy, sandboxing, or the
PostgreSQL journal boundary. The Pi 0.84 session tree is a repairable rollback
projection, not a second recovery authority; named Pi lanes, operation records,
and mutable facts remain fail-closed until their mutations can share Kaizen's
atomic backend sequencing and recovery protocol. Unlike the turn engine, the
compatibility rollback does not exclude incomplete reasoning-only assistant
responses from model history. Setting TURN_ENGINE_ENABLED to off can therefore
replay that Thinking in a later provider request.
Rolling out a change
- Read the dependency order. Enable forward, disable backward.
- Change one flag at a time and let it settle.
- Watch the surface it gates, not just the service health.
- Record what you changed. Effective-config diff makes this cheap.
For Tasks, reconcile and activate workload and resource registrations first;
verify TokenReview and offline-baseline health, all capability/component
statuses, freshness, model/dataset guards, revocation, audit, and credential
broker paths; then enable TASKS_ENABLED. Roll back in reverse: drain or
durably cancel current runs while the flag is still on, because the owner
controls that end a run close with it, then disable the flag. Do not retire
registrations, revoke bindings, or stop workers until each accepted run and
effect has a terminal or explicitly recorded held state.
Reducing blast radius without a rollback
Lowering bounded caps is often better than turning a feature off: dispatch batch, backfill window and count, queue depth, concurrency, runtime, token, cost, and output ceilings all reduce exposure while keeping the feature working. See Limits & quotas.
Related
embedded browser sessions need the sandbox pool sized for a
browser. The controller ships with SANDBOX_RESOURCE_MEMORY_LIMIT: 4Gi,
clearing Chromium's enforced 1.5 GB floor. A custom lower limit refuses every
session and names that setting. See
Configuration.