Install & upgrade

Kaizen is a Kamiwaza extension. Installation, validation, build, and deployment all go through the extension CLI.

info

App Garden lists two entries. Deploy Kaizen, which these pages describe. Kaizen (Legacy) is the earlier product, kept only for sites that already run it; see Current install.

Validate first

export KAMIWAZA_BASE_URL=https://your-cluster.example/api
export KAMIWAZA_PUSH_REGISTRY=127.0.0.1:5001
./scripts/kz-ext preflight
./scripts/kz-ext validate .

Validation checks the extension contract: services, images, settings, secret metadata, and route ownership. It must pass before a build is worth starting. The read-only preflight separately proves that the platform API and registry are reachable. It accepts the delegated route's authentication challenge but rejects a missing route, server failure, or unreachable registry before any image work begins.

Three supporting checks keep the release inputs honest:

make check-version-sync        # product version agrees across manifests, including release/product.yaml
make check-release-contract    # images, bake targets, and external dependencies align
make check-owner-manifest      # owner inventory matches the v1 schema and Compose image set
make ci-validate               # extension validation plus these three checks

A version drift between the manifest and the built images is the most common cause of a deployment that pulls the wrong tag.

Build and deploy

Use the repository wrapper rather than invoking the platform CLI directly:

./scripts/kz-ext dev --revision <revision>

The wrapper keeps non-secret build state in sandbox-writable temporary directories and uses the cluster's loopback registry endpoint. Invoking the platform CLI directly from an automated environment fails in two predictable ways: the builder tries to write read-only state under the home directory, and registry pushes miss the loopback transport configuration.

The loopback endpoint is only where images are pushed from the build host. The image reference written into the extension resource is resolved by the cluster node, whose loopback has no registry behind it, so the two are deliberately different addresses. Leave KAMIWAZA_REGISTRY unset: the CLI then uses the node-routable registry the installer publishes as KAMIWAZA_EXTENSION_REGISTRY in kamiwaza/core-config (on a local k0s dev cluster, normally host.docker.internal:5001). Check what the platform publishes with:

kubectl -n kamiwaza get configmap core-config \
  -o jsonpath='{.data.KAMIWAZA_EXTENSION_REGISTRY}'

Set KAMIWAZA_REGISTRY explicitly only when that prints nothing, and then to a registry the node's containerd is configured to pull from. The wrapper refuses a loopback value (localhost, 127.0.0.0/8, ::1) for a remote dev run, because every pod would fail to pull it.

The wrapper prepares the local build environment and delegates deployment to the extension CLI exactly once. The CLI owns the complete lifecycle: it creates a missing extension on first install and uses a zero-downtime update for an existing deployment. The wrapper does not read or patch the Kubernetes extension resource.

A complete remote image cycle requires a 50 GiB free-space budget by default. That floor applies only to commands that build; help, unload, and explicit --no-build operations remain available for diagnosis and rollback when the host is below the build budget.

Building any of the nine release images requires connected build infrastructure to reach registry-1.docker.io for the pinned Dockerfile frontend, cgr.dev for Chainguard bases, and ghcr.io for the private, digest-pinned containers notice image. Before a local build, run docker login ghcr.io with an account allowed to read that package; keep credentials outside Git. The release workflow already has read access.

The API-family image build also needs openaipublic.blob.core.windows.net for the exact, hash-verified cl100k_base tokenizer payload. The build stores that payload in the image-owned read-only cache and proves it can load with networking disabled. Deployed API and worker processes do not need that runtime egress.

That generic update behavior does not apply to the Kaizen 0.4 identity cut. Never use the platform's image-only Upgrade All operation to replace either the outgoing Kaizen 2.x product or a pre-0.4 deployment from this repository. Follow the two hard-cut procedures below.

Installing from a manifest

For a cluster you want to provision declaratively, deploy/kaizen-install.yaml installs Kaizen with a single kubectl apply. It creates the deployment in the global workroom, so every member sees one shared instance, and it is safe to re-apply: an existing install is detected and left alone.

The manifest carries no deployment-specific values. It reaches Kamiwaza through the in-cluster ingress gateway, whose Service name is the same on every cluster, and reads the platform hostname the gateway routes on from Kamiwaza's own configuration. Supply an administrator credential and nothing else.

note

The manifest drives Kamiwaza's deployment API rather than writing a KamiwazaExtension resource directly. Applying that resource by hand produces a running but unregistered instance: App Garden lists deployments from Kamiwaza's records, not from the cluster, so a hand-written resource never appears there, never receives a generated field encryption key, and stays outside the stop, purge, and upgrade lifecycle. Treat a bare KamiwazaExtension as an internal detail of the platform, not an install interface.

Single-service rollouts

You can roll a single service, but read the revision-coupling rule first.

sandbox-worker is created on demand by the controller from whatever image tag the deployment selected. An api-only rollout is safe only after the matching worker tag has been pushed — otherwise the controller stays healthy while every new worker enters ImagePullBackOff, and Kaizen reports the create timeout as sandbox unavailability.

The wrapper checks the registry before a filtered rollout and refuses when the companion tag is missing. Do not work around that check.

Trusted Artifact renderer packs are also revision-coupled. Their separate release process produces paired derived API and web images from exact base and pack digests; it does not mutate a running Kaizen image or mount a shared pack volume. Roll the API-family and frontend images as one reviewed deployment. See Artifact renderer packs for the default-off install, exact-digest enablement, UAT, and rollback sequence.

Migrations

Schema changes are versioned migrations applied at deploy time. The api-backend's 15-minute start period exists to accommodate them alongside dependency verification.

Migration 0116_legacy_artifact_quarantine is a one-time data safety boundary for renderable Artifacts. It reclassifies every pre-attestation member_artifact row with kind=code and the reserved kamiwaza-renderable-artifact language as ordinary JSON code. Exact content, title, identity, lineage, versions, ownership, timestamps, and publication state are preserved. Operators do not need to run a manual SQL preflight. The migration's downgrade is intentionally a no-op because changing arbitrary JSON rows back to the reserved language would manufacture server attestation.

A database downgrade is not an operational rollback. Feature flags are the rollback mechanism. Schema or data cleanup is a separate reviewed operation, never a side effect of turning a flag off.

PostgreSQL 18

The database image moves to PostgreSQL 18 on a FIPS-validated base. A major version will not start against an older data directory, so this step is a migration rather than a restart: take a verified backup, then either dump and restore into a freshly initialized volume or run pg_upgrade against it. A dev Compose volume can simply be recreated with docker compose down -v.

Changing the base image can also change libc collation metadata. After the upgrade, reindex affected indexes and run ALTER DATABASE … REFRESH COLLATION VERSION before serving traffic.

warning

Do not start the new image against the existing volume to "see what happens". PostgreSQL refuses the mismatch by design, but the surrounding automation is easier to reason about if the migration is a deliberate step.

The server also runs in FIPS approved-only mode, where md5() is unavailable. Operator SQL and reporting queries that hash a value must use sha256().

Data volume ownership on an existing install

No Kaizen service pins runAsUser, runAsGroup or fsGroup any more, so the cluster assigns the identity each pod runs as. A fresh install needs nothing: the assigned identity creates its own directories on an empty volume. An existing volume was written by the identities the previous release pinned — postgres 70, seaweedfs 1000 — and the database directory is mode 0700, which no other identity can read whatever group it carries.

The symptom is a data pod in CrashLoopBackOff immediately after the upgrade:

chmod: /var/lib/postgresql/data/pgdata: Operation not permitted
initdb: error: could not access directory "/var/lib/postgresql/data/pgdata": Permission denied

Take the assigned uid from the pod the cluster just created, stop the two data services, hand their volumes to that identity with group 0, then start them again. Group 0 is the one group an assigned identity always carries, so this ownership works for the image's own uid and for an assigned one alike.

NS=kamiwaza                     # the namespace holding the deployment
EXT=kaizen-…                    # the deployment name
UID_ASSIGNED=$(kubectl -n "$NS" get pod -l app="$EXT-postgres" \
  -o jsonpath='{.items[0].spec.securityContext.fsGroup}')
kubectl -n "$NS" scale deploy "$EXT-postgres" "$EXT-seaweedfs" --replicas=0
# Run one short root pod that mounts both claims and re-owns them:
#   chown -R "$UID_ASSIGNED:0" /pg /sw && chmod -R g+rwX /pg /sw
kubectl -n "$NS" scale deploy "$EXT-postgres" "$EXT-seaweedfs" --replicas=1

warning

The repair pod needs root and must be deleted afterwards. On a cluster that refuses root outright — OpenShift under restricted-v2, which is why the pins were removed — there is no in-cluster repair: restore the backup into a freshly initialized volume instead.

Kaizen 0.4 identity transition

Kaizen 0.4 publishes the canonical App Garden key kaizen and tile Kaizen, taking that catalog entry over from the outgoing product. The product, Compose, package, image, API, and durable-state identities are kaizen too, so every identity now agrees and the temporary kaizen-next suffix is retired. This is not a rolling-compatible change, and App Garden Upgrade All must never cross the two products.

There are two different old products, and their procedures must not be mixed.

Current install

Install Product Kaizen from the kaizen template at version 0.7.6. deploy/kaizen-install.yaml targets that identity. It does not adopt, patch, or purge an outgoing Kaizen 2.x app: because that app now shares the kaizen deployment name, the installer treats it as a same-name match and fails closed rather than taking it over (see the adoption rules at the end of this section). Successful develop builds publish this identity to the development catalog automatically. Stage and production catalog promotion remains a revision-bound, explicitly approved operator action.

Every published entry names each Kaizen image by an immutable build tag and its content digest, in the form <repository>:build-<commit>-<run>-<attempt>@sha256:<digest>. After uploading, publication reads the stored entry back and fails unless those digests are the ones the build produced. Because the platform records a deployment's Compose body as it was launched, the deployment record identifies exactly which image bytes are running; two deployments of the same version string can be compared by digest rather than trusted to match. The 0.7.6 tags in the repository's docker-compose.yml are for local builds only and are not what a catalog install runs.

warning

The read-back detects a mismatch; it does not prevent one. When it fails, the entry it rejected is already in the catalog, and a platform that syncs before it is corrected can offer it. Treat a failed publication as a bad catalog entry to repair, not as a publication that never happened. The failed run keeps the catalog it replaced as the kaizen-catalog-backup-* workflow artifact; the repair procedure is in docs/catalog-publication.md.

note

A disconnected install relocates these images onto the site registry by tag, and the platform drops the source digest during that relocation. An offline deployment record therefore names images by the tag they were relocated under, not by digest. Record the bundle that supplied those tags where the next operator will find it.

On a platform upgraded from before v1.3.0, the kaizen selection can still be the running v3 product. In Extension version selections, choose 0.7.6. The panel labels it Different product (kaizen-v4) and opens Switch to a different product. Read the warning, select the acknowledgement that nothing is migrated, then choose Switch to kaizen-v4. Until the acknowledgement is selected, the switch is disabled; cancelling sends no change. The confirmed selection affects future deployments only. Existing v3 deployments keep running from their saved snapshots and are not migrated, upgraded, or restarted.

Create v4 as a separate deployment. If the v3 deployment remains in the same workroom under the default kaizen deployment name, set a distinct KAIZEN_DEPLOYMENT_NAME in deploy/kaizen-install.yaml before applying it. Before applying it, also confirm the workroom has no v4 deployment under the retired kaizen-next name; the installer refuses to create a second v4 instance while that deployment remains. Do not purge v3 merely to make room for v4, and never use Upgrade All to cross between the products.

Kaizen v3 is still published, under its own catalog entry kaizen-legacy ("Kaizen (Legacy)"), and is a supported install on this release rather than a leftover. It is a different entry from kaizen: installing one never resolves, adopts, or upgrades the other. Choose it only for continuity of a v3 deployment you already run; every new deployment takes kaizen.

info

Kaizen v3 is removed in v1.4.0, and no migration from Legacy to Kaizen is supported in any release. A Legacy deployment's data and configuration do not carry forward, so adopting Kaizen means standing up a new deployment and re-entering its configuration — plan for that rather than for a v1.4.0 upgrade path. Existing Legacy deployments keep running undisturbed across a platform upgrade to this release and are never migrated for you.

Disconnected site overrides and version identity

A disconnected site does not install from GHCR. It layers a site-owned values file that carries its own copy of the Compose body with mirrored image references. Three version strings are involved and they are not the same thing:

String	Where it lives	What it controls
Manifest version	kamiwaza.json, release/product.yaml, pyproject.toml, web manifests	The extension's own identity; make check-version-sync keeps these aligned
Catalog entry version	The packaged catalog's kaizen app template	Which packaged entry a site override merges over
Image tag	The refs inside the Compose body	Which build actually runs

The chart merges app template fragments by name plus version, so a site override that does not state the exact version of the packaged entry it is replacing is merged as a second, different entry — and the packaged one keeps winning. Read the version out of the packaged catalog before writing the override rather than assuming it matches the release you are deploying.

The image tag inside the override is independent of both other strings. A site may legitimately pin mirrored images from a release candidate under a catalog entry whose version string belongs to the packaged catalog. Record which build those tags correspond to somewhere the next operator will find it; the catalog entry version will not tell them.

warning

An override that supplies core.templates.apps replaces only the entries it names when the chart merges by identity. On a chart that treats the key as a whole-document replacement, the same file silently hides every other packaged template. Confirm which behaviour your platform release has before shipping a single-entry override.

Two deploy paths, one declaration

A cluster creates this extension's workloads one of two ways, and Kaizen has to run identically on both:

• Operator-managed. The platform operator reconciles the extension resource, creates the Services, rewrites sibling service references, and injects its own transport settings.
• Namespace-admin direct apply. With operator emission disabled, the scheduler applies Deployments and Services itself. It still rewrites service references and honours the x-kamiwaza declarations in the Compose body, but nothing injects the operator's settings.

Two consequences worth stating before an install:

1. Transport settings are not injected on the direct path. Anything the operator would have supplied — the internal platform gateway origin above — has to be set explicitly on api-backend, background-worker and document-worker.
2. Only some reference shapes are rewritten. Service references inside http(s):// URLs, after an @ in a credentialed URL, and bare host:port values are translated to the deployment's Service names. A scheme neither path understands — a redis:// URL, for example — is not, which is why Kaizen resolves its own Redis endpoint from the deployment prefix it can already see rather than trusting the reference to be rewritten.

Health checks are declared once, in x-kamiwaza.healthCheck, and both paths read them. A service whose cold start needs more than the default startup budget must declare that budget there — the Compose-level healthcheck block governs local Compose only.

Container identity is the cluster's to choose

Kaizen declares no runAsUser, runAsGroup or fsGroup for any service. It declares that containers are non-root, drop every capability, and run under the default seccomp profile — which is the security property — and leaves the identity itself to the image and the cluster.

That is one rule covering two regimes rather than a setting that selects between them. On ordinary Kubernetes the container runs as the numeric user its image declares. On a cluster that allocates identities — OpenShift's restricted-v2 SCC assigns a uid, gid and fsGroup from the namespace's own high range — it runs as the assigned one, and a pod that pinned an id outside that range would be refused at admission rather than started and corrected.

What makes that safe is in the images, not in configuration: each declares a numeric non-root user, and every path it writes at runtime is owned by group 0 and group-writable. Group 0 is the one group an assigned identity always carries, and the image's own user still owns those paths elsewhere, so a single ownership scheme satisfies both. There is no OpenShift mode to switch on, and adding one would mean shipping a posture nobody exercises by default.

note

If you fork an image or add a service, keep both halves. A named USER, or a runtime path owned uid:uid, reintroduces the failure on exactly the clusters that cannot report it clearly.

Everything below about outgoing Kaizen 2.x deployments concerns apps created before the cutover, which keep the names they were created under and are unaffected by this entry.

A cluster that installed Kaizen v4 before the cutover is running it under the retired kaizen-next name, and the same-name check cannot see it — it looks up kaizen. The installer scans the target workroom for kaizen-next separately and exits non-zero naming the deployment it found, rather than installing a second v4 instance beside it on its own PVCs. Retire it the same way as any other outgoing deployment: export, purge, then re-apply. It is reported, never adopted, patched, or purged for you.

Optional retirement of outgoing Kaizen 2.x

Outgoing Kaizen is a separate product with an incompatible five-service topology. It has no supported in-place data migration into Product Kaizen. You do not need to retire it to install v4 separately. Use this procedure only when the business has deliberately decided to remove the old deployment.

1. Export any outgoing-Kaizen data that the business must retain and preserve its last catalog manifest and image digests. This is an archive, not a Product Kaizen database restore.
2. Stop and purge every outgoing Kaizen deployment in the target namespace, then remove its old catalog/template entry. Purging deletes the extension CR and its deployment-owned PVCs, so the export must already be verified.
3. Confirm no old workload, route, or App Garden deployment remains. Do not use App Garden Upgrade All: that operation changes matching images only and preserves the incompatible old services, environment, volumes, and commands.
4. Confirm the separately installed Product Kaizen v4 deployment remains listed and healthy. Do not create another v4 deployment under the newly freed default kaizen deployment name. The coordinate an image path points at is a build location only; it is never the catalog template name.

The install Job refuses multiple same-name kaizen deployments in the target workroom. With exactly one match, it refuses adoption unless the deployment API proves both the Product Kaizen template identity and a 0.4 deploy-time Compose snapshot in runtime_artifacts. The API's top-level compose_yml tracks the linked template's current value and is not accepted as installed evidence. Missing evidence is an error, never an "already installed" success.

Migrating a pre-0.4 deployment from this repository

The prior version of this codebase can carry selected durable data forward, but the topology and runtime contract still require a hard cut. Do not use App Garden Upgrade All for this path either; it preserves retired process and sidecar-callback environment names, entrypoints, and service shape.

For a data-continuity migration:

1. Quiesce every pre-0.4 process and Admin write, then take a matched, verified backup of PostgreSQL and the object store. Preserve the exact field-encryption key with that backup. Do not run old and new processes against one database. On a raw Docker Compose host, the explicit pre-rename volume names make the old and new project names resolve to the same durable volumes, so never start those two Compose stacks side by side.
2. Fully re-render the Product Kaizen 0.4 deployment, or restore the matched backup and canonical field key into a fresh Product Kaizen deployment. A template image-only patch is not a deployment re-render.
3. Run the migration chain through 0116_legacy_artifact_quarantine. The preceding setting migration moves the six persisted model, connector, and approval-default keys. If a previous and canonical row both exist with different values, it aborts before changing any row; resolve that conflict explicitly and rerun. It also aborts if an unexpected pre-0.4 product-prefixed Admin setting remains, reporting key names but never setting values; migrate or remove those rows before rerunning. The package-rename migration verifies the image-baked Builder assets and materializes the complete canonical package records in place so existing UUID-backed agent grants survive. Missing or inconsistent assets, residual retired identity, or simultaneous old and canonical rows abort transactionally; correct the reported condition before rerunning. The chain then joins the migration histories, adds project-library access and durable orchestration commands, and quarantines historical member artifacts that used the now-reserved renderable language. That final quarantine preserves their stored bytes and version history while treating them as ordinary JSON code.
4. Explicitly set KAIZEN_A2A_POC_ENABLED, KAIZEN_A2A_DISCOVERY_ENABLED, KAIZEN_A2A_INBOUND_ENABLED, KAIZEN_A2A_OUTBOUND_ENABLED, and KAIZEN_A2A_SEND_ENABLED to the intended post-migration values. The old overrides were deleted and all five canonical settings default to false, so leaving them alone keeps A2A closed until the new peer identities are ready; a deployment that was running A2A before the migration must set them back to true explicitly. Start only the Kaizen 0.4 processes and verify the migration head before enabling traffic.
5. Replace PI_UNKNOWN_MODEL_MAX_OUTPUT_TOKENS with PI_MODEL_MAX_OUTPUT_TOKENS in deployment configuration and Admin overrides. The compatibility alias preserves the old value during migration, but that value now clamps every Pi model request rather than only unknown models. If both names are present, they must match or startup fails.
6. Require members to close every pre-0.4 browser tab and reopen Kaizen from its current App Garden entry. Browser lifecycle events are part of the hard cut: a tab retaining the old bundle neither shares Kaizen 0.4 logout and activity signals nor reaches Product Kaizen after its old route is retired. Closing it also removes any member data that bundle had already rendered.
7. Configure a new KAIZEN_A2A_SHARED_SECRET where the deployment-authored compatibility path is used, create fresh Agent Studio connection keys, and re-register every A2A peer before enabling the intended flags. The old secret, inbound-agent mappings, and all prior-release Agent Studio connection keys are deliberately deleted and are not restored by downgrade.

The database, roles, object-store bucket, queues, package path, and Compose volume names retain selected historical identifiers so existing durable data remains readable. Do not rename those stores as part of this application upgrade. Browser preferences and A2A wire identity use a hard cut and therefore do not carry forward.

First boot

On first start the backend runs migrations, verifies PostgreSQL, verifies the object store, and waits for the sandbox controller. The API and worker image already contains the integrity-checked tokenizer data used by document chunking and chat token accounting, so importing the service or starting a process does not bootstrap that data from the public internet. An offline installation does not need an exception for a first-start tokenizer download. TIKTOKEN_CACHE_DIR is image-owned; do not override or clear it.

App Garden generates and retains Kaizen's field-encryption key, and its service tokens and object-store keys, as deployment secrets; there is no value to invent in the launch dialog. Kaizen needs Kamiwaza 1.3.3 or later for this. Direct Compose installations generate the same values into .env with the commands in .env.example. Direct Compose installations that set KAIZEN_FIELD_KEY themselves must preserve that valid 32-byte base64 key with the database across upgrades and restores. Generate and append it to .env with the command shown in .env.example; never reuse an example or shared default key.

You do not need cluster-admin access merely because Kaizen has a sandbox. On an existing Kamiwaza installation, the extension operator provisions the sandbox controller's namespace and permissions. The default worker uses Kubernetes' normal container runtime. gVisor is optional and can only be installed and registered as a RuntimeClass by the cluster's platform administrator. If you are deploying to an arbitrary cluster without the Kamiwaza operator or its sandbox permissions, that is a missing platform prerequisite rather than something Kaizen can provision from an unprivileged extension install.

If it sits at not-ready, check object-store reachability first. A restrictive network policy that omits the seaweedfs:8333 rule produces exactly this symptom, and the error surfaces as a blob-store retry rather than an obvious policy denial. See Troubleshooting.

Air-gapped installations

Kaizen's bundled agent packages and its baked sandbox stack work with no route to the internet; the one built-in exception, pi-lens's language-server tools, is noted below. Customer package mirrors may exist, but nothing depends on one. Installing an extra package is the only thing that needs a registry.

What is already in the images:

• Bundled agent packages. The npm dependencies of every agent package Kaizen ships (Software Engineer's pi-hypa and pi-lens, Web Researcher's pi-web-access) are pre-installed in the package-runner image at build time from exact, integrity-checked lockfiles, for both architectures. The runner matches an admitted package to its tree by the package's own integrity hash and never calls npm for it. One residual: pi-lens's lsp_* tools download a language server on first use and therefore fail offline; its ast_grep_* and read_symbol tools, and every hypa_* tool, do not.
• Sandbox Python. pandas, NumPy, SciPy, Matplotlib, the Excel, Word, PowerPoint and PDF libraries, Pillow, ReportLab, Requests, Beautiful Soup, lxml, tabulate and PyYAML are baked into the sandbox image. See Code sandbox for the exact set.
• Tokenizer data, as described under First boot.

Recommended settings for an air-gapped installation:

Setting	Value	Effect
SANDBOX_EGRESS_MODE	isolated	Sandbox code gets no outbound network. The shipped default, public_internet, has nothing to reach offline and only produces failed connections.
SANDBOX_PACKAGE_REGISTRIES	{}, or your internal mirror	With {}, an import outside the baked stack fails at once and the error names what is available. Point it at an internal PyPI/npm mirror (exact allowed_hosts) to allow extra packages.
PKG_RUNNER_OFFLINE	1 (package-tool-runner)	A package that the runner image does not bundle fails immediately with an explicit error instead of waiting out an npm install timeout. Bundled agent packages are unaffected.

None of these is required for correctness: with the public defaults left in place, an unbundled package request now fails within seconds with an explicit "package registry is unreachable" error rather than a two-minute stall. They make the posture deliberate and the failures immediate.

Post-install configuration

Nothing about knowledge or models is assumed. After the stack is healthy, an administrator must:

1. Select models — chat, embedding, and optionally reranker, from the user menu → Admin area → Models, and save a Default chat model. Until one is saved, interactive chat uses whichever live chat model comes first, but Tasks and other background work cannot start. See Models & embeddings for three checks a first install most often misses:

   • a locally served chat model should declare a context window of at least 32,768 tokens (see Context window for local chat models);
   • redeploying the chat model on the platform does not carry Kaizen's selection over, so choose it again afterwards (see After a model is redeployed);
   • the embedding model must return vectors of at most 768 dimensions. The platform-provisioned llama.cpp all-MiniLM-L6-v2 (384) works; smaller vectors are zero-padded.

   Choosing an embedding target that does not serve an embeddings endpoint is the single most common misconfiguration.

2. Expose connectors — decide which discovered instances are tools, which participate in RAG, and which are neither. See Connectors administration.

3. Review feature availability — see Feature flags & rollout.

4. Review limits against your population. See Limits & quotas.

Upgrading

1. Read the release notes for behavior changes.
2. Remove any carried-forward TIKTOKEN_CACHE_DIR or DATA_GYM_CACHE_DIR setting from .env, Compose overrides, Helm values, and process environments. The new image owns /opt/kaizen/tiktoken-cache; either override fails closed with exit code 78 before API, background-worker, or document-worker startup.
3. Run make ci-validate against the new revision.
4. Confirm every image tag for the new revision is published — especially sandbox-worker.
5. Confirm the external controller source commit and digest in release-dependencies.json match the intended operators release, and read the release notes for sandbox sizing changes: a namespace quota on limits must fit the worker ceilings (SANDBOX_RESOURCE_*_LIMIT), and any lowered ceiling must not sit below its SANDBOX_RESOURCE_*_REQUEST, or the sandbox controller refuses to start.
6. Deploy with a full rollout unless you are certain the companion tags exist.
7. Watch the api-backend start period; migrations run here.
8. Verify: search returns results, a chat turn completes with citations, a run_code call leases a sandbox, task dispatch lag is flat.

Rolling the scheduler

Never change timezone data on only part of a scheduler deployment. Drain, roll all replicas, verify the exact identity, then resume. A split timezone-data version across replicas produces occurrence decisions that disagree.

Multipart upload writer cutover

The fenced multipart protocol requires a coordinated rollout. Stop multipart admission at the deployment's ingress and drain every already admitted old writer across all API replicas before enabling new writers; do not mix old and new writers. For rollback, quiesce writers and resolve or cancel pending sessions before starting old code. Follow the upload protocol, and qualify the cutover and storage provider in the target deployment; local checks do not prove either.

DOCUMENT_IMPORT_ENABLED is not a live drain switch: the upload routes are mounted at startup, and disabling the setting removes completion and cancellation routes too. It also disables scheduled multipart cleanup.

Sessions expire 24 hours after creation and must have been idle for at least one hour before cleanup. Each cleanup pass scans at most 64 expired sessions, every 15 minutes: at an uninterrupted cadence, at most 6,144 candidate scans per day per worker scheduling cleanup. This is not a guaranteed removal rate; provider failures, revisited sessions and slow passes reduce actual throughput, and a large backlog can outpace cleanup. A provider failure defers that session's cleanup eligibility by one hour. Monitor retained receipt and unregistered-byte growth and multipart_upload.cleanup_retry warnings; cleanup requires document import to be enabled.

Rollback

Feature flags first, in reverse dependency order. See Feature flags & rollout.

To reduce blast radius without a full rollback, lower the bounded caps — dispatch batch, backfill, queue, concurrency, runtime, token, cost, and output.

Before rolling an image back, decide and record whether queued and running work should drain or be durably cancelled. If a worker must be replaced immediately, preserve PostgreSQL and Valkey state so deterministic handoff repair can resume.

Keep definitions, runs, events, notifications, and audit evidence. Rollback is about stopping new behavior, not erasing what happened.

Verifying a deployment

./scripts/kz-ext validate .
make ci-validate

Then check, in order:

Check	Healthy looks like
Service health	All services passing their probes
api-backend readiness	2/2, not stuck at 1/2
Search	Returns ranked cards
Chat	A turn completes with clickable citations
Sandbox	run_code leases and returns
Tasks	kaizen_task_dispatch_lag_seconds flat
Ingestion	Uploads reach indexed state

Related

• Deployment topology
• Configuration reference
• Backup & disaster recovery