Upgrade Kamiwaza

Upgrade Kamiwaza in place: rerun the helm upgrade --install command from your environment's install page with the new --version and the same values files. An in-place upgrade keeps Kamiwaza's models, workrooms, users, data, and storage volumes, and existing extension deployments keep running.

A 1.3 values file has no cluster owner trust settings for model serving. From 1.4.1 the upgrade needs none, and no cluster owner: local CPU models then run under Kamiwaza's built-in CPU baseline profile. 1.4.0 refuses to upgrade without them. See Owner trust.

Supported upgrades

You can upgrade in place from a 1.3 release by following this page, and from one release of this line to a later one by rerunning the upgrade command. There is no in-place upgrade from Kamiwaza 1.2 or earlier: contact your Kamiwaza representative to plan an assisted migration to a new installation.

You cannot roll 1.4.0 back to a 1.3 release in place. The way back is to restore the backup you take before the upgrade, so that backup is required. See No rollback to 1.3.

Before you start

Read the release notes for every release after the one you run, up to and including the one you are installing. Each release lists any action it requires. See Release notes.
Back up. This step is required. The backup you take now is the only way back to the release you run, because 1.4.0 cannot be rolled back in place (see No rollback to 1.3). Back up Kamiwaza's PostgreSQL databases by running pg_dumpall in each Kamiwaza PostgreSQL pod:
```
(
  set -o pipefail
  umask 077
  STAMP=$(date +%Y%m%dT%H%M%S)
  for STS in core-postgres keycloak-postgres core-spicedb-postgres; do
    OUT="kamiwaza-$STS-$STAMP.sql.gz"
    if ! FOUND=$(kubectl -n kamiwaza get statefulset "$STS" \
           --ignore-not-found -o name); then
      echo "FAILED $STS: cannot read the cluster"
      continue
    fi
    if [ -z "$FOUND" ]; then
      echo "SKIPPED $STS: not found"
      continue
    fi
    if kubectl -n kamiwaza exec "$STS-0" -c postgres -- \
         sh -c 'pg_dumpall -U "${POSTGRES_USER:-postgres}"' \
         | gzip > "$OUT.partial" \
       && gzip -t "$OUT.partial" \
       && gzip -dc "$OUT.partial" | tail -n 5 \
         | grep -q 'PostgreSQL database cluster dump complete' \
       && mv "$OUT.partial" "$OUT"; then
      echo "OK $OUT"
    else
      rm -f "$OUT.partial"
      echo "FAILED $STS"
    fi
  done
)
```
The output must show OK for core-postgres and keycloak-postgres. core-spicedb-postgres is OK too, unless SpiceDB uses another datastore, when it is SKIPPED. If any line says FAILED, or one of the first two is SKIPPED, do not upgrade: check that kubectl points at the Kamiwaza cluster and run the backup again. Only a file named on an OK line is a backup. Each run writes new, time-stamped files and never replaces older ones.

Keep the files, your values files and the version you run now together, outside the cluster, and keep them after the upgrade: they are the only backup of this release, and backups you take later are of 1.4. The dumps contain credentials and user data, so store them as you store your other Kamiwaza secrets. They cover the platform databases only. The object store, the platform's Kubernetes Secrets (its generated passwords and keys) and its storage volumes stay in the cluster; do not delete them. Kamiwaza support restores from these backups if an installation has to be recovered. Then back up each Kaizen deployment:
- Kaizen v3 (Kaizen (Legacy), kaizen-legacy, in the App Garden): follow Back up and restore a Kaizen v3 deployment. A Kaizen v3 deployment has no object store.
- Kaizen v4 (Kaizen, kaizen): run pg_dumpall in its PostgreSQL pod, and archive the /data directory of its SeaweedFS pod. Restoring this backup is not documented; contact Kamiwaza support if you need it.
Find your values files, and the order you installed with.
Air-gapped installations: copy the new release into your registry, as described in Install Without Internet Access. Keep the previous release's engine images in your registry until every model deployment has been redeployed on the new release, and its extension images while deployments of those extension versions still run. Keep the chart and images of the release you run now as well, until you are sure you will not need to restore the backup from step 2. If the installation runs Kaizen v3, see Keep the Kaizen v3 images as well.
From 1.3: read Owner trust. From 1.4.1 it needs no cluster owner.
If you created the kamiwaza-edge-trust ConfigMap by hand, as the 1.3 prerequisites pages described, move its CAs into your values file: the CA that signs the Kamiwaza domain into global.trust.edgeAnchor.pem, and any other CA into global.trust.extraAnchors. Keep the CA files. Delete the ConfigMap right before you run the upgrade. The chart is now the only writer of that ConfigMap, and an upgrade that sets these values while one that Helm does not manage exists stops before it changes anything. If the upgrade stops for another reason, concatenate the same CA files into one and recreate the ConfigMap with the 1.3 command, so that 1.3 pods that restart meanwhile still trust the domain (if the chart already published it, kubectl create reports that it exists, which is fine), and delete it again right before you rerun the upgrade. See TLS Trust for Private Certificate Authorities.
Remove credentials from Keycloak and OIDC URLs. From 1.4.0, Core refuses to start while a Keycloak, JWKS, issuer, or OIDC endpoint URL setting contains user:password@. See Remove credentials from Keycloak and OIDC URLs.
If you created local users, move their passwords off argon2. From 1.4.0, Keycloak runs its FIPS build, which cannot verify the argon2 password hashes that Keycloak gives every user created with a password on 1.3, for example on the Local Users settings page. The upgrade stops before it changes any Kamiwaza resource while one of those passwords remains. Signing in on 1.3 does not change the hash by itself: set the realm's hash policy first, then have each user sign in once. See Move local users' passwords off argon2. If the upgrade has already stopped, see Upgrade halted: Keycloak passwords (argon2).
GreyMatter and BYO edge certificates: publish the edge CA. From 1.4.0, Context Service model calls to the platform's own origins verify TLS. On OpenShift with GreyMatter, or Istio or Envoy with an edge certificate the chart does not create or cannot verify (network.tls.mode: byo, the default and EZUA's mode; customerCA; or cert-manager with a custom issuer), the upgrade stops unless your values set global.trust.edgeAnchor.pem, global.trust.edgeAnchor.publiclyTrusted: true (only for an edge certificate from a public CA), or core.context.embedding.verifySsl: false. Values that already turn verification off, and a hand-made kamiwaza-edge-trust ConfigMap you have not yet moved into values, also let it pass. See Upgrade halted: Context TLS preflight.
Remove surface: extension from global.workroomManager. Kamiwaza 1.4.0 removes the standalone Workroom Manager extension, and the upgrade stops before it changes anything while your values still set global.workroomManager.surface: extension. Remove the key to keep the default, disabled, or set it to core for the built-in Workrooms page. See Workroom Manager extension removed.
Public installs, mirrored and air-gapped included: remove the extension operator. The public release does not support the extension-operator runtime. Remove extensionOperator.enabled: true from your values, and core.extensionRuntime.mode: operator if you set it, to keep the default direct runtime. From 1.4.0, on an install whose images come from ghcr.io/kamiwaza-ai/releases/, the upgrade stops before it changes anything while the operator is enabled. A mirrored or air-gapped install is not stopped, so its operator pod would fail to pull its image. See Extension operator not supported on public installs.
Kubernetes with Istio installations: if your values files set global.authz.extAuthz.serviceHost, it must name core-api.<namespace>.svc.<cluster domain> in Kamiwaza's namespace, for example core-api.kamiwaza.svc.cluster.local. From 1.3.3, any other value, such as core-api.kamiwaza.svc without the cluster domain, makes helm upgrade fail before it changes anything, with an error that the host does not resolve into release namespace "<namespace>". Remove the value from your values files, or correct it. Do not turn off the access check that the error message names instead: that removes authentication from model or tool routes.
Execution or dataset gates that use your own class: from 1.3.4, Kamiwaza loads a gate class only from its own gate modules, from an installed gate package, or from a package listed in AUTH_GATE_CLASSPATH_PREFIXES. After the upgrade, an execution gate whose class is not allowed makes every job submission fail, local ones included, and a dataset whose gate class is not allowed is not served. Check both before you upgrade, as an administrator:
- The execution gate: call GET /api/cluster/execution-gate.
- Each gated dataset, one whose properties include gate or gate_type: call GET /api/catalog/datasets/<URL-encoded URN>/gate. The response names the dataset's gate class.
A 403 with reason classpath_not_allowed means the class is not allowed. List its package in your values file before you upgrade, as described in Allow your gate's classpath.

Do not uninstall Kamiwaza before an upgrade. Uninstalling deletes the platform databases, the object store, and every extension's data.

Owner trust

Every installation now serves local models through tenant-mode inference. In 1.3 this was off by default, so a 1.3 values file has no owner public key and no cluster binding.

From 1.4.1, the upgrade needs neither, and no step by a cluster owner. After the upgrade, external models deploy, and local CPU models, the default embedding model included, deploy under Kamiwaza's built-in CPU baseline profile. That trust level is unattested: a tenant default, not owner-attested. GPU models are refused until a cluster owner signs a profile. The upgrade says so: the chart notes print WARNING: no cluster owner trust, including inference trust is UNATTESTED. See Owner trust levels.

Remove core.inferenceResources.enabled: false from your values files, if one sets it. With false the render fails, as it does on a new install; leave the value out, or set it to true.
Carry forward the values files of the previous install, including any core.inferenceResources values they set, such as bundleConfigMap. If they set only one of ownerPublicKey and clusterBindingId, set both or remove both: with only one, the upgrade stops before it changes anything, with owner trust is half-configured: .... To remove both from an install that had them, first check whether the namespace has an owner anchor:
```
kubectl get namespace kamiwaza \
  -o jsonpath='{.metadata.annotations.inference\.kamiwaza\.ai/owner-anchor}'; echo
```
An empty line means no anchor. Without an anchor, delete the profile bundle ConfigMap that bundleConfigMap names first, as in Move back to the built-in baseline. While it is still published, the upgrade stops before it changes anything, with owner trust was removed: .... The check goes by the ConfigMap name in bundleConfigMap, so when you restore the values, restore all three. With an anchor, do not delete the anchored ConfigMap: the anchor still selects it, and without it Core refuses every local model. Removing the Helm values there changes nothing, because the anchor wins; going back to the baseline needs the cluster owner to remove the anchor first.
Upgrade, as described in Upgrade.
Deploy your local models again, as described in After the upgrade.
Optional: move to owner-attested mode. This is the hardened option for a site that has a cluster owner, and what GPU models need. It can be done before or after the upgrade. Before the upgrade, the cluster owner signs with the new release's core image, from the release manifest, not the image the install runs now; see Move to owner-attested mode.

To check the values file before you upgrade, render the chart with it. The render needs no cluster access, and exits without an error when the file is complete. From 1.4.1, a file without owner trust renders too. This render cannot see the cluster, so it does not catch half-configured or removed owner trust; the upgrade itself stops on those before it changes anything (step 2 above):

helm template kamiwaza oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <new version> --namespace kamiwaza \
  --values kamiwaza-values.yaml > /dev/null

Pass every values file you install with, in the same order. On an air-gapped installation, use the chart in your own registry. To check for those two states as well, run the same command as a server-side dry run against the cluster, which changes nothing:

helm upgrade kamiwaza oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <new version> --namespace kamiwaza \
  --values kamiwaza-values.yaml --dry-run=server > /dev/null

Upgrading to 1.4.0

1.4.0 has no built-in baseline. Its chart refuses to render without both ownerPublicKey and clusterBindingId, with missing cluster owner trust, before the upgrade changes anything, even on an anchored namespace. Its Core refuses every model deployment, external models included, while either value is empty. Upgrade to 1.4.1 or later instead. On 1.4.0 the two Helm values are mandatory: set them as in Tenant-mode inference, step 6, from a profile signed as in steps 1 to 5 of that page.

Remove credentials from Keycloak and OIDC URLs

From 1.4.0, when authentication is enabled, Core refuses to start while a Keycloak, Keycloak admin, JWKS, issuer, or external OIDC endpoint URL setting contains embedded credentials (user:password@). The pods log RuntimeError: Refusing to start: <SETTING> contains '@' (or '%40') and restart in a loop. The settings checked are listed in the 1.4.0 release notes. Do this before the upgrade, and before a new 1.4.0 installation that reuses earlier values or overrides:

List the settings that contain @ or %40. Run the whole block in one shell. It prints setting names only, never values, and prints CHECK FAILED if kubectl could not read the cluster.

(
set -o pipefail
names='^(AUTH_GATEWAY_(KEYCLOAK(_PUBLIC|_ADMIN)?_URL|JWKS_URL|JWT_ISSUER|OIDC_(ISSUER|AUTHORIZATION_ENDPOINT|TOKEN_ENDPOINT|USERINFO_ENDPOINT|JWKS_URI))|KEYCLOAK_URL)$'
kubectl -n kamiwaza get configmap core-config -o json |
  jq -r --arg n "$names" '.data // {} | to_entries[]
    | select((.key | ascii_upcase | test($n)) and (.value | test("@|%40"; "i")))
    | "configmap/core-config: \(.key)"' || echo 'CHECK FAILED: configmap'
kubectl -n kamiwaza get configmap core-config -o json |
  jq -r --arg n "$names" 'if [.data // {} | keys[] | ascii_upcase | test($n)] | any
    then empty else "NOTE: core-config holds none of these settings; find where they are set" end' ||
  echo 'CHECK FAILED: configmap'
kubectl -n kamiwaza get deployment,statefulset,daemonset,job,cronjob,pod -o json |
  jq -r --arg n "$names" '.items[] | "\(.kind)/\(.metadata.name)" as $w
    | (.spec.jobTemplate.spec.template.spec // .spec.template.spec // .spec)
    | ((.containers // []) + (.initContainers // []))[]
    | .env[]? | select(.name | ascii_upcase | test($n))
    | if .valueFrom then "\($w): \(.name) (from a reference; check it by hand)"
      elif ((.value // "") | test("@|%40"; "i")) then "\($w): \(.name)"
      else empty end' | sort -u || echo 'CHECK FAILED: workloads'
)

No output means that neither core-config nor the inline env of any Deployment, StatefulSet, DaemonSet, Job, CronJob, or pod sets a value that needs a change. The block does not read settings loaded with envFrom from a Secret or another ConfigMap. Check those by hand.

Delete the user:password@ part of each listed setting, where you set it. If you manage the installation with GitOps, change it in the source, or the next sync puts it back. In an OIDC, JWKS or issuer URL, an @ in the path or query of a well-formed URL can stay. A listed KEYCLOAK_URL is ignored while AUTH_GATEWAY_KEYCLOAK_URL is set, but remove its credentials too.
Put a credential that Core needs in its own setting:
- Keycloak client: AUTH_GATEWAY_KEYCLOAK_CLIENT_ID and AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET.
- External OIDC identity provider: AUTH_GATEWAY_OIDC_CLIENT_ID and AUTH_GATEWAY_OIDC_CLIENT_SECRET.
- Keycloak administrator: nothing to move. Core reads KEYCLOAK_ADMIN_PASSWORD, which the chart sets from the password key of the keycloak-admin Secret.
- JWKS and issuer URLs need no credentials.
Load a client secret from a Kubernetes Secret, not as plain text in core-config.
Rotate every password that was in one of these URLs. Earlier releases wrote it to the logs in clear, and copies already sent to a log store stay there after you rotate it. To rotate the Keycloak administrator password, change it in Keycloak and in the keycloak-admin Secret together, then restart the Core pods; changing only the Secret makes Core's admin calls fail.

Move local users' passwords off argon2

From 1.4.0, Keycloak runs its FIPS build, keycloak-fips, which cannot verify a password hashed with argon2. Keycloak hashes passwords with argon2 in any realm whose password policy names no hash algorithm, and on 1.3 that includes the kamiwaza realm. So every user created with a password on 1.3, on the Local Users settings page or through the API, has an argon2 password, and could not sign in after the upgrade. Users who sign in only through an external identity provider have no Keycloak password and are not affected. If your values set global.fips.enabled: false, Keycloak keeps its standard image and you can skip this section.

The upgrade moves the accounts whose passwords the chart keeps in its own Secrets. It cannot move your users' passwords, because it does not know them, so it stops while any remain (see Upgrade halted: Keycloak passwords (argon2)). Keycloak replaces a password's hash when its user signs in, using the realm's current policy. A user who signs in after the realm's policy names pbkdf2 moves to pbkdf2 without changing their password; a sign-in before that keeps argon2.

Do this on 1.3, before you upgrade, in one shell:

Give the kamiwaza realm a pbkdf2 hash policy. Sign Keycloak's admin CLI in inside the Keycloak pod and print the realm's current password policy. The administrator password goes from the keycloak-admin Secret to the CLI through standard input, never on a command line:
```
NS=kamiwaza
POD=$(kubectl -n "$NS" get pod -l app.kubernetes.io/name=keycloak \
  -o jsonpath='{.items[0].metadata.name}')
kubectl -n "$NS" get secret keycloak-admin -o jsonpath='{.data.password}' \
  | base64 -d \
  | kubectl -n "$NS" exec -i "$POD" -c keycloak -- sh -c \
    'IFS= read -r p; KC_CLI_PASSWORD="$p" /opt/keycloak/bin/kcadm.sh config credentials \
       --config /tmp/kcadm.config --server http://localhost:8080 --realm master --user admin'
kubectl -n "$NS" exec "$POD" -c keycloak -- /opt/keycloak/bin/kcadm.sh \
  get realms/kamiwaza --config /tmp/kcadm.config --fields passwordPolicy
```
admin, the keycloak-admin Secret and its password key are the chart's defaults. If your values change them, use yours in the sign-in command: the Secret name and key in kubectl get secret, and the username in --user. auth.keycloak.initUsers.keycloak.adminUsername, adminSecretName and adminPasswordKey set the administrator, its Secret and its key. An empty adminSecretName or adminPasswordKey falls back to auth.keycloak.auth.existingSecret or auth.keycloak.auth.passwordSecretKey. With auth.keycloak.initUsers.enabled: false, the administrator is always admin, and only existingSecret and passwordSecretKey apply.

{ } means the realm has no password policy. Set the hash policy that the 1.4.0 upgrade itself sets:
```
kubectl -n "$NS" exec "$POD" -c keycloak -- /opt/keycloak/bin/kcadm.sh \
  update realms/kamiwaza --config /tmp/kcadm.config \
  -s 'passwordPolicy=hashAlgorithm(pbkdf2-sha512) and hashIterations(210000)'
```
If the realm already has a policy, update replaces it, so keep its entries: set passwordPolicy to the current policy followed by and hashAlgorithm(pbkdf2-sha512) and hashIterations(210000), without any hashAlgorithm or hashIterations entry it had. For example, length(12) becomes length(12) and hashAlgorithm(pbkdf2-sha512) and hashIterations(210000). Run the get command again to check. If you created users with passwords in another realm, set the same policy there.

The admin CLI is the supported path: Kamiwaza's edge does not route the Keycloak admin console, and a port-forwarded console cannot load. If you reach the console another way, you can set the same policy there: open the kamiwaza realm, then Authentication, Policies, Password policy. Add Hashing Algorithm with pbkdf2-sha512 and Hashing Iterations with 210000, and save.

Users created from now on get pbkdf2 passwords. The upgrade's password migration job sees that the realm already hashes with pbkdf2 and leaves the policy as you set it. Later in the same upgrade, though, the chart applies its own password policy to the kamiwaza realm. It replaces the hash algorithm, hash iterations, length, uppercase, lowercase, digits, special characters, password history and not-the-username entries with the values under auth.keycloak.realm.passwordPolicy, and keeps any other entry. A value you leave unset takes the chart's preset: pbkdf2-sha512 with 210,000 iterations, at least 15 characters with one uppercase letter, one lowercase letter, one digit and one special character, a history of 5 passwords, and not the username. So an entry such as length(12) lasts only until the upgrade, unless you also set it in your values (auth.keycloak.realm.passwordPolicy.length: 12) or turn the chart's policy off with auth.keycloak.realm.passwordPolicy.enabled: false. The hardened profile refuses both: it allows no value below its preset and never allows turning the policy off. Existing passwords keep working, but every password set after the upgrade, including a reset, must meet the chart's policy. See Local user password policy.
Have each local user sign in to Kamiwaza once. Each sign-in moves that user's password to pbkdf2. For a user who cannot sign in, reset the password instead, under Local Users in Settings: a reset also uses the new policy. You can also delete users who no longer need access.
Check that no user is left on argon2. List every password that Keycloak still holds as argon2, or whose stored hash names no algorithm. The query matches the stored text rather than parsing it as JSON, so a hash it cannot read is listed instead of failing the query:
```
kubectl -n "$NS" exec -i keycloak-postgres-0 -c postgres -- \
  psql -U keycloak -d keycloak <<'SQL'
SELECT r.name AS realm, u.username,
       CASE WHEN c.credential_data ~ '"algorithm"\s*:\s*"argon2"'
            THEN 'argon2' ELSE 'unreadable hash' END AS hash
  FROM credential c
  JOIN user_entity u ON u.id = c.user_id
  JOIN realm r ON r.id = u.realm_id
 WHERE c.type = 'password'
   AND (c.credential_data IS NULL
        OR c.credential_data ~ '"algorithm"\s*:\s*"argon2"'
        OR c.credential_data !~ '"algorithm"\s*:\s*"[^"]+"');
SQL
```
Some argon2 rows are expected: the upgrade signs in the accounts whose passwords the chart keeps, and moves them itself. Those are the administrator in the master realm (admin unless your values name another, see step 1, and temp-admin if it still exists), and in the kamiwaza realm each user listed under auth.keycloak.initUsers.users in your values (by default admin and svc-core; none when auth.keycloak.initUsers.enabled is false). The chart keeps each listed user's password in a kamiwaza-user-<username> Secret, and the upgrade moves only a listed user whose Secret exists. kubectl -n "$NS" get secret -o name | grep kamiwaza-user- lists those Secrets as a hint, but a Secret left behind for a user no longer in the list does not make that user expected. Any other argon2 row is a user who still needs to sign in or have their password reset. The upgrade also stops on an unreadable hash row, and the job's log names that account with (an unreadable hash): reset its password.

Remove the admin CLI's session file from the Keycloak pod:

kubectl -n "$NS" exec "$POD" -c keycloak -- rm -f /tmp/kcadm.config

Upgrade

As the identity that installed Kamiwaza, rerun the helm upgrade --install command from your environment's install page, with the new --version and every values file you installed with, in the same order. For a Kubernetes with Istio installation with one values file:

helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <new version> \
  --namespace kamiwaza \
  --values kamiwaza-values.yaml \
  --timeout 70m

Keep --timeout 70m (4200 seconds). Core's first start on a new release can take about an hour, and the upgrade's own jobs run inside the same helm upgrade, so a shorter timeout can fail an upgrade that would have succeeded. One of those jobs, the Keycloak password migration (see Upgrade halted: Keycloak passwords (argon2)), can take up to 20 minutes on its own: it waits up to 10 minutes for Keycloak to answer (auth.keycloak.fips.migration.waitSeconds: 600), then reads every user's credentials, about one admin request per user, twice. Its limit, auth.keycloak.fips.migration.activeDeadlineSeconds: 1200, covers about 10,000 users. For a larger Keycloak, raise that limit by about 60 seconds for each further 1,000 users, and raise --timeout by the same amount.

Your environment's install page has the exact command and any additions:

Kubernetes with Istio
OpenShift with GreyMatter. First do steps 1 and 2 of its upgrade steps. Then repeat step 3 with the new version. Publishing replaces the configuration files, so hand edits to them, such as an edge image, are lost; set them in your values file first, as in Keep site settings across publishes. Then rerun the install command. With Helm 4, keep --server-side=false on every upgrade.
Install Without Internet Access. Use the chart in your own registry, with the values from step 5 of that page.

Upgrade with helm upgrade --install against the cluster, or from 1.4.1 through Argo CD (see Upgrade through Argo CD, which also lists the upgrades Argo CD does not support). Never apply a rendered 1.4.0 or earlier chart with any tool: those charts generate new keys and passwords in every render. Do not upgrade by applying the output of helm template with kubectl apply, at any version: the upgrade's jobs then run in no order.

If a 1.4.0 upgrade stops with missing cluster owner trust, nothing was changed: upgrade to 1.4.1 or later. To stay on 1.4.0, set both ownerPublicKey and clusterBindingId, which 1.4.0 requires even on an anchored namespace, and run the command again; see Upgrading to 1.4.0.

Upgrade with your values files, as above, and never with --reuse-values. With --reuse-values, Helm renders the new chart with the previous release's values, the previous chart's defaults included, in place of the new chart's defaults. A default that changed keeps its old value, a value the new chart adds has no default, and the new chart cannot tell those old defaults apart from values you set.

The upgrade restarts each Kamiwaza PostgreSQL pod once, to add the check-pgdata-ownership init container. From 1.4.1, core-postgres and keycloak-postgres also get the check-pgdata-directory init container and a new mount path for the same data; core-spicedb-postgres gets neither. Some of the upgrade's jobs, such as core-db-init-cleanup, are kept when they fail, for kubectl -n kamiwaza describe job <job>; the next upgrade replaces them. From 1.4.1, kamiwaza-namespace-prepare and the Keycloak password migration are kept until the next upgrade or sync, whether they succeeded or not, so their logs stay readable, and there is no core-upgrade-migration job any more: kamiwaza-namespace-prepare does its work. Most of the upgrade's other jobs are removed an hour after they finish, so read a failed one's log within the hour.

Upgrade through Argo CD

From 1.4.1 you can upgrade by pointing an Argo CD Application at the new chart version. This also moves an installation that Helm made to Argo CD. The Application, the identity that applies it and the limits are in Install with Argo CD.

Not supported in 1.4.1:

A target of 1.4.0 or earlier is never supported: do not sync one.
On an Istio mesh, an upgrade from a Helm-installed 1.3 through Argo CD is not supported in 1.4.1, because the mesh policies of core-api would keep their 1.3 selector labels beside the new ones and core-api would stay without endpoints: upgrade with Helm first, then move the installation to Argo CD.
On OpenShift with GreyMatter, an upgrade from 1.3 through Argo CD is not supported in 1.4.1, because Argo CD's render contains no Keycloak password migration there and local accounts, the administrators included, could not sign in afterwards: upgrade with Helm first, then move the installation to Argo CD.

The steps below were tested from a Helm-installed 1.3.4 on Kubernetes with the standalone Envoy edge and no Istio. To move an installation that Helm already upgraded to 1.4.1 or later, follow the same steps; that move was not tested separately, and not at all with Istio or on OpenShift with GreyMatter. From a Helm-installed 1.4.0 the steps are the same; that path was not run through Argo CD in the lab (the Helm upgrade from 1.4.0 was).

Do everything in Before you start first, as for an upgrade with Helm. Do not count on the upgrade to stop for a missed item: a check that reads the cluster while the chart renders, such as the PostgreSQL data directory halt, does not run under Argo CD. A check on the values alone still fails the render, and Argo CD then applies nothing. Then:

Have whoever administers Argo CD create the Application for the new version with your 1.4 values, with the existing installation's namespace as destination.namespace. Set releaseName to the existing release's name, from helm list -n kamiwaza. If the release is not named kamiwaza, replace kamiwaza- in the Job name kamiwaza-keycloak-fips-migration with <release>-.
Sync, with Prune enabled. Before anything else in the namespace changes, the sync runs three Jobs:
- kamiwaza-namespace-prepare finds the existing keys, passwords and certificates and keeps them exactly as they are.
- kamiwaza-keycloak-fips-migration moves Keycloak's passwords off argon2, as the Helm upgrade does.
- core-trust-bundle-publisher publishes the platform's trust bundle.
To read what the first one did:
```
kubectl -n kamiwaza logs job/kamiwaza-namespace-prepare
```
It lists each existing Secret as kept and each Secret the new release adds as created (from 1.3.4 the lab saw two: kamiwaza-audit-fingerprint and kamiwaza-service-grant-credentials), with its key names; generated keys also show a short fingerprint. It never logs a value.
If the sync stops in kamiwaza-keycloak-fips-migration, a local user still has an argon2 password. The Job's log names each one:
```
kubectl -n kamiwaza logs job/kamiwaza-keycloak-fips-migration -c keycloak-fips-migration
```
```
ERROR: passwords keycloak-fips cannot verify remain: kamiwaza/<username> (argon2). ...
```
No workload of the new release has been applied, and the platform keeps serving on 1.3. kamiwaza-namespace-prepare has created the Secrets that 1.4 adds, which 1.3 does not read. Keycloak itself has changed: the Job has already set the pbkdf2 password policy on the realms and moved the chart's own accounts, which 1.3 handles as before. Have each named user sign in once, or reset their password, as in option 1 of Upgrade halted: Keycloak passwords (argon2). Where that section says to run the upgrade again, sync again. Its option 2 is a value on the Application.
Expect Succeeded. The sync that gets past the three Jobs (the first one, or the one after step 3) applies the release and runs the post-sync Jobs in one go. In the lab it took about 4.5 minutes from 1.3.4. If a PostgreSQL pod stays in Init:Error or Init:CrashLoopBackOff instead, its check-pgdata-directory container has stopped it. Its log lists the directories it found and the values that decide (kubectl -n kamiwaza logs core-postgres-0 -c check-pgdata-directory). The sync does not end by itself while the pod waits: have whoever administers Argo CD terminate the sync operation, put those values in the Application, and sync again, with Prune enabled. global.postgres.allowDataDirectoryChange does not apply to this check. This case was not run in the lab on the upgrade path.
Verify. Argo CD shows the Application Synced and Healthy at the new targetRevision. Then do Verify, except its helm list line.

What stays behind from the Helm installation:

helm list still shows the old release, because Argo CD writes no Helm release record. Leave the record in place and do not run helm upgrade, helm rollback or helm uninstall on it afterwards: helm uninstall would delete the installation.
Some fields the old release set and the new chart no longer renders stay on the live objects, because a tool that applies a rendered chart cannot remove what it did not write: a second, unused mount on the two PostgreSQL pods, and a few unused environment variables and mounts on the Ray pods. They change nothing. The two that broke upgrades to 1.4.0, the core-api Service selector and the PostgreSQL data directory, are handled by the chart.
Finished Jobs of the 1.3 release (kamiwaza-spicedb-datastore-migrate-r<n>, kamiwaza-spicedb-schema-bootstrap-r<n>) and its kamiwaza-extension-baseline-<hash> ConfigMap may stay in the namespace. They are unused; delete them if you want.

If Argo CD already synced 1.4.0

An Argo CD sync of 1.4.0 over a Helm-installed 1.3 damaged the installation. It replaced the platform's generated keys and passwords, so stored model credentials stopped working and personal access tokens were refused. It left core-api without endpoints. It started both PostgreSQL servers on a new, empty directory inside the original one. It skipped the Keycloak password migration.

This procedure puts back the keys and the databases from before the 1.4.0 sync. It was run on Kubernetes with the standalone Envoy edge and no Istio. On an Istio mesh or on OpenShift with GreyMatter, do not follow it: contact Kamiwaza support. The procedure was run once, on an earlier release candidate. It was not repeated on the final 1.4.1 chart, which changed after that run, among other things in how the core-api autoscaler is applied and in the checks kamiwaza-namespace-prepare makes.

Decide first. Keeping the original databases has a cost: everything written since the 1.4.0 sync stays behind, unused, in the inner directory. That includes users, models and credentials created since then. If the site needs that data instead, stop here and ask Kamiwaza support. The chart offers no values to keep the inner directory of core-postgres, because the same value also places the SpiceDB datastore.

Check first, before step 1. Step 1 stops both databases, and step 3 cannot be done without these. As the installer identity:

python3 --version
helm list -n kamiwaza
helm history <release> -n kamiwaza

python3 must print a version. helm history must list at least one REVISION whose CHART is a 1.3 release (kamiwaza-1.3.x) and whose STATUS is deployed or superseded. Note the highest one; step 3 uses it. If either is missing, do not start: contact Kamiwaza support.

Then take the pg_dumpall backup from Before you start, while the databases still run. It holds what was written since the 1.4.0 sync, which this procedure leaves behind.

Sync by hand during this procedure: remove syncPolicy.automated from the Application until step 7 is done, so that every sync below is one you start. Start each sync with Prune enabled (the Prune box in the sync dialog, or argocd app sync kamiwaza --prune). Without it the Application stays OutOfSync on objects that 1.4.0 rendered and 1.4.1 does not. Prune does not touch the generated Secrets once step 1 has marked them. Whoever manages the Application in Argo CD changes it. Run the kubectl and helm commands as the installer identity. Do the steps in this order.

Point the Application at 1.4.1 or later and sync. If you added ignoreDifferences on Secrets and RespectIgnoreDifferences as a stopgap, remove them in the same change; they are no longer needed.

Expect this sync to end Failed on the hook kamiwaza-namespace-prepare, with exit code 3. The stop is on purpose and happens once:
```
kubectl -n kamiwaza logs job/kamiwaza-namespace-prepare | tail -3
```
```
kamiwaza-namespace-prepare: STOPPED ON PURPOSE, ONCE: <N> generated Secret(s) that Argo CD tracks (...) are no longer part of the chart's render. They are now marked Prune=false and are unchanged. ...
```
Nothing was applied or pruned. Argo CD applied the generated Secrets itself at 1.4.0, so it tracks them, and 1.4.1 no longer renders them. The Job has marked them Prune=false, so Argo CD never prunes them, and labeled them kamiwaza.ai/generated-secret=true.

In Argo CD each of these Secrets now shows OutOfSync and as requiring pruning, and stays that way, while the Application itself shows Synced; the sync result lists them as ignored (no prune). That is expected. Do NOT prune or delete them by hand, in Argo CD or with kubectl: they are the keys this procedure exists to save. After step 3, only the Secrets that are not in the Helm release record still show this (in the lab: kamiwaza-service-grant-credentials).

Sync a second time. This sync applies 1.4.1 and is not meant to finish. Both PostgreSQL pods stop in their check-pgdata-directory init container, which prints both directories and changes nothing on the volumes. Watch for it:
```
kubectl -n kamiwaza get pod core-postgres-0 keycloak-postgres-0
```
When both show Init:Error or Init:CrashLoopBackOff, ask whoever administers Argo CD to terminate the sync operation, and continue with step 2. The databases stay down until step 4. To read what the init container printed:
```
kubectl -n kamiwaza logs core-postgres-0 -c check-pgdata-directory
kubectl -n kamiwaza logs keycloak-postgres-0 -c check-pgdata-directory
```
If the second sync instead ends Failed on kamiwaza-keycloak-fips-migration, it applied nothing and the PostgreSQL pods keep running. Do not go on to step 2. Read the Job's log and contact Kamiwaza support with it:
```
kubectl -n kamiwaza logs job/kamiwaza-keycloak-fips-migration -c keycloak-fips-migration
```
The lab damaged its installation with a single sync of 1.4.0. An installation that 1.4.0 was synced to more than once was not run.

Do not skip this second sync. With its database down, Keycloak does not answer, and that is what lets the sync in step 4 pass kamiwaza-keycloak-fips-migration: the Job waits about a minute, logs did not answer, so stored passwords were not checked this time, and continues. Without the second sync, Keycloak would still answer in step 4 with the administrator passwords that step 3 replaces. That order was not run in the lab. If the sync in step 4 fails in kamiwaza-keycloak-fips-migration, contact Kamiwaza support.
Save the current generated Secrets. They are the only copy of the keys for anything written since the 1.4.0 sync, and step 3 overwrites them:
```
(
  set -euo pipefail
  umask 077
  OUT="kamiwaza-secrets-since-1.4.0-$(date +%Y%m%dT%H%M%S).yaml"
  kubectl -n kamiwaza get secret -l kamiwaza.ai/generated-secret=true -o yaml > "$OUT"
  echo "saved $(grep -c '^- apiVersion:' "$OUT") Secrets to $OUT"
)
```
The command prints no key. If it says saved 0 Secrets, stop: step 1 has not run. Keep the file outside the cluster and store it as you store your other Kamiwaza secrets. Kamiwaza support needs it if the site later needs data from the inner directory.
Restore the generated Secrets from the Helm release record. 1.4.1 keeps the keys the installation has now. It cannot bring back the ones the 1.4.0 sync replaced. The record of the release that Helm installed still holds them. Use the release name and the revision you noted before step 1:
```
helm list -n kamiwaza
helm history <release> -n kamiwaza
```
Use the highest REVISION whose CHART is a 1.3 release (kamiwaza-1.3.x) and whose STATUS is deployed or superseded. Set both in the first two lines, then run the block. It prints no key:
```
(
  set -euo pipefail
  umask 077
  RELEASE=<release>
  REV=<revision>
  WORK=$(mktemp -d)
  cleanup() {
    shred -u "$WORK/record" "$WORK/secrets.yaml" 2>/dev/null \
      || rm -f "$WORK/record" "$WORK/secrets.yaml"
    rmdir "$WORK"
  }
  trap cleanup EXIT
  kubectl -n kamiwaza get secret "sh.helm.release.v1.$RELEASE.v$REV" \
    -o jsonpath='{.data.release}' > "$WORK/record"
  python3 -c '
import base64, gzip, json, re, sys
record = gzip.decompress(base64.b64decode(base64.b64decode(sys.stdin.read())))
docs = re.split(r"(?m)^---[ \t]*\n", json.loads(record)["manifest"])
secrets = [d for d in docs if re.search(r"(?m)^kind: Secret[ \t]*$", d)]
if not secrets:
    sys.exit("no Secret in this revision: check RELEASE and REV")
sys.stdout.write("".join("---\n" + d for d in secrets))
print(len(secrets), "Secrets found", file=sys.stderr)
' < "$WORK/record" > "$WORK/secrets.yaml"
  kubectl -n kamiwaza apply -f "$WORK/secrets.yaml"
)
```
In the lab, a 1.3.4 release held 20 Secrets. The block stops at the first error and removes its working files either way. With a wrong revision it stops at kubectl get secret with NotFound and applies nothing. The marks from step 1 stay on the Secrets. If one lacked them, the next sync would put them back and stop once more with exit code 3 before it pruned anything.

The record holds the values as of that Helm revision. Anything rotated in place afterwards comes back as the old value and must be rotated again after step 7: the ForwardAuth signing secret, if you did Rotate the ForwardAuth signing secret, and any database password changed by hand.
Choose the original databases, and replace what cannot start.
1. Add these values to the Application:
```
core:
  postgresql:
    dataDirectoryGuard:
      nestedDirectory: keep-outer
auth:
  keycloak:
    postgresql:
      dataDirectoryGuard:
        nestedDirectory: keep-outer
```
2. Sync, with Prune enabled.
3. Wait until the sync has applied the values to both database StatefulSets. In the lab that took about two minutes, because the Keycloak migration Job waits a minute first:
```
kubectl -n kamiwaza get statefulset core-postgres keycloak-postgres -o yaml \
  | grep -c 'value: keep-outer'
```
  It prints 2 when both have the value. While it prints 0, wait a minute and run it again. Then list the pods that are not running and ready. The selector leaves out the pods of Jobs: a failed attempt of a Job stays in Error after the Job completes, and you must not delete it:
```
kubectl -n kamiwaza get pods -l '!job-name,!batch.kubernetes.io/job-name' --no-headers \
  | awk '{ split($2, r, "/"); if ($3 != "Running" || r[1] != r[2]) print }'
```
4. Delete each pod it lists that has not become Ready after two minutes. A StatefulSet does not, by itself, replace a pod that never became Ready:
```
kubectl -n kamiwaza delete pod <pod>
```
  Delete core-etcd pods one at a time, and only one that is not Ready. Wait until it is Ready before you delete the next.
  
  Repeat substeps 3 and 4 every minute until the list is empty, or holds only pods of apps and models you deployed.
5. Restart every Deployment in the namespace. This restarts deployed apps too:
```
kubectl -n kamiwaza rollout restart deployment
```
Every Deployment is now restarted and every StatefulSet or Deployment pod that was not Ready is replaced, so those pods read the restored Secrets. A StatefulSet pod that was Ready keeps the values it started with until it restarts. Restart the StatefulSets that read restored Secrets and have no restart trigger, if their pods were Ready and were not replaced in substep 4:
```
kubectl -n kamiwaza rollout restart statefulset core-spicedb
```
Under the hardened profile, or with core.sessionStore.mode: bundled, also:
```
kubectl -n kamiwaza rollout restart statefulset core-session-store
```
Restart core-etcd one pod at a time. For each pod, wait until it is Ready before you delete the next:
```
kubectl -n kamiwaza delete pod core-etcd-<n>
kubectl -n kamiwaza wait --for=condition=Ready pod/core-etcd-<n> --timeout=5m
```
If kubectl wait answers NotFound, the StatefulSet has not created the new pod yet: run it again.

In the lab these pods were not Ready and were replaced in substep 4, so this restart of Ready pods was not run there. The lab also did substeps 3 to 5 more bluntly than they are written here: it deleted core-postgres-0, keycloak-postgres-0, core-spicedb-0 and the one core-etcd pod that was not Ready as soon as the StatefulSets carried the new value, and restarted six Deployments (keycloak, core-api, core-scheduler, core-raycluster-head, core-raycluster-worker, core-user-jobs-head) while the sync was still running. The wait of two minutes, the repeated listing, the restart of every Deployment, the save in step 2 and the python3 lines of step 3 as printed were not run there; the lab read the same record with gunzip and awk.

This sync can end Failed on Deployment core-api exceeded its progress deadline, left from the time core-api could not start; it can also end Succeeded. Either way, judge by the platform, not by the sync result: wait until the pods are Ready, then check: a personal access token from before the 1.4.0 sync is accepted, and a model with a stored credential answers. Do not sync again yet.
Have Kamiwaza support reset the local Keycloak accounts. This step needs Kamiwaza support. The 1.4.0 sync skipped the password migration, so every local account, the administrators included, is still on argon2 and cannot sign in. A sync before the reset fails in kamiwaza-keycloak-fips-migration and changes nothing:
```
ERROR: neither the durable nor the bootstrap Keycloak administrator can sign in (last answer: HTTP 400). ...
```
Neither documented way out works in this state. The route that message prints needs a Keycloak image that is not in the public registry. Upgrade halted: Keycloak passwords (argon2) needs an administrator who can sign in. Contact Kamiwaza support, and continue when the accounts have been reset.
Remove the values from step 4 and sync, with Prune enabled. Expect this sync to end Succeeded, with the Application Synced. Healthy is expected too, but was not observed in this procedure: on the earlier candidate the lab used, the Application still showed Degraded on the core-api autoscaler when the sync ended, and 1.4.1 applies the autoscaler differently. Each PostgreSQL pod then logs, in its check-pgdata-directory container:
```
Starting on <volume>/postgres, the database that ran last. A second, older database is inside it and is left alone:
```
Verify. Users can sign in. A model registered before the 1.4.0 sync answers, and a personal access token from before it is accepted. Then do Verify, except its helm list line. Models registered and tokens issued between the 1.4.0 sync and step 3 are in the databases this procedure leaves behind: register those models and issue those tokens again. Put syncPolicy.automated back if you use it.

If the 1.4.0 damage was already repaired by hand, so that the original keys are back and PostgreSQL runs on the original directory again, steps 2 to 4 do not apply. Do the first sync of step 1 and, if it stopped, sync again. 1.4.1 starts PostgreSQL on the original directory and names the leftover inner one in the log of check-pgdata-directory. If a sync stops in kamiwaza-keycloak-fips-migration with the message of step 5, continue at step 5.

Upgrade halted: PostgreSQL data directory

Before it changes anything, the upgrade compares where each Kamiwaza PostgreSQL StatefulSet (core-postgres, keycloak-postgres, core-spicedb-postgres) keeps its data with where the new release would look for it. When they differ, PostgreSQL would start on an empty data directory, so the upgrade stops:

UPGRADE HALTED (HR-9): StatefulSet core-postgres keeps its PostgreSQL data in volume directory "/" (...), but this render would use "/postgres" (...). Proceeding would start PostgreSQL on an empty data directory. ...

The message names the values that keep the current layout, for example core.postgresql.dataMount.mountPath, .subPath, and .pgData. Add them to your values file and run the upgrade again.

To move the data on purpose, for example to let SpiceDB's PostgreSQL start empty and rebuild from Kamiwaza's database, list that StatefulSet for one upgrade, then remove the entry and upgrade again:

global:
  postgres:
    allowDataDirectoryChange: [core-spicedb-postgres]

Only the StatefulSets you list skip the check. Move any data you need yourself first, and take the pg_dumpall backup from Before you start before you list a StatefulSet whose data you cannot recreate. The check needs the cluster, so helm template and dry runs on the client do not run it, and neither does Argo CD. From 1.4.1 the check-pgdata-directory init container of core-postgres and keycloak-postgres makes a similar check inside the pod, under every tool. Its message starts with PostgreSQL will not start (<StatefulSet>), not with UPGRADE HALTED (HR-9). It names its own values, and allowDataDirectoryChange does not skip it.

Upgrade halted: Context TLS preflight

From 1.4.0, the Context Service verifies TLS on its embedding and LLM calls to the platform's own origins, on upgraded installations too. When the edge certificate is minted outside the chart or by a CA it does not hold (OpenShift with GreyMatter; Istio or Envoy with network.tls.mode: byo, including EZUA, or customerCA, or cert-manager with a custom issuer, or generated when the chart does not generate the certificate) and nothing publishes its CA, those calls would fail after the upgrade. The upgrade stops first, before it changes anything, with:

Context TLS preflight: Context runtime model calls verify TLS by default since 1.4.0 (core.context.embedding.verifySsl: true), but the certificate for https://<domain> is minted outside this chart (provider=<provider>, network.tls.mode=<mode>) and nothing publishes global.trust.edgeAnchor.pem<settings that do not count>, so Core cannot verify its own origin and embedding and Context LLM calls would fail after this upgrade. Fix one of: (1) set global.trust.edgeAnchor.pem to the CA chain that signs <domain><CA hint>; (2) if that certificate chains to a public CA, set global.trust.edgeAnchor.publiclyTrusted: true (verification stays on); (3) to keep the pre-1.4.0 behavior without verification, set core.context.embedding.verifySsl: false (named in NOTES and at every Core start).

A hand-made kamiwaza-edge-trust ConfigMap that Helm does not manage satisfies this check on a real upgrade. If you move its CA into global.trust.edgeAnchor.pem, as step 6 of Before you start describes, delete the ConfigMap right before you rerun, or the upgrade stops with the already exists error instead.

The check runs when the chart renders, so helm template also fails with it, including the --show-only command in GreyMatter step 3, until your values carry a fix.

<settings that do not count> is empty, or one or more of these, in parentheses and separated by semicolons:

global.trust.edgeAnchor.publishedBySubstrate is true, but no substrate-created ConfigMap kamiwaza-edge-trust is in this namespace
global.trust.edgeAnchor.pem is set, but network.enabled=false, so this release does not publish it
core.scheduler.extraEnv sets <name> from valueFrom, which this chart cannot read, so it counts as verifying; use a literal value

<CA hint> names the CA to publish:

on GreyMatter: (on GreyMatter: the ca.crt of Secret greymatter-edge-ingress);
with a custom cert-manager issuer: (cert-manager with issuer <issuer>: that issuer's CA);
with customerCA: (customerCA: the customer CA; one added through the ca chart's trustBundle.customerCASecret is not visible to this chart, so put the same CA here);
otherwise nothing.

Add one fix to your values file and run the upgrade again:

Recommended: set global.trust.edgeAnchor.pem to the CA chain that signs the domain: on GreyMatter, the ca.crt of Secret greymatter-edge-ingress; on EZUA, the EZUA gateway's CA; with customerCA, the customer CA; with a custom cert-manager issuer, that issuer's CA.
If the edge certificate chains to a public CA, set global.trust.edgeAnchor.publiclyTrusted: true. This never applies on GreyMatter.
Only until you can publish the CA, set core.context.embedding.verifySsl: false. This turns verification off, as before 1.4.0. The install notes and Core's startup log name it.

The fixes carry different risks. The full list of conditions, including the values that already satisfy the check, is in Edge certificates minted outside the chart.

Upgrade halted: Keycloak passwords (argon2)

From 1.4.0, Keycloak runs its FIPS build, which cannot verify argon2 password hashes, and every user created with a password on 1.3 has one (see Move local users' passwords off argon2). Before the upgrade's other jobs run, while the 1.3 Keycloak still serves, its kamiwaza-keycloak-fips-migration job sets hashAlgorithm(pbkdf2-sha512) and hashIterations(210000) in the password policy of the master and kamiwaza realms and of any other realm with an argon2 password, keeping the policy's other entries. It then signs in once as each account whose password the chart keeps in a Secret, which moves those accounts to pbkdf2. If any argon2 password is left, the upgrade stops with an error that names job kamiwaza-keycloak-fips-migration failed (the exact wording depends on your Helm version):

Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* job kamiwaza-keycloak-fips-migration failed: BackoffLimitExceeded

The Kamiwaza release you run keeps running, and none of its resources has changed. Only the realm hash policies and the chart's own accounts have moved to pbkdf2, which 1.3 handles as before and which the next attempt needs. The job's log, kept for an hour (from 1.4.1, until the next upgrade or sync), names each account as <realm>/<username>:

kubectl -n kamiwaza logs job/kamiwaza-keycloak-fips-migration -c keycloak-fips-migration

ERROR: passwords keycloak-fips cannot verify remain: kamiwaza/<username> (argon2), kamiwaza/<username> (argon2). keycloak-fips cannot verify these passwords, so these accounts could not sign in after the upgrade. ...

Fix it in one of two ways:

Recommended: move the listed accounts, then run the same helm upgrade again. The job has already set the pbkdf2 policy, so have each listed user sign in to Kamiwaza once, or reset their password under Local Users in Settings. Then run the same upgrade command. To check first, use the query in step 3 of Move local users' passwords off argon2. If the list names one of the chart's own accounts, the job could not sign it in: its password no longer matches the one in its Secret (keycloak-admin for the master administrator, kamiwaza-user-<username> in the kamiwaza realm), or its realm requires a one-time code. Sign it in once through the browser, or reset its password to the one in its Secret.
Upgrade anyway and lock the listed accounts out. Add this to your values file and run the upgrade again:
```
auth:
  keycloak:
    fips:
      allowArgon2Lockout: true
```
The upgrade then continues, and the job's log lists the accounts with a WARNING instead. After the upgrade, those users cannot sign in at all, with any password, until an administrator resets their password under Local Users in Settings. Users cannot reset their own unless you have set up Keycloak's email password reset, which needs a mail server the chart does not configure. When the realm requires MFA (always under the hardened profile), the chart turns email reset off, so only an administrator can reset these passwords. Plan the resets and tell the users before you choose this. It never covers the chart's own administrator in the master realm: if that account is listed, the upgrade still stops.

Do not use this while the list names one of the chart's accounts in the kamiwaza realm, such as admin or svc-core. The setting locks those out too, including the Kamiwaza administrator who resets the other passwords. Fix those accounts with option 1 first.

Keep allowArgon2Lockout: true in your values until every locked-out account has been reset. Keycloak then runs its FIPS build, so an upgrade without the setting finds the same accounts, can no longer move them, and stops again. Its error message then gives the longer recovery, which moves Keycloak back to its standard image for one upgrade.

Upgrade from the OpenShift overlay files

The kamiwaza-openshift.yaml and kamiwaza-openshift-greymatter.yaml values files are retired, and global.platform: osgm replaces them. With them in the values file list, the upgrade stops with INSTALLATION HALTED; see Install stops with a workload identity error.

Remove both files from your values file list, and set global.platform: osgm in your values file. Keep the chart's default direct extension runtime. The preset replaces the files' defaults; copy over only values your site chose for itself. One set the preset does not yet supply: kamiwaza-openshift.yaml paced GreyMatter route publishing, so copy these into your values file:
```
global:
  greymatter:
    routeWriter:
      postPushSettleSeconds: 30
      retainDeletedRoutes: true
      startupFlushTimeoutSeconds: 120
```
If your PostgreSQL data uses the layout those files set, set core.postgresql.dataMount and auth.keycloak.postgresql.dataMount to it. SpiceDB's PostgreSQL (core-spicedb-postgres) follows core.postgresql.dataMount too. The PostgreSQL data directory check names the values when they are missing.
dataMount sets only the path. Data written under the files' fixed user ID, 70 or 65532, stays owned by it, and PostgreSQL refuses a data directory it does not own. Do not change the owner before the upgrade: the running PostgreSQL pods still run as the fixed user ID, and the upgrade's database jobs use them before the pods are replaced. Change it during the upgrade instead:
1. Run the upgrade. As it replaces each PostgreSQL pod, the new pod stops at its check-pgdata-ownership init container before PostgreSQL starts, so the data is not touched. kubectl -n kamiwaza describe pod <pod> shows an RS-16 message with the volume's owner and the user ID the pod now runs as.
2. Have the cluster administrator change the owner of that volume's data to the user ID in the message. The init container retries on its own, or delete the pod to retry at once, and PostgreSQL starts.
Have the administrator ready before you start: the upgrade waits for the PostgreSQL pods only until its --timeout. If it times out first, change the owners, then run the same upgrade command again. See also Troubleshooting.

External model endpoints and `tlsInsecure`

From 1.4, core.scheduler.tlsInsecure: true turns off certificate verification only for the platform's own origins (with one OmniParse exception, see core.scheduler.tlsInsecure): the platform's domain, in-cluster services, and paired federation clusters. Core verifies the certificate of every external model host it calls, such as in Context Service and OmniParse model calls, as Kaizen already did on 1.3 except in its reranker, which now verifies too. The Context Service and OmniParse TLS settings, including OmniParse's force insecure model SSL and a request's vision model verify_ssl: false, no longer relax an external host either; only core.scheduler.externalTlsInsecureWithDnsRetargetingRisk does. A model server addressed by a private IP address or a short hostname, such as an on-premises server at https://10.0.0.5, counts as external.

If an external model endpoint has a certificate that Kamiwaza does not trust, and it worked on 1.3 because certificate verification was off, do one of these before you upgrade:

Recommended: add the endpoint's root CA, and any issuing CA, to global.trust.extraAnchors. After the upgrade, restart Core's pods so they load it (kubectl -n kamiwaza rollout restart deployment/core-api deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker, plus deployment/core-retrieval-streamer if you enabled it) and redeploy the extensions that call the endpoint, such as Kaizen. If you created the kamiwaza-edge-trust ConfigMap by hand, first move it into values as described in Before you start. See TLS Trust for Private Certificate Authorities.
For a trial only: list the host in core.scheduler.externalTlsInsecureWithDnsRetargetingRisk:
```
core:
  scheduler:
    externalTlsInsecureWithDnsRetargetingRisk:
      - models.example.com
```
List a fully qualified hostname or an IPv4 address. A short hostname or an IPv6 address fails the install; Core covers those only with true, and Kaizen never relaxes either (see the configuration reference). The setting never covers an OmniParse translation endpoint.

This sends the API keys Kamiwaza holds for that provider to whoever answers DNS for the host. See externalTlsInsecureWithDnsRetargetingRisk. Redeploy the extensions that call the endpoint, such as Kaizen, so they pick it up, and again when you remove it.

If you upgrade without either, Core's and Kaizen's calls to that endpoint fail (an OmniParse instance not yet re-saved behaves as described below). For its own model calls, Core logs one error for each external host that fails verification, naming the host and both fixes, and, with tlsInsecure: true, one warning that it now covers only platform origins. OmniParse and Kaizen report their verification failures in their own logs.

After the upgrade, open and save each OmniParse instance that uses any external endpoint, including a translation endpoint, so Core rebuilds its pod-wide TLS setting. Add any external translation host's CA first. Until you do, its translation calls and document downloads keep the 1.3 setting, which may leave verification off for an external host.

Extension platform gateway

Kamiwaza 1.4 gives extensions one in-cluster platform address, KAMIWAZA_PLATFORM_GATEWAY_URL, on every routing provider. See Platform Gateway for Extensions.

Core needs KAMIWAZA_ORIGIN. On Kubernetes, Core refuses to start unless KAMIWAZA_ORIGIN is an http(s) URL with a host, and core-api goes into CrashLoopBackOff with:

FATAL: KAMIWAZA_ORIGIN is unset; extensions would be told the platform lives at https://localhost. Set global.domain (or core KAMIWAZA_ORIGIN) to the public https origin.

The chart always sets it from global.domain, so only a hand-edited Core configuration is affected. Set global.domain, or Core's KAMIWAZA_ORIGIN, to the public HTTPS origin.

A KAMIWAZA_INTERNAL_GATEWAY_URL override must be an origin. If Core has one with a path, such as https://edge/api, extension deploys are refused and GET /api/kamiwaza/install-modes reports invalid, with:

KAMIWAZA_INTERNAL_GATEWAY_URL=<value> must be an origin (scheme://host[:port]); Core appends /api itself, so this value yields /api/api/... and extensions get 404.

Remove the path. Such values already made extension calls fail with 404.

GreyMatter on a custom cluster domain. The GreyMatter edge host now follows the cluster's DNS domain instead of a fixed .svc.cluster.local. Chart installs need no certificate change: the chart sets KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME on Core to global.greymatter.edge.tlsServerName, or to global.domain when that is unset, and extensions verify the edge against that name. Without the chart, when that variable is unset, Core uses <edge-service>.<namespace>.svc.<domain> as KAMIWAZA_GATEWAY_TLS_SERVER_NAME, so the edge certificate must carry that name, or set the variable to a name it does carry. Do not set it blank: Core then withholds KAMIWAZA_PLATFORM_GATEWAY_URL and says why on the deployment status. Clusters on cluster.local see no change.

Envoy on a custom cluster domain. Core now points extensions at https://kamiwaza-envoy.<namespace>.svc.<domain>, including their KAMIWAZA_API_URL. Before 1.4 that host was fixed to .svc.cluster.local, which does not resolve on such a cluster. Set global.clusterDomain to the cluster's domain. A certificate you provide must carry that name. The chart reissues its generated certificate with the name, and restarts Envoy, Core and Keycloak to load the new edge anchor. Redeploy running extensions afterward. A client outside the cluster that pinned the old self-signed certificate must fetch the new one. See TLS names for the platform gateway.

Istio on a custom cluster domain. Extensions verify the gateway against the public host, which the gateway certificate already carries. Set global.clusterDomain so that the default gateway host uses the cluster's domain.

Istio. Extensions now reach the platform in-cluster through the ingress gateway instead of through the public origin. Most chart installs need no values change. With the extension operator (not supported on public installs; see Extension operator not supported on public installs), an upgrade can stop on the install identity's rights in the gateway namespace, on a missing gateway namespace or DestinationRule CRD, or on a serviceHostname form it no longer accepts, and a namespace-admin install needs a Role from a cluster administrator; see Extension operator on Istio.

Extension pods restart once. After Core is upgraded, it applies each extension deployment again with the new variables, and each extension pod restarts once. Plan for it; no action is needed.

Remove hand-set gateway overrides. Remove any TASKS_PLATFORM_GATEWAY_URL or other gateway override you set by hand that points at the address Kamiwaza now sets.

Extension operator not supported on public installs

The extension-operator runtime is not supported on any install of the public Kamiwaza 1.4.0 release, including mirrored and air-gapped installs. No public release has published an extension-operator image, so a public install with extensionOperator.enabled: true never had a running operator. From 1.4.0 the chart refuses that value when global.imageRegistryPrefix is under ghcr.io/kamiwaza-ai/releases/, and the upgrade stops before it changes anything. On a mirrored or air-gapped install the chart cannot tell that the images are the public release's, so it does not stop you: check your values yourself.

Before you upgrade, remove extensionOperator.enabled: true from your values, and core.extensionRuntime.mode: operator if you set it. Also remove any global.namespaces.extensions or global.namespaces.sandboxes override that names a namespace other than Kamiwaza's, and any core.clusterScopedRbac.create: true or core.clusterScopedRbac.extensionManagerMode override, which the direct runtime refuses. Extensions then run through the direct runtime, the chart default. See Extension operator layout.

Workroom Manager extension removed

Kamiwaza 1.4.0 removes the standalone Workroom Manager extension (catalog name workroom-manager), and the extension value of global.workroomManager.surface with it. Workrooms themselves are not affected: workroom records, the workroom APIs, permissions and isolation stay as they are, and the data a workroom held through the extension stays in Kamiwaza.

global.workroomManager.surface accepts:

Value	Result
absent, empty or `disabled` (default)	Workrooms are hidden from users.
`core`	The built-in Workrooms page serves `/workrooms`.
`extension`	The chart refuses to render, so the install or upgrade stops, on every ingress provider.

While your values set extension, the chart refuses to render, before the upgrade changes anything:

Error: execution error at (kamiwaza/charts/...): global.workroomManager.surface=extension is retired: the standalone Workroom Manager extension is no longer supported. Remove the key to keep the default (disabled) or set it to core for the built-in Workrooms page. ...

If you used the extension:

Find the setting. In the values files you upgrade with, look for workroomManager: under global:. On the running release, helm get values kamiwaza -n kamiwaza | grep -A1 workroomManager shows it.
Remove surface: extension, or change it to core if users should keep a Workrooms page.
Upgrade, as described in Upgrade.
Delete the standalone extension. After the upgrade it is hidden and cannot be reached, but its pods keep running until you delete it. Find its name with kubectl get kamiwazaextensions -A, then, as an administrator, delete it through the platform API, which runs the normal extension teardown:
```
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "https://<your Kamiwaza domain>/api/extensions/<extension name>"
```
Do not delete the KamiwazaExtension resource with kubectl: it then waits on its finalizer and is not cleaned up.

A 1.4.0 Core that is still given extension, for example by an older chart, treats it as disabled and logs an error at startup. It does not switch to the built-in page, which you did not choose.

Connectors at internal addresses

From 1.4, Kamiwaza refuses connector endpoints that resolve to private, CGNAT or other non-public addresses unless you admit them. Loopback, link-local and cloud metadata addresses are refused even when admitted. See Connector Destinations.

Existing connectors keep working without a values change: on its first start with 1.4, Kamiwaza approves each existing connector's private host:port for that connector only, shown as Grandfathered on upgrade under Settings > Connectors. Connectors Kamiwaza deployed itself are admitted anyway. After the upgrade:

Check Settings > Connectors > Blocked connector destinations. Allow a destination for its connector there, or re-register a connector whose endpoint is a floor address or an invalid URL. See Existing connectors on upgrade for what the upgrade does not grandfather.
To register new connectors at internal hosts, list those hosts in core.connectors.endpointAllowlist, as narrowly as you can. If you set KAMIWAZA_CONNECTOR_ENDPOINT_ALLOWLIST in core.scheduler.extraEnv (or core.scheduler.linuxNvidia.extraEnv), move it to that value, one list item per comma-separated entry, and remove the extraEnv entry; setting both fails the render, and so does an item that contains a comma.

Extension runtime policy under the hardened profile

This applies only to installs with global.securityProfile: stig. From 1.4, an unset global.extensionRuntimeCsp.mode enforces the Content-Security-Policy on App Garden extension responses under the hardened profile; before, it was report-only. The upgrade changes this with no values change. See Extension runtime policy under the hardened profile.

Upgrade Kaizen first, to the Kaizen release in the 1.4 App Garden catalog, or later, which supports an enforced extension policy. Earlier Kaizen releases cannot show artifact previews under it. Check your other extensions before you upgrade: an extension's report-only policy will be enforced as written, and the platform's floor policy blocks cross-origin framing and cross-origin form targets even for an extension that sends its own policy. See Extension runtime policy under the hardened profile for what to check. On Istio, extension pods in ambient mode get no platform policy; record them as not meeting the enforced policy requirement.

If your values set global.extensionRuntimeCsp.mode: report-only, the upgrade stops with an error that names the setting and says the hardened profile requires an enforced extension policy.

Remove the setting, so the policy enforces, or keep report-only for now by also setting global.extensionRuntimeCsp.acknowledgeReportOnly: true. The install output then warns that the enforced policy requirement is not met.

Exempt paths on the standalone Envoy edge

This applies only to installs with global.ingress.provider: envoy. From 1.4, the Envoy edge builds a route for each /api/... entry of global.authz.extAuthz.exemptPaths, and the render fails, before the upgrade changes anything, on an entry the edge cannot serve. The shipped list passes; check entries you added. Separately, on every install type, if you use SAML or certificate login, add its paths to the list before you turn it on, keeping every shipped entry (see the next link). See Public paths at the edge.

Verify

Every Kamiwaza deployment's pods are Ready, and the upgrade's jobs have completed. Helm can return before the pods are ready; wait with the command at the start of your install page's verify step.
helm list -n kamiwaza shows the new chart version. This holds for an upgrade with Helm only. After an upgrade through Argo CD, helm list still shows the old release, or nothing for an installation Argo CD made: check in Argo CD that the Application is Synced and Healthy at the new targetRevision instead.
The platform answers 200 at / and 401 at /api/ without credentials.
GET /api/kamiwaza/metadata, called as a signed-in user, reports the new version.
A local CPU model you deploy after the upgrade runs, and a GPU model too when the install has a signed profile, as described in Tenant-mode inference, step 7. Under the built-in CPU baseline, a GPU model is refused with HTTP 422.

After the upgrade

Do each action that the release notes list for the releases you crossed.

Deploy your models again. A model deployment created before tenant-mode inference was on carries no tenant resources, so it cannot start again once its pods are replaced on the new release. Deploy each one again when the install's trust level admits it: a CPU model under the built-in CPU baseline or a signed profile, and a GPU model only under a signed profile. Until then, leave the old deployment running, because stopping it does not bring it back. To deploy one again, stop it and deploy it again; the new deployment takes its resources from the profile. The platform's embedding model may need only the stop: see step 6 of Tenant-mode inference.

Running extensions do not change. Each running extension deployment keeps the version it was deployed with. A selection changes only the version that new deployments use.

Clean up an air-gapped registry only as Clean up the registry safely describes. A cleanup after the upgrade can break images that still have their tags, including the new release's.

From 1.3.4, the App Garden moves older selections up to the release's versions. Each release ships a version of each extension it packages. When Kamiwaza 1.3.4 or later starts after the upgrade, every App Garden selection older than the version the release ships moves up to that version, including a version an administrator chose. A selection at or above that version does not change. A selection of an extension the release does not package, such as one published after the release, does not change either. An upgrade to 1.3.3 or earlier moves nothing.

To use a version newer than the release ships, as an administrator, open the App Garden and select Check for updates, then Update on a row, Update all, or Choose version.

If Kamiwaza cannot move a selection, it keeps the older one and records why. To check, use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token:

curl --fail --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/apps/catalog" \
  | jq '{moved: [.baseline_advance.advanced[]? | "\(.name) \(.from_version) -> \(.to_version)"],
         not_moved: [.baseline_advance.blocked[]? | {name, message}]}'

moved lists the selections Kamiwaza moved. not_moved should be empty. For each entry in it, follow its message, for example by selecting the release's version with Choose version.

A new Kaizen version is a new deployment. Each Kaizen deployment has its own database and storage volumes, and no data carries over from an existing deployment to a new one. Keep your current Kaizen deployment unless you are starting fresh. If you run Kaizen v3, see Kaizen v3 on Kamiwaza 1.4 as well.

What stopping a Kaizen deployment does to its storage volume claims depends on how you stop it. On an installation with the chart's default extension runtime:

Stop in the App Garden's list of deployments deletes the deployment's storage volume claims and, with a Delete reclaim policy, the Kubernetes default for dynamically provisioned volumes, its data. For Kaizen v3 this is permanent: see Stop a Kaizen v3 deployment without losing it.
Stopping it through the API, with DELETE /api/apps/deployment/<id>, keeps its storage volume claims and their data, and they stay bound to the same volumes through an upgrade. DELETE /api/apps/deployment/<id>/purge deletes them.

Back up before you stop a Kaizen deployment either way.

To run a new Kaizen version alongside the existing one:

Back up the existing deployment, as described in Before you start.
Make sure the cluster has room for a second complete Kaizen deployment. Without it, the new deployment's pods stay Pending and the deployment fails. Never stop the existing deployment to make room. If the new deployment fails before anyone has used it, or opens the Kamiwaza dashboard instead of Kaizen, add capacity if it was short, then stop that new, unused deployment and deploy again. Do not stop a new deployment that anyone has added data to; contact Kamiwaza support instead.
If the App Garden does not already select the new Kaizen version, select it, as described above.
Deploy Kaizen from the App Garden.
In the new deployment's Admin area, under Models, select the chat and embedding models. Upload your documents again, then choose Index on each one, or Index selected for several.

Kubernetes API addresses changed by the upgrade

Each extension deployment reaches the Kubernetes API through a NetworkPolicy whose name ends in -kube-api. If the upgrade changed core.extensionRuntime.kubernetesApi.egressCidrs, you don't need to update these policies. From 1.4.0, Kamiwaza rewrites each deployed extension's policy to the configured addresses when it starts after the upgrade. Without configured addresses, it uses the addresses of the cluster's API server, and leaves a policy unchanged if it cannot read them.

To check, list the policies and the addresses each one allows. Every policy should list the addresses you configured:

kubectl get networkpolicy -A -l extensions.kamiwaza.io/managed-by=kamiwaza-platform -o json \
  | jq -r '.items[] | select(.metadata.name | endswith("-kube-api"))
    | "\(.metadata.namespace) \(.metadata.name) \([.spec.egress[0].to[].ipBlock.cidr] | join(","))"'

With a namespace-scoped identity, replace -A with -n <extension namespace>. If a policy still lists the old addresses, contact Kamiwaza support.

Kaizen v3 on Kamiwaza 1.4

On Kamiwaza 1.3 the App Garden lists two Kaizen products. Kaizen (Legacy), catalog name kaizen-legacy, is Kaizen v3. Kaizen, catalog name kaizen, is Kaizen v4. Deployments made before 1.3 may run Kaizen v3 under another name, such as kaizen or kaizen-v3. Kamiwaza recognizes Kaizen v3 by its images as well as by the name kaizen-legacy, so this section applies to those deployments too.

Kamiwaza 1.4 stops installing Kaizen v3 and keeps the Kaizen v3 deployments you already have.

New Kaizen v3 deployments are refused

From 1.4, Kamiwaza refuses every request that would create a new Kaizen v3 deployment, before it creates anything. That covers the App Garden, the API, the SDK and the CLI, and every catalog, including imported, retained and local ones. Cloning a Kaizen v3 deployment, or creating another one beside it, is refused too: a Kaizen v3 deployment you already have does not allow a new one, in its workroom or anywhere else.

The API answers 409, with code set to product_retired and this message in detail:

Kaizen v3 (kaizen-legacy) is retired on Kamiwaza 1.4 and later: new installations are refused. Existing Kaizen v3 deployments are preserved and stay manageable. Install Kaizen v4 ('kaizen') as a separate deployment; v3 data and configuration are not migrated to it.

Deploying Kaizen (Legacy) is refused

After the upgrade, the App Garden can still show a Kaizen (Legacy) card with a Deploy button. Deploying it is refused with this message:

Kaizen v3 (kaizen-legacy) is retired on Kamiwaza 1.4 and later: new installations are refused. Existing Kaizen v3 deployments are preserved and stay manageable. Install Kaizen v4 ('kaizen') as a separate deployment; v3 data and configuration are not migrated to it.

This is expected, and nothing needs repairing. The refusal comes before anything is created, so it leaves no deployment, storage or route behind. Kaizen v3 deployments you already have are not affected.

To install Kaizen, deploy Kaizen v4, the Kaizen card (catalog name kaizen), as a separate deployment, as described in After the upgrade.

Existing Kaizen v3 deployments are kept

A Kaizen v3 deployment that exists when you upgrade, whether it is running, stopped or failed, keeps its ID, data, configuration, storage, route, owner and workroom. A running deployment keeps running through the upgrade and after it.

Restore a stopped or failed deployment with POST /api/apps/deployment/<id>/restore. The App Garden has no button for it.

The deployment's owner or an administrator can restore it. An owner who is not an administrator gets 404 when the request is made from a different workroom than the deployment's.
A restore brings back the same deployment, with the same ID, storage volume claims, stored secrets, route, owner and workroom. It is never a new installation.
Only a stopped or failed deployment can be restored. A request for a running one is refused with 409 and code not_restorable.
Kamiwaza answers 202 once the launch is scheduled. Follow its progress with GET /api/apps/deployment/<id>.
A refused restore answers 409 and changes nothing; its code says why. A 503 means the previous instance is still being cleaned up: retry after the number of seconds in its Retry-After header. If the code is reconciliation_required or launch_compose_mismatch, contact Kamiwaza support.
A 500 means the restore failed after it started. Check the deployment's status: it may still launch. If it is FAILED, restore it again.

A restore reattaches the storage volume claims the deployment still has. It cannot bring back data that was deleted. To stop a Kaizen v3 deployment and keep its data, stop it through the API, as described in Stop a Kaizen v3 deployment without losing it.

Kaizen v4 is a separate deployment. Deploy Kaizen v4, the Kaizen row (kaizen) in the App Garden, beside your Kaizen v3 deployment, as described in After the upgrade. It has its own identity, route and storage. It does not inherit Kaizen v3 agents, conversations, data or configuration: there is no in-place upgrade from Kaizen v3 to Kaizen v4, and no importer. Keep your Kaizen v3 deployment for as long as you need its history.

App Garden updates never replace Kaizen v3. Check for updates, Update, Update all and Choose version never delete, migrate or replace a Kaizen v3 deployment.

Deleting a Kaizen v3 deployment is final. DELETE /api/apps/deployment/<id>/purge deletes the deployment and its storage volume claims, and, with a Delete reclaim policy, their data. A purged deployment cannot be restored, and you cannot install a new Kaizen v3 deployment in its place. Stop in the App Garden's list of deployments deletes a Kaizen v3 deployment in the same way: see Stop a Kaizen v3 deployment without losing it. Back up first, as described in Before you start.

Stop a Kaizen v3 deployment without losing it

Never use Stop in the App Garden's list for Kaizen v3

Stop in the App Garden's list of deployments calls DELETE /api/extensions/<name>. That deletes the Kaizen v3 deployment and its storage volume claims and, with a Delete reclaim policy, the Kubernetes default for dynamically provisioned volumes, their data. The deployment cannot be restored afterwards: a restore request answers 404. Kamiwaza 1.4 cannot install Kaizen v3 again, so the deployment is gone for good.

Stop a Kaizen v3 deployment through the API instead. That keeps the deployment and its storage volume claims, so you can restore it in place later. These commands use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token.

Back up the deployment, as described in Back up and restore a Kaizen v3 deployment.
Find the deployment's ID, its id in GET /api/apps/deployments.

Stop the deployment:

ID=<deployment ID>
curl --fail --silent --show-error --request DELETE \
  --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/apps/deployment/$ID"

The deployment's storage volume claims and their data stay, bound to the same volumes.

To bring the deployment back, restore it in place:
```
curl --fail --silent --show-error --request POST \
  --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/apps/deployment/$ID/restore"
```
Kamiwaza answers 202 once the launch is scheduled. Follow its progress with GET /api/apps/deployment/<id> until it reports DEPLOYED. The rules for a restore, and what each refusal means, are in Existing Kaizen v3 deployments are kept.

Do not add /purge to the stop request: DELETE /api/apps/deployment/<id>/purge deletes the deployment and its storage volume claims, and cannot be undone.

Back up and restore a Kaizen v3 deployment

A Kaizen v3 deployment keeps its data in two places: its own PostgreSQL database, openhands_ui, and its backend storage volume, mounted at /tmp/kaizen/storage. A backup is a copy of both.

You can restore a backup only into the same deployment it was taken from, with the same deployment ID. Its stored secrets are encrypted with a key that belongs to that deployment, so another deployment cannot read them. Kamiwaza does not install new Kaizen v3 deployments, so there is no other deployment to restore into.
You can restore a backup taken on 1.3 after you upgrade to 1.4. The Kaizen v3 images do not change in the upgrade.
Files in a conversation's agent workspace are not part of the backup.

These commands are for an installation with the chart's default extension runtime. On another extension runtime, contact Kamiwaza support. If the deployment sets POSTGRES_USER or POSTGRES_DB to something other than the defaults, postgres and openhands_ui, use its values.

Find the deployment's ID, its id in GET /api/apps/deployments. Then find its namespace and its postgres and backend pods:

ID=<deployment ID>
SEL="kamiwaza.io/deployment-id=$ID"
NS=$(kubectl get pods -A -l "$SEL" -o jsonpath='{.items[0].metadata.namespace}')
PG=$(kubectl -n "$NS" get pods -l "$SEL,extensions.kamiwaza.io/service=postgres" \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
BE=$(kubectl -n "$NS" get pods -l "$SEL,extensions.kamiwaza.io/service=backend" \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
echo "$NS $PG $BE"

All three must be set. If kubectl prints an error instead, fix your access first; with a namespace-scoped identity, use kubectl -n <extension namespace> get pods -l "$SEL" and set NS to that namespace. If there is no error and a value is empty, the deployment is not running: see Restore.

Then check that the backend keeps its storage at /tmp/kaizen/storage. This command prints the backend's mount paths, and they must include it:

kubectl -n "$NS" get pod "$BE" \
  -o jsonpath='{.spec.containers[?(@.name=="backend")].volumeMounts[*].mountPath}'

If it does not, the deployment runs an older Kaizen v3 layout that these commands do not cover: contact Kamiwaza support.

Back up

Ask users to finish their conversations, then run:

kubectl -n "$NS" exec "$PG" -c postgres -- \
  pg_dump -U postgres -Fc openhands_ui > "kaizen-$ID-db.dump"
kubectl -n "$NS" exec "$BE" -c backend -- \
  tar -C /tmp/kaizen/storage -czf - . > "kaizen-$ID-storage.tgz"

Do not add -t to these commands: it corrupts the files. Keep both files together, labelled with the deployment ID and the time you took them, and store them where you keep your other Kamiwaza backups.

Restore

A restore replaces everything the deployment stored after the backup.

Check that the deployment ID is the one in the backup's file names.
If the deployment is failed, or was stopped through the API, restore it in place first, as described in Existing Kaizen v3 deployments are kept, and wait until GET /api/apps/deployment/<id> reports DEPLOYED. Then set NS, PG and BE again.

Check that both backup files are complete before you delete anything. The database check restores the dump into a scratch database, so the database volume needs room for a second copy of the data. Every command must finish without an error; if one fails, stop here, because the next step deletes the deployment's current data:

tar -tzf "kaizen-$ID-storage.tgz" > /dev/null
kubectl -n "$NS" exec "$PG" -c postgres -- \
  psql -U postgres -d postgres -c 'DROP DATABASE IF EXISTS openhands_ui_check'
kubectl -n "$NS" exec "$PG" -c postgres -- \
  psql -U postgres -d postgres -c 'CREATE DATABASE openhands_ui_check'
kubectl -n "$NS" exec -i "$PG" -c postgres -- \
  pg_restore -U postgres -d openhands_ui_check --exit-on-error < "kaizen-$ID-db.dump"
kubectl -n "$NS" exec "$PG" -c postgres -- \
  psql -U postgres -d postgres -c 'DROP DATABASE openhands_ui_check'

If the restore fails, drop openhands_ui_check with the last command before you stop.

Make sure nobody is using the deployment, then load the backup. The commands run in a subshell that stops at the first command that fails:

(
  set -e
  kubectl -n "$NS" exec "$PG" -c postgres -- \
    psql -U postgres -d postgres -c 'DROP DATABASE IF EXISTS openhands_ui WITH (FORCE)'
  kubectl -n "$NS" exec "$PG" -c postgres -- \
    psql -U postgres -d postgres -c 'CREATE DATABASE openhands_ui'
  kubectl -n "$NS" exec -i "$PG" -c postgres -- \
    pg_restore -U postgres -d openhands_ui --exit-on-error < "kaizen-$ID-db.dump"
  kubectl -n "$NS" exec "$BE" -c backend -- \
    sh -c 'rm -rf /tmp/kaizen/storage/* /tmp/kaizen/storage/.[!.]* /tmp/kaizen/storage/..?*'
  kubectl -n "$NS" exec -i "$BE" -c backend -- \
    tar -C /tmp/kaizen/storage -xzf - < "kaizen-$ID-storage.tgz"
  kubectl -n "$NS" delete pod "$BE"
)

If a command fails, nothing after it runs. Running this step again starts the load over. If the backend pod was already deleted, wait until the new one is Ready and set PG and BE again first. If it fails a second time, contact Kamiwaza support.

When the new backend pod is Ready, open Kaizen. Check that your conversations and uploaded files are back, and that a new conversation starts.

Keep the Kaizen v3 images

A Kaizen v3 deployment keeps pulling images after the upgrade. Kubernetes pulls them again when it re-creates the deployment's pods, for example after a restart or a move to another node, and each new conversation starts an agent sandbox from its own image. A 1.4 release does not include Kaizen v3 images.

An installation that pulls App Garden images from Kamiwaza's public release registry, and upgrades with the published chart and its existing values, needs nothing more for Kaizen v3 deployments made on Kamiwaza 1.3: the images of the Kaizen v3 versions that 1.3 shipped stay published there. On an air-gapped installation, the images must stay in your registry. Before you upgrade one that runs Kaizen v3:

Keep every image your registry received from a Kamiwaza release before 1.4 for as long as you run Kaizen v3. Kaizen v3 images have had several names over its releases, and a stopped or failed deployment runs no pods that would show which ones it needs, yet a restore pulls them again. Keep each image at the same path, tag and digest, and do not remove the previous releases' copies after the upgrade. A tag does not protect an image from every kind of registry cleanup: before you garbage-collect or prune the registry, read Clean up the registry safely.

Check the running deployments. These commands list the images that running Kaizen v3 deployments use, and the agent sandbox image that each new conversation starts. Confirm that each one is in your registry:

# Every Kaizen and PostgreSQL image a running pod uses
kubectl get pods -A -o json | jq -r '
  .items[] | .spec.containers[], (.spec.initContainers // [])[] | .image' |
  grep -iE 'kaizen|postgres' | sort -u

# The agent sandbox image each new conversation starts
kubectl get deploy -A -o json | jq -r '
  .items[] | .spec.template.spec.containers[] | .env // [] | .[] |
  select(.name == "AGENT_SERVER_IMAGE") | .value' | sort -u

The list can include Kaizen v4 and platform images as well.

If you are moving to a new registry, copy every image from step 1 there first, at the same path, with every platform and the same digest, as in step 3 of Install Without Internet Access.

After the upgrade, open a new conversation in each running Kaizen v3 deployment. It pulls the agent sandbox image, so a missing image shows up there. This check does not cover a stopped or failed deployment: its images are pulled only when you restore it.

Clean up the registry safely

This section applies to every air-gapped installation that cleans up its registry, whether or not it runs Kaizen v3.

Registry cleanup can break an image that still has its tag. Every Kamiwaza image is a multi-platform image: an index that names one manifest for each platform. Only the index has a tag. A cleanup that removes the platform manifests leaves the tag and the index in place, so the registry looks intact. Pods that are already running keep running from the copy on their node, so nothing fails at first. The failure comes later, when a pod starts on a node that has no copy: after a reschedule, on a new or rebuilt node, and in every new Kaizen v3 conversation, which pulls the agent sandbox image. This applies to the running release's images as much as to Kaizen v3's.

The commands below use crane and jq, and the SOURCE and REGISTRY values from Install Without Internet Access. SOURCE is where Kamiwaza publishes the release. REGISTRY is your registry's bare host:port, with no scheme or path. Set both in the shell you run the commands from, replacing registry.example.com:5000 with your registry:

SOURCE=ghcr.io/kamiwaza-ai/releases/kamiwaza
REGISTRY=registry.example.com:5000

If your registry requires credentials, log in with crane auth login first. Deleting from a registry:2 registry also needs its delete support turned on (storage.delete.enabled: true). Without it, every delete answers 405.

Garbage collection

Do not run registry garbage-collect --delete-untagged on a registry built on distribution 2.x, such as the registry:2 image. Platform manifests have no tags of their own, so it deletes the platform manifests of every multi-platform image in the registry. Run registry garbage-collect without --delete-untagged, or use a registry whose garbage collection follows image indexes, such as Harbor or distribution 3.x.
Without the flag, garbage collection frees only what you delete first. It deletes content, such as layers, that no manifest in the registry still names. It never deletes a manifest itself. Deleting only an image's tag or index frees almost nothing, because its platform manifests stay and keep their layers. To free an image's space, delete its index and its platform manifests, as in Delete an earlier release's image.
Run garbage collection with the registry's own image and configuration file. For the registry:2 image (distribution 2.x), the command is registry garbage-collect /etc/docker/registry/config.yml. For the registry:3 image (distribution 3.x), it is registry garbage-collect /etc/distribution/config.yml. If you mounted your own configuration file, give its path instead. Run it inside the registry's container while the registry is in read-only mode. With the registry stopped, run it in a one-off container of the same image, with the same storage and configuration file mounted. For a registry:2 registry that uses the image's own configuration file and keeps its storage in <storage>: docker run --rm -v <storage>:/var/lib/registry registry:2 garbage-collect /etc/docker/registry/config.yml.
Stop pushes during garbage collection, and restart the registry after it. Run it with the registry stopped, or in read-only mode (storage.maintenance.readonly.enabled: true in its configuration file, which takes a restart). Then restart the registry, out of read-only mode, before you copy anything into it. The registry caches which content it holds, and until it restarts it can accept a copy of deleted content without storing it. The copy reports success, and the image then fails to pull. A restart clears that cache only when it is kept in memory (storage.cache.blobdescriptor: inmemory, as in the registry:2 image's own configuration). If your registry keeps it in Redis (storage.cache.blobdescriptor: redis), a restart does not clear it: clear the registry's Redis database as well before you copy anything in.

Decide what to keep

Write the images that must stay to keep-refs.txt, one reference in your registry per line. Keep at least:

The new release's images and the images of the release you upgraded from, each from its own images.txt (step 1 of Install Without Internet Access). The release you upgraded from stays until every model and extension deployment runs on the new release and you no longer need to restore the backup, as in Before you start.
If the installation runs Kaizen v3: the images of every release before 1.4 that you copied into the registry, as Keep the Kaizen v3 images requires. Stopped and failed Kaizen v3 deployments run no pods, but a restore pulls their images.

Use each release's full images.txt, not a list you filtered in step 1, and pick the images you delete from the same lists. The step 1 filter for a cluster without GPUs replaces images.txt with the filtered list, so if you ran it, fetch that release's list again without the filter. If you no longer have the image list of a release you must keep, delete nothing from the registry.

Start with an empty list, then add each image list, rewritten to your registry:

: > keep-refs.txt

sed "s#^$SOURCE/#$REGISTRY/kamiwaza-ai/releases/kamiwaza/#" images.txt >> keep-refs.txt

Then add every image from your registry that a pod runs, and the agent sandbox image that Kaizen v3 deployments start. A platform administrator runs this, because it lists pods and deployments in every namespace. If REGISTRY is not set, or kubectl or jq fails, it removes keep-refs.txt, and nothing can be deleted:

if [ -n "$REGISTRY" ] &&
   kubectl get pods -A -o json > pods.json && kubectl get deploy -A -o json > deploys.json &&
   jq -r '.items[] | .spec.containers[], (.spec.initContainers // [])[] | .image' pods.json > live-refs.txt &&
   jq -r '.items[] | .spec.template.spec.containers[] | .env // [] | .[] |
     select(.name == "AGENT_SERVER_IMAGE") | .value' deploys.json >> live-refs.txt; then
  grep "^$REGISTRY/" live-refs.txt | sort -u >> keep-refs.txt
else
  rm -f keep-refs.txt
  echo "Cannot list the images in use: keep-refs.txt removed, delete nothing"
fi
rm -f pods.json deploys.json live-refs.txt

Three kinds of image are not listed, and you add them to keep-refs.txt by hand: the images of a workload with no running pod, such as one scaled to zero; any image you copied into the registry yourself; and an agent sandbox image that a deployment sets through valueFrom or envFrom, because the command reads only a literal AGENT_SERVER_IMAGE value.

Then record every digest those images use, indexes and platform manifests both. The command first removes duplicate lines from keep-refs.txt. It writes keep-digests.txt, and keep-digests.refs, a copy of the list it was built from, only if it can read every image:

rm -f keep-digests.txt keep-digests.refs
ok=1
[ -s keep-refs.txt ] || { echo "keep-refs.txt is missing or empty"; ok=0; }
[ "$ok" = 1 ] && sort -u -o keep-refs.txt keep-refs.txt
: > keep-digests.new
while read -r ref || [ -n "$ref" ]; do
  [ -z "$ref" ] && continue
  if index=$(crane manifest "$ref") && digest=$(crane digest "$ref"); then
    { echo "$digest"; printf '%s\n' "$index" | jq -r '.manifests[]?.digest'; } >> keep-digests.new
  else
    echo "Cannot read $ref"; ok=0
  fi
done < keep-refs.txt
if [ "$ok" = 1 ]; then
  sort -u keep-digests.new > keep-digests.txt
  cp keep-refs.txt keep-digests.refs
  echo "keep-digests.txt: $(wc -l < keep-digests.txt) digests"
fi
rm -f keep-digests.new

crane prints the error above each Cannot read line. A not-found error, such as MANIFEST_UNKNOWN, means that the reference is not in your registry. Any other error, such as an authentication or connection failure, says nothing about the image: fix it and run the command again. Two not-found cases need no repair:

An image you never copied, such as a GPU inference engine that the step 1 filter left out.
An image that a pod names but your registry never held, such as the image of a pod that has failed to pull it (ErrImagePull or ImagePullBackOff) ever since it was created, before any cleanup.

Remove each such line from keep-refs.txt, replacing REFERENCE with the reference after Cannot read, then run the command above again:

ref=REFERENCE
grep -vxF "$ref" keep-refs.txt > keep-refs.new
mv keep-refs.new keep-refs.txt

Any other not-found Cannot read line is damage, for example an image that you did copy from a release's image list: run the check and repair the registry before you delete anything.

Delete an earlier release's image

Deleting a tag deletes the image under every tag. The registry deletes a manifest by its digest, so a tool that deletes a tag deletes the digest the tag names. Every other tag in that repository that names the same digest goes with it. When an image is unchanged between two releases, deleting the old release's tag, for example envoy:release-1.3.4, also deletes the new release's tag, envoy:release-1.3.5.

Delete earlier releases' images only with this command. It refuses an image that an image you keep uses, and deletes the index and every platform manifest that no kept image uses. It also refuses to delete anything when keep-refs.txt is missing or differs from the copy the last build kept in keep-digests.refs: build keep-digests.txt again after every change to the list. Set old to the image, in your registry, by tag, replacing NAME:TAG. For an image listed by digest only, as in lists before 1.3.1, use NAME@sha256:… instead:

old=$REGISTRY/kamiwaza-ai/releases/kamiwaza/images/NAME:TAG
repo=${old%@*}
case ${repo##*/} in *:*) repo=${repo%:*} ;; esac
if [ ! -s keep-digests.txt ]; then
  echo "Build keep-digests.txt first: nothing deleted"
elif ! cmp -s keep-refs.txt keep-digests.refs; then
  echo "keep-refs.txt changed since keep-digests.txt was built: build it again, nothing deleted"
elif ! index=$(crane manifest "$old") || ! digest=$(crane digest "$old"); then
  echo "Cannot read $old: nothing deleted"
elif grep -qxF "$digest" keep-digests.txt; then
  echo "Keep $old: an image you keep uses $digest"
elif ! crane delete "$repo@$digest"; then
  echo "Could not delete $old: nothing deleted"
else
  failed=""
  for d in $(printf '%s\n' "$index" | jq -r '.manifests[]?.digest'); do
    grep -qxF "$d" keep-digests.txt || crane delete "$repo@$d" || failed="$failed $repo@$d"
  done
  if [ -z "$failed" ]; then
    echo "Deleted $old ($digest)"
  else
    echo "Deleted $old ($digest), but not:$failed"
  fi
fi

Each run prints exactly one result, and only a Deleted result deleted anything. Deleted …, but not: lists platform manifests that are left over: delete each one with crane delete <reference>. The command protects only the images in your keep list: a platform manifest shared with an image you left out of it is deleted, so build the list completely. When you have deleted what you need to, run garbage collection without --delete-untagged, restart the registry, and run the check.

Check the registry after every cleanup

Check that each image in keep-refs.txt still resolves, with each of its linux platform manifests, and that each tag still names the digest it should. The check reads manifests and does not download layers. An image copied in after garbage collection and before the registry restarted may show only as a bare MISSING line: restart the registry and copy it again. Without keep-refs.txt, the check reports that it checked nothing and does not print missing: 0.

missing=0
[ -s keep-refs.txt ] || { echo "keep-refs.txt is missing or empty: nothing checked"; missing=1; }
while read -r ref || [ -n "$ref" ]; do
  [ -z "$ref" ] && continue
  name=${ref%@*}
  repo=$name
  case ${repo##*/} in *:*) repo=${repo%:*} ;; esac
  if [ "$name" != "$ref" ] && [ "$name" != "$repo" ] &&
     [ "$(crane digest "$name" 2>/dev/null)" != "${ref##*@}" ]; then
    echo "MISSING tag $name"; missing=$((missing + 1))
  fi
  if ! index=$(crane manifest "$ref" 2>/dev/null); then
    echo "MISSING $ref"; missing=$((missing + 1)); continue
  fi
  for d in $(printf '%s\n' "$index" | jq -r '.manifests[]? | select(.platform.os == "linux") | .digest'); do
    crane manifest "$repo@$d" > /dev/null 2>&1 \
      || { echo "MISSING $repo@$d, a platform of $ref"; missing=$((missing + 1)); }
  done
done < keep-refs.txt
echo "missing: $missing"
# Expect "missing: 0"

Repair a damaged image

If the check lists anything missing, restart the registry first if garbage collection ran since its last restart. Then:

Delete each damaged index once. For each MISSING …, a platform of <reference> line, delete the index, replacing REFERENCE with the reference after a platform of. Two references can name the same index, for example a pod's NAME:TAG and the image list's NAME:TAG@sha256:…, so the check can list the same platform manifest under each. Delete once per index: when a second delete of the same index fails with MANIFEST_UNKNOWN, the first one already deleted it. Copying over a damaged image does not repair it: some copy tools, such as crane, find the index already there and skip it, and others, such as oras, fail with manifest blob unknown. Deleting the index first works with each of them:
```
ref=REFERENCE
repo=${ref%@*}
case ${repo##*/} in *:*) repo=${repo%:*} ;; esac
digest=$(crane digest "$ref") && crane delete "$repo@$digest" && echo "Deleted $repo@$digest"
```
A bare MISSING <reference> or a MISSING tag line needs no delete.
Copy the images again as in step 3 of Install Without Internet Access: rerun that step's copy for every release whose image list names a damaged image. Deleting an index removes every tag in its repository that names it, and an image unchanged between releases is in more than one release's list. A 1.4 release does not include Kaizen v3 images, so copy those from the earlier release that shipped them.
Run the check again. If you no longer have a release's files or access to its registry, contact Kamiwaza support.

No rollback to 1.3

Kamiwaza 1.4.0 cannot be rolled back to a 1.3 release in place, 1.3.4 included. The 1.4 upgrade moves Kamiwaza's database to a schema that no 1.3 release can run. If you need to return to the release you ran before the upgrade, the supported way is to restore the backup you took in Before you start: contact Kamiwaza support, who restore the platform from those pg_dumpall files. A Kaizen v3 deployment is restored from its own backup, as described in Back up and restore a Kaizen v3 deployment.

Do not try to roll back with Helm:

Do not run helm upgrade --install with --version 1.3.4, or any other 1.3 version, against a 1.4 installation. The 1.3 chart rolls some of Kamiwaza's workloads back to 1.3 before its database job refuses the 1.4 schema. Core then cannot start, and the platform API stays down.
Do not use helm rollback, and do not uninstall Kamiwaza. By default, uninstalling deletes the platform databases, the object store, and every extension's data.

If a 1.3 upgrade command has already run against a 1.4 installation, stop there. Do not run further Helm commands, and do not rerun the 1.4 upgrade as a way to recover: after a failed rollback it can complete without errors while Kamiwaza is not using your data. Keep the backup from Before you start and the release history (helm history kamiwaza -n kamiwaza), and contact Kamiwaza support.

The release notes of each later 1.4 release say whether this still applies.

If you upgrade from 1.3.3 or earlier

Kamiwaza 1.3.4 made the changes below. They apply when you upgrade from 1.3.3 or earlier.

A custom-named default Global Milvus

From 1.3.4, only the Milvus VectorDB named milvus-global-workroom is the default for the Global Workroom. In 1.3.3 the default was the newest Milvus VectorDB in the Global Workroom, whatever its name. If an administrator created a Global Milvus with another name after milvus-global-workroom, or instead of it, that instance received every user's Global ingest.

On an affected installation, after the upgrade:

The next Global ingest creates a new, empty milvus-global-workroom, unless one already exists. Global ingest and search use it.
The custom instance's collections stay where they are, but no longer appear in default Global search or listings.
The custom instance becomes an ordinary Global VectorDB owned by the administrator who created it. That administrator and other administrators can still reach it through the VectorDB API.

Check before you upgrade. As an administrator, list the Global Workroom's VectorDBs:

curl -sS -H "Authorization: Bearer $TOKEN" \
  -H "X-Workroom-ID: ffffffff-ffff-ffff-ffff-ffffffffffff" \
  "$KAMIWAZA_URL/api/context/vectordbs" |
  jq 'sort_by(.created_at) | .[] | {id, name, engine, created_at}'

The rows are listed oldest first. The installation is affected if the last row with engine milvus is not named milvus-global-workroom.

If you are affected, choose one:

Ingest again (recommended). Upgrade, then run ingestion again for the Global datasets and documents. The data lands in the new default. Then delete the custom instance if nothing else uses it.
Adopt the custom instance. This works only before any Global ingest runs on 1.3.4, while no milvus-global-workroom row exists. Rename the custom row. With the bundled PostgreSQL, in Kamiwaza's namespace:
```
kubectl exec -i core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza <<'SQL'
UPDATE context_vectordb_instances
   SET name = 'milvus-global-workroom'
 WHERE id = '<custom instance id>'
   AND NOT EXISTS (
     SELECT 1 FROM context_vectordb_instances
      WHERE name = 'milvus-global-workroom'
   );
SQL
```
UPDATE 1 means the row was renamed. UPDATE 0 means a milvus-global-workroom row already exists, so adopting no longer works; ingest again instead. If you renamed it after the upgrade, restart Core, and also deployment/core-api if your installation runs it:
```
kubectl -n kamiwaza rollout restart \
  deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker
```
On startup, Kamiwaza releases the instance's owner, and it becomes the shared default with its existing collections.

More than one tenant. In 1.3.3, the first user to ingest into the Global Workroom became the owner of milvus-global-workroom. 1.3.4 releases that owner on startup, and removes the owner's access records only in the Global Workroom's tenant, or the default tenant. That covers every single-tenant installation. If other tenants exist, Kamiwaza logs:

Canonical Global VectorDB owner release cleaned relation tuples in tenant ... only; N other tenant(s) exist.

A record left in another tenant grants nothing on the shared VectorDB. To keep the access records clean, remove it with /api/authz/resources/context_vectordb/grants.

From 1.3.4, on Kubernetes with Istio and on OpenShift with GreyMatter, an App Garden app no longer receives the Kamiwaza sign-in cookies (access_token, access_token_refresh, and the rest of that family) with each request. An app still receives its own cookies, unchanged. In earlier releases, any app a user opened could read that user's session and call Kamiwaza as that user, including as an administrator.

On OpenShift with GreyMatter, a Tool Shed tool also no longer receives any cookies, an Authorization header, or an X-App-Session-Token header, as on Kubernetes with Istio. Nothing in this section changes that for a tool. The usual upgrade applies this change, and the trust and enforce settings below; on OpenShift with GreyMatter, that includes repeating step 3 of the install page.

Apps that call Kamiwaza as the signed-in user with the sign-in cookies stop working unless an administrator trusts them:

Kaizen and Kaizen (Legacy) are trusted when an administrator deployed them from the App Garden on Kamiwaza 1.3.3 or later. A deployment created on an earlier release, or by a user who is not an administrator, is not trusted. Its model list is empty, chat reports that no chat models are available, and documents still upload but are not indexed for semantic search. Other features that call Kamiwaza as the user, such as tasks, are affected the same way.
Skills Library, Connector Builder, and your own apps that pass the user's cookies to Kamiwaza are not trusted by default.

To trust a deployment, add its runtime identifier, the path segment after /runtime/apps/ in the app's address, to core.auth.runtimeCookies.trustedRuntimeTargets in your values file, then rerun the upgrade command. Listing entries replaces the default list, so keep the two template: entries:

core:
  auth:
    runtimeCookies:
      trustedRuntimeTargets:
        - template:kaizen
        - template:kaizen-legacy
        - kaizen-1a2b3c4d   # an existing Kaizen deployment, by its identifier

A trusted app receives the session of every user who opens it, administrators included. Trust only apps you would trust with that session. Do not use wildcards or name prefixes: users choose deployment names.

For an existing Kaizen deployment, prefer its identifier to a redeploy. A new Kaizen deployment that an administrator deploys from the App Garden is trusted without an entry, but it does not carry over the existing deployment's data. See After the upgrade.

If an app you depend on breaks and you cannot trust it yet, set core.auth.runtimeCookies.enforce: false to restore the earlier behavior for every app. Tools still receive no cookies, Authorization header, or X-App-Session-Token header. This setting is temporary and will be removed in a later release.

The upgrade does not revoke anything an app could have obtained before it, such as an API token it created as a user. Administrators cannot see other users' API tokens. If you ran apps you do not trust, ask your users to open Settings, review the API Tokens on the My Settings tab, and revoke any they do not recognize.

Rotate the ForwardAuth signing secret

On Kubernetes with Istio and on OpenShift with GreyMatter, a user who could deploy an App Garden app or tool before 1.3.4 could read Kamiwaza's ForwardAuth signing secret through the API. Kamiwaza uses that secret to sign and verify the identity it passes to its own components. From 1.3.4 the API no longer returns it, but the secret itself is unchanged. Any signed-in user can deploy an App Garden app, so replace the secret after you upgrade, unless only administrators you trust could deploy apps and tools. Installations on Envoy ingress without Istio do not need this.

Do it after the upgrade, not before: until 1.3.4 is running, the new value could be read the same way.

What to expect. Users stay signed in. While the Ray head restarts, the console, the API, and app and model routes are unavailable, and jobs running on the Ray head are interrupted. Choose a quiet time.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

Optional: find the deployments that stored the secret. List App Garden deployments and look for the value Kamiwaza now withholds:
```
for kind in app tool service; do
  curl --fail --silent --show-error \
    --header "Authorization: Bearer $ADMIN_TOKEN" \
    "https://kamiwaza.example.com/api/apps/deployments?workroom_id=all&template_type=$kind" \
    | jq -r '.[] | select(tostring | contains("platform ForwardAuth signing secret"))
             | "\(.id) \(.name) owner=\(.owner_user_id)"'
done
```
Each line is a deployment whose environment holds the secret, so its owner could read it. Some deployments that held it are not listed: one that no longer exists, and one that stored it under a name Kamiwaza treats as a credential, such as a name ending in _TOKEN, _SECRET, _PASSWORD, or _API_KEY, because the response leaves such variables out. So an empty result does not show that nobody read the secret.
Replace the value. This writes a new random value into the forwardauth-signature Secret without printing it:
```
kubectl -n kamiwaza patch secret forwardauth-signature --type merge \
  --patch-file <(printf '{"data":{"secret":"%s"}}' \
    "$(openssl rand -base64 32 | tr -d '\n' | base64 | tr -d '\n')")
```
To confirm the change without printing the value, run kubectl -n kamiwaza get secret forwardauth-signature -o jsonpath='{.data.secret}' | openssl dgst -sha256 before and after: the hash changes. Later upgrades keep the new value: the chart never replaces a key that exists (from 1.4.1, under Argo CD too).

Kamiwaza signs and verifies with AUTH_FORWARD_HEADER_SECRET, which the chart reads from this Secret, and passes FORWARDAUTH_SIGNATURE_SECRET to apps when it is set. If your values files set either variable, for example in core.scheduler.extraEnv, that entry takes precedence over the Secret. Its value is the one that was exposed, and replacing the Secret does not replace it. To find such entries, run this command. It prints Deployment and variable names, never values:
```
kubectl -n kamiwaza get deployments -o json \
  | jq -r '.items[] | .metadata.name as $d
      | .spec.template.spec.containers[] | (.env // [])
      | select(map(.valueFrom.secretKeyRef.name) | index("forwardauth-signature"))
      | .[] | select(.name | test("^(AUTH_FORWARD_HEADER_SECRET|FORWARDAUTH_SIGNATURE_SECRET)$"))
      | select(.valueFrom.secretKeyRef.name != "forwardauth-signature")
      | "\($d) \(.name)"'
```
If it prints nothing, go on to step 3. Otherwise, remove each listed variable from your values files, so that Kamiwaza uses the Secret you just replaced. If you must keep AUTH_FORWARD_HEADER_SECRET there, give it a new random value of at least 32 characters instead, for example from openssl rand -base64 32. Then rerun the upgrade command from your environment's install page instead of step 3: the upgrade restarts the same workloads.

Restart every Kamiwaza workload that reads the secret, together:

kubectl -n kamiwaza get deployments -o json \
  | jq -r '.items[]
      | select([.spec.template.spec.containers[].env[]?.valueFrom.secretKeyRef.name]
               | index("forwardauth-signature"))
      | "deployment/" + .metadata.name' > forwardauth-readers.txt
cat forwardauth-readers.txt
xargs kubectl -n kamiwaza rollout restart < forwardauth-readers.txt
xargs -n 1 kubectl -n kamiwaza rollout status --timeout 15m < forwardauth-readers.txt

The list is core-scheduler, core-raycluster-head, the Ray worker Deployment (core-raycluster-worker), and core-api if you enabled it.

Verify. The pods of each listed Deployment are Ready and were created after step 2. Sign in to the console, open an App Garden app, and send a request to a model deployment. If you removed variables in step 2, its command now prints nothing.
OpenShift with GreyMatter: redeploy each diffusion (image generation) model deployment. Until it is redeployed, it refuses requests.

Running App Garden apps keep working and need no action. An app of your own that verifies Kamiwaza's identity signature itself rejects requests until you redeploy it.

Users disabled or deleted before the upgrade

From 1.3.4, disabling or deleting a user who is linked to a Keycloak account also disables that account, ends its sessions, and revokes its API tokens. Users you disabled or deleted on an earlier release are refused by Kamiwaza once 1.3.4 runs, but their Keycloak accounts stay enabled until you repeat the change. Repeat it for each of them. A user you disabled in Keycloak itself needs nothing.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

Disabled users. List them:

curl --fail --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/auth/users/" \
  | jq -r '.[] | select(.linked_subject_id != null and .active == false)
           | "\(.id) \(.username)"'

For each <id>, disable the user again:

curl --silent --show-error --request PUT \
  --header "Authorization: Bearer $ADMIN_TOKEN" \
  --header 'Content-Type: application/json' --data '{"active": false}' \
  --write-out ' %{http_code}\n' \
  "https://kamiwaza.example.com/api/auth/users/<id>"

Deleted users. The API does not list them, so read their IDs from the database. This query changes nothing:

kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
  psql -U core -d kamiwaza -At -c "
    SELECT id, username FROM users
    WHERE deleted AND linked_subject_id IS NOT NULL
      AND linked_subject_id NOT IN (
        SELECT linked_subject_id FROM users
        WHERE NOT deleted AND linked_subject_id IS NOT NULL)"

Each line is <id>|<username>. For each <id>, delete the user again:

curl --silent --show-error --request DELETE \
  --header "Authorization: Bearer $ADMIN_TOKEN" \
  --write-out ' %{http_code}\n' \
  "https://kamiwaza.example.com/api/auth/users/<id>"

Check the responses. Each request prints the user and then the HTTP status. 200 means the Keycloak account is now disabled, or no longer exists. A 503 names the step that failed; repeat the same request until it returns 200.

Extension credentials from before 1.3.4

From 1.3.4, a new App Garden deployment keeps the passwords, tokens, and API keys in its environment in its own Kubernetes Secret, and App Garden deployment responses no longer return them. A deployment created before 1.3.4 still holds them as plain environment variables, in its pods and in its ConfigMap, until it is removed. Anyone who can read pods or ConfigMaps in Kamiwaza's namespace can read them there, including with the built-in view role, which cannot read Secrets.

For the extensions in the App Garden catalog, these values are the catalog's published defaults, such as dev, postgres, and minioadmin, unless an administrator entered a different value when deploying. A published default is not a secret, and a new deployment gets the same default, so these need nothing. Act only on a value that you or another administrator entered when deploying an extension before 1.3.4, such as an API key, a token, or a password.

Do not stop or replace a Kaizen or Kaizen (Legacy) deployment for this. A new deployment starts empty, and stopping a deployment in the App Garden deletes its data.

List the names each deployment from before 1.3.4 holds in plain text. The command prints names only, never values:

kubectl -n kamiwaza get configmap -l kamiwaza.io/extension-runtime=direct -o json \
  | jq -r '.items[]
      | .metadata.labels as $l
      | [ (.metadata.annotations["extensions.kamiwaza.io/rendered-spec"] // "{}" | fromjson)
          | .spec.services[]?.env[]?
          | select((.value // "") != "")
          | select((.name | test("(^|_)(PASSWORD|PASSWD|PASSPHRASE|SECRET|TOKEN|APIKEY|CREDENTIALS?)$|(^|_)(API|SECRET|PRIVATE|ACCESS|ENCRYPTION|SIGNING|MASTER)_KEY$"; "i"))
                   or (.value | tostring | test("://[^/?#@ ]*:[^/?# ]+@")))
          | .name ] | unique | select(length > 0)
      | "\($l["kamiwaza.io/deployment-id"] // "-") \($l["extensions.kamiwaza.io/deployment-id"]): \(join(" "))"'

Each line names a deployment by its ID and its runtime, then the variables it holds in plain text. A deployment created on 1.3.4 or later does not appear.

For each variable whose value an administrator entered, rotate the credential where it was issued: for example, revoke the API key at its provider and create a new one. If you used the same password anywhere else, change it there too. Until the deployment has the new value, whatever in it uses the credential stops working.
Check whether the deployment keeps data, using the runtime from step 1:
```
kubectl -n kamiwaza get pvc -l extensions.kamiwaza.io/deployment-id=<runtime>
```
- No volume claims, and not Kaizen or Kaizen (Legacy): deploy the extension again from the App Garden with the new value, check that the new deployment works, then stop the old deployment in the App Garden. Stopping it there also removes its plain-text copy.
- Volume claims, or Kaizen or Kaizen (Legacy): do not stop or replace the deployment. Contact Kamiwaza support.
A deployment that was stopped through the API (DELETE /api/apps/deployment/<id>), rather than in the App Garden, keeps its ConfigMap and its volumes, so it still appears in step 1. When you no longer need its data, remove it with DELETE /api/apps/deployment/<id>/purge, which deletes its volumes too. <id> is the ID that step 1 prints first.

Check: run step 1 again. The line of each deployment you replaced or purged is gone, and the new deployments do not appear. A deployment that step 3 leaves to Kamiwaza support is still listed.

Workroom context settings removed by the upgrade

From 1.3.4, only administrators can set the settings of an ontology or VectorDB instance, and the settings of an OmniParse instance other than its model selection, enable, timeout, and processing settings. Before 1.3.4, any member of a workroom could set them, including where an instance connects and with what credentials. The upgrade removes such settings from OmniParse, ontology, and VectorDB instances in a workroom other than the Global Workroom, and from the record each instance is deployed from, including values an administrator set: Kamiwaza cannot tell who set them. Instances in the Global Workroom do not change.

OmniParse: every setting is removed except the model selection, enable, timeout, and processing settings.
Ontology and VectorDB: every setting whose name marks an endpoint, URL, or host, a credential, a TLS or authentication setting, a proxy, a trust store, an identity, or a loader path is removed. Settings such as model names, timeouts, and sizes are kept, and a VectorDB keeps the credentials Kamiwaza generated for its own Milvus.
Graphiti endpoints: Kamiwaza restores an ontology instance's chat and embedding endpoints from its chat and embedding models on the next ontology operation, which can return a 503 while the instance restarts; retry it. If Kamiwaza cannot restore an endpoint, the operation returns a 503 with the code pre_gate_endpoint_scrubbed, and an administrator selects the instance's chat and embedding models again, as in step 4 below.

Running pods. The first time Core starts after the upgrade, it also removes these settings from the running pods of the instances the upgrade changed. On each such pod, Core changes only the removed settings and any setting built from one of them. Each one gets the value the instance would be deployed with now, or is removed if it would not be set. For an OmniParse instance, Core first restores the settings Kamiwaza writes itself, such as its model endpoints and the Kamiwaza API address. The affected pods of each changed instance restart once, and its data volumes are kept. Other instances do not restart. A stopped or failed instance has no running pod to change; if it is started again, it starts without these settings. An ontology instance's chat and embedding endpoints are restored as described above.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

Check what Core did. Core's scheduler logs each instance it changed, and a summary. The lines name instances and settings, never their values, and each can appear twice, once as JSON:
```
kubectl -n kamiwaza logs deployment/core-scheduler -c core \
  | grep -e 'scrubbed context instance' -e 'scrubbed settings' \
    -e 'left the pod of' -e 'changed meanwhile' -e 'on its running pod'
```
- Startup: removed scrubbed settings <settings> from the running pod of <type> instance <instance ID>; its pods restart once: Core removed the settings from that instance's pod.
- Startup: scrubbed context instance pod env reconcile {'found': …, 'updated': …, 'unchanged': …, 'skipped': …, 'errors': …}: the summary, logged at every start. At a later start, it counts only the instances still listed. errors should be 0, and skipped should count only instances whose deployment is not running, described next. If no summary line appears, restart the scheduler as shown below. If it still does not appear, contact Kamiwaza support.
- Startup: left the pod of <type> instance <instance ID> for the next start: its deployment is not running: the instance is stopped or failed, so it has no running pod to change. Core has already removed the settings from what the instance is deployed from, so if it is started again, it starts without them. Core logs this line, and counts the instance in skipped, at every start until the instance runs again or is deleted. At the first start after it runs again, Core checks its pod and stops listing it. No action is needed, unless the removed settings included a password for data the instance already holds: it would start without that password and could not open that data, so contact Kamiwaza support before you start it.
- Startup: left the pod of <type> instance <instance ID> for the next start: <reason> with any other reason, or a line that says Core could not remove scrubbed settings from an instance's pod: the instance was deploying, stopping, or being deleted, or the change failed. Core tries again at its next start. To try again now, wait until the instance has finished deploying or stopping, then restart the scheduler and check the log again:
```
kubectl -n kamiwaza rollout restart deployment/core-scheduler
kubectl -n kamiwaza rollout status deployment/core-scheduler --timeout 15m
```
  If Core leaves the instance again for the same reason, contact Kamiwaza support.
- Startup: the pod env record of <type> instance <instance ID> changed meanwhile; it is reconciled on the next start: the instance's record changed while Core worked on it, so Core checks the instance again at its next start. The summary does not count this in skipped. To check now, restart the scheduler as shown above.
- Startup: <type> instance <instance ID> keeps <settings> on its running pod, logged as an error: Core leaves some settings on the pod, such as identity and credential settings. A new password, for example, would lock an instance out of data created with the old one. The value stays on the running pod, and Core logs that line again at every start. If the value is one an administrator set or Kamiwaza generated, you can leave it on the pod; otherwise, contact Kamiwaza support. Core does not keep the value for the instance's next deployment, though. If it is a password that protects data the instance already created, stopping and starting the instance would leave it unable to open that data. Do not stop that instance, and contact Kamiwaza support if it must keep the value.

List the instances the upgrade could change. This query changes nothing. It prints each instance's type, workroom ID, instance ID, and name, never its settings:

kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
  psql -U core -d kamiwaza -At -c "
    SELECT 'omniparse', workroom_id, id, name FROM context_omniparse_instances
    WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
    UNION ALL
    SELECT 'ontology', workroom_id, id, name FROM context_ontology_instances
    WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
    UNION ALL
    SELECT 'vectordb', workroom_id, id, name FROM context_vectordb_instances
    WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
    ORDER BY 1, 2, 4"

Each line is <type>|<workroom ID>|<instance ID>|<name>.

Find the values the upgrade removed. Compare against your own records of what administrators set, or read an instance's settings from before the upgrade out of the backup you took in Before you start. This command changes nothing. It prints the instance's row from the backup. The instance's settings are the row's JSON object, credentials included, so run it where no one else can see the output:
```
kubectl -n kamiwaza exec -i core-postgres-0 -c postgres -- \
  pg_restore --data-only --table=context_omniparse_instances --file=- \
  < kamiwaza-before-upgrade.dump \
  | grep '^<instance ID>'
```
For an ontology or VectorDB instance, use the table context_ontology_instances or context_vectordb_instances. Optionally, cross-check the upgrade's own record: the database job's log names each instance the upgrade removed settings from, with its workroom and the names of the removed settings, never their values. By default the job and its log are removed about an hour after the job finishes:
```
kubectl -n kamiwaza logs -l app.kubernetes.io/name=core-db-init -c db-init --tail=-1 \
  | grep -e 'removed privileged' -e 'config not an object'
```
A line that says config not an object names an instance whose settings Kamiwaza could not read. Core has no record of what the upgrade removed from it, so the values may stay on its running pod. Save it as below, or delete it. If its values must leave the running pod, contact Kamiwaza support.
Re-enter the values an administrator set. Save an instance only to re-enter values. A save stores them in the instance's settings and the record it is deployed from. It does not change the instance's running pod: if a value you re-enter must reach the running pod, contact Kamiwaza support. Each request prints the HTTP status; 200 means the instance was saved.
- OmniParse: put the values in a file, so they stay off the command line, as {"config": {"<setting>": "<value>"}}. Then send it with the instance's workroom ID:
```
curl --silent --show-error --request PUT \
  --header "Authorization: Bearer $ADMIN_TOKEN" \
  --header "X-Workroom-ID: <workroom ID>" \
  --header 'Content-Type: application/json' \
  --data @omniparse-settings.json \
  --write-out ' %{http_code}\n' \
  "https://kamiwaza.example.com/api/context/omniparses/<instance ID>"
```
- VectorDB: send them the same way to /api/context/vectordbs/<instance ID>.
- Ontology: Kamiwaza 1.3 has no request to set an ontology instance's settings. Kamiwaza restores the chat and embedding endpoints itself, as described above. If an operation returns pre_gate_endpoint_scrubbed, select the instance's models again with PUT /api/context/ontologies/<instance ID>/model-bindings and the body {"chat": {"deployment_id": "<chat model deployment ID>"}, "embedding": {"deployment_id": "<embedding model deployment ID>"}}. To restore another setting the upgrade removed from an ontology instance, contact Kamiwaza support.

Supported upgrades​

Before you start​

Owner trust​

Upgrading to 1.4.0​

Remove credentials from Keycloak and OIDC URLs​

Move local users' passwords off argon2​

Upgrade​

Upgrade through Argo CD​

If Argo CD already synced 1.4.0​

Upgrade halted: PostgreSQL data directory​

Upgrade halted: Context TLS preflight​

Upgrade halted: Keycloak passwords (argon2)​

Upgrade from the OpenShift overlay files​

External model endpoints and tlsInsecure​

Extension platform gateway​

Extension operator not supported on public installs​

Workroom Manager extension removed​

Connectors at internal addresses​

Extension runtime policy under the hardened profile​

Exempt paths on the standalone Envoy edge​

Verify​

After the upgrade​

Kubernetes API addresses changed by the upgrade​

Kaizen v3 on Kamiwaza 1.4​

New Kaizen v3 deployments are refused​

Deploying Kaizen (Legacy) is refused​

Existing Kaizen v3 deployments are kept​

Stop a Kaizen v3 deployment without losing it​

Back up and restore a Kaizen v3 deployment​

Back up​

Restore​

Keep the Kaizen v3 images​

Clean up the registry safely​

Garbage collection​

Decide what to keep​

Delete an earlier release's image​

Check the registry after every cleanup​

Repair a damaged image​

No rollback to 1.3​

If you upgrade from 1.3.3 or earlier​

A custom-named default Global Milvus​

App Garden apps no longer receive the sign-in cookies​

Rotate the ForwardAuth signing secret​

Users disabled or deleted before the upgrade​

Extension credentials from before 1.3.4​

Workroom context settings removed by the upgrade​

Supported upgrades

Before you start

Owner trust

Upgrading to 1.4.0

Remove credentials from Keycloak and OIDC URLs

Move local users' passwords off argon2

Upgrade

Upgrade through Argo CD

If Argo CD already synced 1.4.0

Upgrade halted: PostgreSQL data directory

Upgrade halted: Context TLS preflight

Upgrade halted: Keycloak passwords (argon2)

Upgrade from the OpenShift overlay files

External model endpoints and `tlsInsecure`

Extension platform gateway

Extension operator not supported on public installs

Workroom Manager extension removed

Connectors at internal addresses

Extension runtime policy under the hardened profile

Exempt paths on the standalone Envoy edge

Verify

After the upgrade

Kubernetes API addresses changed by the upgrade

Kaizen v3 on Kamiwaza 1.4

New Kaizen v3 deployments are refused

Deploying Kaizen (Legacy) is refused

Existing Kaizen v3 deployments are kept

Stop a Kaizen v3 deployment without losing it

Back up and restore a Kaizen v3 deployment

Back up

Restore

Keep the Kaizen v3 images

Clean up the registry safely

Garbage collection

Decide what to keep

Delete an earlier release's image

Check the registry after every cleanup

Repair a damaged image

No rollback to 1.3

If you upgrade from 1.3.3 or earlier

A custom-named default Global Milvus

App Garden apps no longer receive the sign-in cookies

Rotate the ForwardAuth signing secret

Users disabled or deleted before the upgrade

Extension credentials from before 1.3.4

Workroom context settings removed by the upgrade