Upgrade Kamiwaza

Upgrade Kamiwaza in place: rerun the helm upgrade --install command from your environment's install page with the new --version and the same values files. An in-place upgrade keeps Kamiwaza's models, workrooms, users, data, and storage volumes, and existing extension deployments keep running.

An upgrade from 1.3 needs one more thing first: the cluster owner's trust settings for model serving, which a 1.3 values file does not have. See Set up owner trust.

Supported upgrades

You can upgrade in place from a 1.3 release by following this page, and from one release of this line to a later one by rerunning the upgrade command. There is no in-place upgrade from Kamiwaza 1.2 or earlier: contact your Kamiwaza representative to plan an assisted migration to a new installation.

You cannot roll 1.4.0 back to a 1.3 release in place. The way back is to restore the backup you take before the upgrade, so that backup is required. See No rollback to 1.3.

Before you start

1. Read the release notes for every release after the one you run, up to and including the one you are installing. Each release lists any action it requires. See Release notes.

2. Back up. This step is required. The backup you take now is the only way back to the release you run, because 1.4.0 cannot be rolled back in place (see No rollback to 1.3). Back up Kamiwaza's PostgreSQL databases by running pg_dumpall in each Kamiwaza PostgreSQL pod:

   (
     set -o pipefail
     umask 077
     STAMP=$(date +%Y%m%dT%H%M%S)
     for STS in core-postgres keycloak-postgres core-spicedb-postgres; do
       OUT="kamiwaza-$STS-$STAMP.sql.gz"
       if ! FOUND=$(kubectl -n kamiwaza get statefulset "$STS" \
              --ignore-not-found -o name); then
         echo "FAILED $STS: cannot read the cluster"
         continue
       fi
       if [ -z "$FOUND" ]; then
         echo "SKIPPED $STS: not found"
         continue
       fi
       if kubectl -n kamiwaza exec "$STS-0" -c postgres -- \
            sh -c 'pg_dumpall -U "${POSTGRES_USER:-postgres}"' \
            | gzip > "$OUT.partial" \
          && gzip -t "$OUT.partial" \
          && gzip -dc "$OUT.partial" | tail -n 5 \
            | grep -q 'PostgreSQL database cluster dump complete' \
          && mv "$OUT.partial" "$OUT"; then
         echo "OK $OUT"
       else
         rm -f "$OUT.partial"
         echo "FAILED $STS"
       fi
     done
   )

   The output must show OK for core-postgres and keycloak-postgres. core-spicedb-postgres is OK too, unless SpiceDB uses another datastore, when it is SKIPPED. If any line says FAILED, or one of the first two is SKIPPED, do not upgrade: check that kubectl points at the Kamiwaza cluster and run the backup again. Only a file named on an OK line is a backup. Each run writes new, time-stamped files and never replaces older ones.

   Keep the files, your values files and the version you run now together, outside the cluster, and keep them after the upgrade: they are the only backup of this release, and backups you take later are of 1.4. The dumps contain credentials and user data, so store them as you store your other Kamiwaza secrets. They cover the platform databases only. The object store, the platform's Kubernetes Secrets (its generated passwords and keys) and its storage volumes stay in the cluster; do not delete them. Kamiwaza support restores from these backups if an installation has to be recovered. Then back up each Kaizen deployment:

   • Kaizen v3 (Kaizen (Legacy), kaizen-legacy, in the App Garden): follow Back up and restore a Kaizen v3 deployment. A Kaizen v3 deployment has no object store.
   • Kaizen v4 (Kaizen, kaizen): run pg_dumpall in its PostgreSQL pod, and archive the /data directory of its SeaweedFS pod. Restoring this backup is not documented; contact Kamiwaza support if you need it.

3. Find your values files, and the order you installed with.

4. Air-gapped installations: copy the new release into your registry, as described in Install Without Internet Access. Keep the previous release's engine images in your registry until every model deployment has been redeployed on the new release, and its extension images while deployments of those extension versions still run. Keep the chart and images of the release you run now as well, until you are sure you will not need to restore the backup from step 2. If the installation runs Kaizen v3, see Keep the Kaizen v3 images as well.

5. From 1.3: set up owner trust, as described in the next section.

6. If you created the kamiwaza-edge-trust ConfigMap by hand, as the 1.3 prerequisites pages described, move its CAs into your values file: the CA that signs the Kamiwaza domain into global.trust.edgeAnchor.pem, and any other CA into global.trust.extraAnchors. Keep the CA files. Delete the ConfigMap right before you run the upgrade. The chart is now the only writer of that ConfigMap, and an upgrade that sets these values while one that Helm does not manage exists stops before it changes anything. If the upgrade stops for another reason, concatenate the same CA files into one and recreate the ConfigMap with the 1.3 command, so that 1.3 pods that restart meanwhile still trust the domain (if the chart already published it, kubectl create reports that it exists, which is fine), and delete it again right before you rerun the upgrade. See TLS Trust for Private Certificate Authorities.

7. Remove credentials from Keycloak and OIDC URLs. From 1.4.0, Core refuses to start while a Keycloak, JWKS, issuer, or OIDC endpoint URL setting contains user:password@. See Remove credentials from Keycloak and OIDC URLs.

8. If you created local users, move their passwords off argon2. From 1.4.0, Keycloak runs its FIPS build, which cannot verify the argon2 password hashes that Keycloak gives every user created with a password on 1.3, for example on the Local Users settings page. The upgrade stops before it changes any Kamiwaza resource while one of those passwords remains. Signing in on 1.3 does not change the hash by itself: set the realm's hash policy first, then have each user sign in once. See Move local users' passwords off argon2. If the upgrade has already stopped, see Upgrade halted: Keycloak passwords (argon2).

9. GreyMatter and BYO edge certificates: publish the edge CA. From 1.4.0, Context Service model calls to the platform's own origins verify TLS. On OpenShift with GreyMatter, or Istio or Envoy with an edge certificate the chart does not create or cannot verify (network.tls.mode: byo, the default and EZUA's mode; customerCA; or cert-manager with a custom issuer), the upgrade stops unless your values set global.trust.edgeAnchor.pem, global.trust.edgeAnchor.publiclyTrusted: true (only for an edge certificate from a public CA), or core.context.embedding.verifySsl: false. Values that already turn verification off, and a hand-made kamiwaza-edge-trust ConfigMap you have not yet moved into values, also let it pass. See Upgrade halted: Context TLS preflight.

10. Remove surface: extension from global.workroomManager. Kamiwaza 1.4.0 removes the standalone Workroom Manager extension, and the upgrade stops before it changes anything while your values still set global.workroomManager.surface: extension. Remove the key to keep the default, disabled, or set it to core for the built-in Workrooms page. See Workroom Manager extension removed.

11. Public installs, mirrored and air-gapped included: remove the extension operator. The public release does not support the extension-operator runtime. Remove extensionOperator.enabled: true from your values, and core.extensionRuntime.mode: operator if you set it, to keep the default direct runtime. From 1.4.0, on an install whose images come from ghcr.io/kamiwaza-ai/releases/, the upgrade stops before it changes anything while the operator is enabled. A mirrored or air-gapped install is not stopped, so its operator pod would fail to pull its image. See Extension operator not supported on public installs.

12. Kubernetes with Istio installations: if your values files set global.authz.extAuthz.serviceHost, it must name core-api.<namespace>.svc.<cluster domain> in Kamiwaza's namespace, for example core-api.kamiwaza.svc.cluster.local. From 1.3.3, any other value, such as core-api.kamiwaza.svc without the cluster domain, makes helm upgrade fail before it changes anything, with an error that the host does not resolve into release namespace "<namespace>". Remove the value from your values files, or correct it. Do not turn off the access check that the error message names instead: that removes authentication from model or tool routes.

13. Execution or dataset gates that use your own class: from 1.3.4, Kamiwaza loads a gate class only from its own gate modules, from an installed gate package, or from a package listed in AUTH_GATE_CLASSPATH_PREFIXES. After the upgrade, an execution gate whose class is not allowed makes every job submission fail, local ones included, and a dataset whose gate class is not allowed is not served. Check both before you upgrade, as an administrator:

    • The execution gate: call GET /api/cluster/execution-gate.
    • Each gated dataset, one whose properties include gate or gate_type: call GET /api/catalog/datasets/<URL-encoded URN>/gate. The response names the dataset's gate class.

    A 403 with reason classpath_not_allowed means the class is not allowed. List its package in your values file before you upgrade, as described in Allow your gate's classpath.

Do not uninstall Kamiwaza before an upgrade. Uninstalling deletes the platform databases, the object store, and every extension's data.

Set up owner trust

Every installation now serves models through tenant-mode inference: Kamiwaza places a model deployment only as a profile signed by the cluster owner allows. In 1.3 this was off by default, so a 1.3 values file has no owner public key and no cluster binding. There is no default for them, because they say whose signature Kamiwaza trusts. Without them the chart refuses to render, before the upgrade changes anything:

Error: execution error at (kamiwaza/charts/core/templates/...): missing cluster owner trust:
core.inferenceResources.ownerPublicKey is empty; core.inferenceResources.clusterBindingId is empty.
...

The error lists every key that is missing. A values file that sets core.inferenceResources.enabled: false gets core.inferenceResources.enabled is not true in the list as well.

Do this before you upgrade. With tenant-mode inference off, as it is by default on 1.3, the running platform does not read the profile or the namespace annotation that these steps create.

1. Get the new release's core image, by digest, from the release's image list; see Install Without Internet Access. The owner tool ships in it. Sign with the core image of the release you are installing, not the one you run, so that the release that verifies the profile is the one that signed it.

2. Create the key and publish the signed profile. As the cluster owner, follow Tenant-mode inference, from Before you start through step 5, with that core image:

   • Step 1 creates the signing key and prints its public key.
   • Step 2 chooses the cluster binding ID.
   • Step 3 detects each GPU host's facts. Skip it if you serve models on CPUs only.
   • Step 4 writes the profile, and step 5 signs it and publishes it in the Kamiwaza namespace.

   Keep the private key with the cluster owner, out of source control and out of the Kamiwaza namespace.

3. Add the owner trust to your values file, from step 6. This is the change from a 1.3 values file:

   core:
     inferenceResources:
       enabled: true
       bundleConfigMap: <PROFILE_CONFIGMAP>              # the ConfigMap name from step 5
       ownerPublicKey: <OWNER_PUBLIC_KEY>                # publicKey from step 1
       clusterBindingId: <CLUSTER_BINDING_ID>            # from step 2

   Merge it under your file's existing core: key; do not add a second core:. If the file sets core.inferenceResources.enabled: false, change it to true. Keep any other core.inferenceResources values it sets.

To check the values file before you upgrade, render the chart with it. The render needs no cluster access, and exits without an error when the file is complete:

helm template kamiwaza oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <new version> --namespace kamiwaza \
  --values kamiwaza-values.yaml > /dev/null

Pass every values file you install with, in the same order. On an air-gapped installation, use the chart in your own registry.

Remove credentials from Keycloak and OIDC URLs

From 1.4.0, when authentication is enabled, Core refuses to start while a Keycloak, Keycloak admin, JWKS, issuer, or external OIDC endpoint URL setting contains embedded credentials (user:password@). The pods log RuntimeError: Refusing to start: <SETTING> contains '@' (or '%40') and restart in a loop. The settings checked are listed in the 1.4.0 release notes. Do this before the upgrade, and before a new 1.4.0 installation that reuses earlier values or overrides:

1. List the settings that contain @ or %40. Run the whole block in one shell. It prints setting names only, never values, and prints CHECK FAILED if kubectl could not read the cluster.

   (
   set -o pipefail
   names='^(AUTH_GATEWAY_(KEYCLOAK(_PUBLIC|_ADMIN)?_URL|JWKS_URL|JWT_ISSUER|OIDC_(ISSUER|AUTHORIZATION_ENDPOINT|TOKEN_ENDPOINT|USERINFO_ENDPOINT|JWKS_URI))|KEYCLOAK_URL)$'
   kubectl -n kamiwaza get configmap core-config -o json |
     jq -r --arg n "$names" '.data // {} | to_entries[]
       | select((.key | ascii_upcase | test($n)) and (.value | test("@|%40"; "i")))
       | "configmap/core-config: \(.key)"' || echo 'CHECK FAILED: configmap'
   kubectl -n kamiwaza get configmap core-config -o json |
     jq -r --arg n "$names" 'if [.data // {} | keys[] | ascii_upcase | test($n)] | any
       then empty else "NOTE: core-config holds none of these settings; find where they are set" end' ||
     echo 'CHECK FAILED: configmap'
   kubectl -n kamiwaza get deployment,statefulset,daemonset,job,cronjob,pod -o json |
     jq -r --arg n "$names" '.items[] | "\(.kind)/\(.metadata.name)" as $w
       | (.spec.jobTemplate.spec.template.spec // .spec.template.spec // .spec)
       | ((.containers // []) + (.initContainers // []))[]
       | .env[]? | select(.name | ascii_upcase | test($n))
       | if .valueFrom then "\($w): \(.name) (from a reference; check it by hand)"
         elif ((.value // "") | test("@|%40"; "i")) then "\($w): \(.name)"
         else empty end' | sort -u || echo 'CHECK FAILED: workloads'
   )

   No output means that neither core-config nor the inline env of any Deployment, StatefulSet, DaemonSet, Job, CronJob, or pod sets a value that needs a change. The block does not read settings loaded with envFrom from a Secret or another ConfigMap. Check those by hand.

2. Delete the user:password@ part of each listed setting, where you set it. If you manage the installation with GitOps, change it in the source, or the next sync puts it back. In an OIDC, JWKS or issuer URL, an @ in the path or query of a well-formed URL can stay. A listed KEYCLOAK_URL is ignored while AUTH_GATEWAY_KEYCLOAK_URL is set, but remove its credentials too.

3. Put a credential that Core needs in its own setting:

   • Keycloak client: AUTH_GATEWAY_KEYCLOAK_CLIENT_ID and AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET.
   • External OIDC identity provider: AUTH_GATEWAY_OIDC_CLIENT_ID and AUTH_GATEWAY_OIDC_CLIENT_SECRET.
   • Keycloak administrator: nothing to move. Core reads KEYCLOAK_ADMIN_PASSWORD, which the chart sets from the password key of the keycloak-admin Secret.
   • JWKS and issuer URLs need no credentials.

   Load a client secret from a Kubernetes Secret, not as plain text in core-config.

4. Rotate every password that was in one of these URLs. Earlier releases wrote it to the logs in clear, and copies already sent to a log store stay there after you rotate it. To rotate the Keycloak administrator password, change it in Keycloak and in the keycloak-admin Secret together, then restart the Core pods; changing only the Secret makes Core's admin calls fail.

Move local users' passwords off argon2

From 1.4.0, Keycloak runs its FIPS build, keycloak-fips, which cannot verify a password hashed with argon2. Keycloak hashes passwords with argon2 in any realm whose password policy names no hash algorithm, and on 1.3 that includes the kamiwaza realm. So every user created with a password on 1.3, on the Local Users settings page or through the API, has an argon2 password, and could not sign in after the upgrade. Users who sign in only through an external identity provider have no Keycloak password and are not affected. If your values set global.fips.enabled: false, Keycloak keeps its standard image and you can skip this section.

The upgrade moves the accounts whose passwords the chart keeps in its own Secrets. It cannot move your users' passwords, because it does not know them, so it stops while any remain (see Upgrade halted: Keycloak passwords (argon2)). Keycloak replaces a password's hash when its user signs in, using the realm's current policy. A user who signs in after the realm's policy names pbkdf2 moves to pbkdf2 without changing their password; a sign-in before that keeps argon2.

Do this on 1.3, before you upgrade, in one shell:

1. Give the kamiwaza realm a pbkdf2 hash policy. Sign Keycloak's admin CLI in inside the Keycloak pod and print the realm's current password policy. The administrator password goes from the keycloak-admin Secret to the CLI through standard input, never on a command line:

   NS=kamiwaza
   POD=$(kubectl -n "$NS" get pod -l app.kubernetes.io/name=keycloak \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl -n "$NS" get secret keycloak-admin -o jsonpath='{.data.password}' \
     | base64 -d \
     | kubectl -n "$NS" exec -i "$POD" -c keycloak -- sh -c \
       'IFS= read -r p; KC_CLI_PASSWORD="$p" /opt/keycloak/bin/kcadm.sh config credentials \
          --config /tmp/kcadm.config --server http://localhost:8080 --realm master --user admin'
   kubectl -n "$NS" exec "$POD" -c keycloak -- /opt/keycloak/bin/kcadm.sh \
     get realms/kamiwaza --config /tmp/kcadm.config --fields passwordPolicy

   admin, the keycloak-admin Secret and its password key are the chart's defaults. If your values change them, use yours in the sign-in command: the Secret name and key in kubectl get secret, and the username in --user. auth.keycloak.initUsers.keycloak.adminUsername, adminSecretName and adminPasswordKey set the administrator, its Secret and its key. An empty adminSecretName or adminPasswordKey falls back to auth.keycloak.auth.existingSecret or auth.keycloak.auth.passwordSecretKey. With auth.keycloak.initUsers.enabled: false, the administrator is always admin, and only existingSecret and passwordSecretKey apply.

   { } means the realm has no password policy. Set the hash policy that the 1.4.0 upgrade itself sets:

   kubectl -n "$NS" exec "$POD" -c keycloak -- /opt/keycloak/bin/kcadm.sh \
     update realms/kamiwaza --config /tmp/kcadm.config \
     -s 'passwordPolicy=hashAlgorithm(pbkdf2-sha512) and hashIterations(210000)'

   If the realm already has a policy, update replaces it, so keep its entries: set passwordPolicy to the current policy followed by and hashAlgorithm(pbkdf2-sha512) and hashIterations(210000), without any hashAlgorithm or hashIterations entry it had. For example, length(12) becomes length(12) and hashAlgorithm(pbkdf2-sha512) and hashIterations(210000). Run the get command again to check. If you created users with passwords in another realm, set the same policy there.

   The admin CLI is the supported path: Kamiwaza's edge does not route the Keycloak admin console, and a port-forwarded console cannot load. If you reach the console another way, you can set the same policy there: open the kamiwaza realm, then Authentication, Policies, Password policy. Add Hashing Algorithm with pbkdf2-sha512 and Hashing Iterations with 210000, and save.

   Users created from now on get pbkdf2 passwords. The upgrade's password migration job sees that the realm already hashes with pbkdf2 and leaves the policy as you set it. Later in the same upgrade, though, the chart applies its own password policy to the kamiwaza realm. It replaces the hash algorithm, hash iterations, length, uppercase, lowercase, digits, special characters, password history and not-the-username entries with the values under auth.keycloak.realm.passwordPolicy, and keeps any other entry. A value you leave unset takes the chart's preset: pbkdf2-sha512 with 210,000 iterations, at least 15 characters with one uppercase letter, one lowercase letter, one digit and one special character, a history of 5 passwords, and not the username. So an entry such as length(12) lasts only until the upgrade, unless you also set it in your values (auth.keycloak.realm.passwordPolicy.length: 12) or turn the chart's policy off with auth.keycloak.realm.passwordPolicy.enabled: false. The stig profile refuses both: it allows no value below its preset and never allows turning the policy off. Existing passwords keep working, but every password set after the upgrade, including a reset, must meet the chart's policy. See Local user password policy.

2. Have each local user sign in to Kamiwaza once. Each sign-in moves that user's password to pbkdf2. For a user who cannot sign in, reset the password instead, under Local Users in Settings: a reset also uses the new policy. You can also delete users who no longer need access.

3. Check that no user is left on argon2. List every password that Keycloak still holds as argon2, or whose stored hash names no algorithm. The query matches the stored text rather than parsing it as JSON, so a hash it cannot read is listed instead of failing the query:

   kubectl -n "$NS" exec -i keycloak-postgres-0 -c postgres -- \
     psql -U keycloak -d keycloak <<'SQL'
   SELECT r.name AS realm, u.username,
          CASE WHEN c.credential_data ~ '"algorithm"\s*:\s*"argon2"'
               THEN 'argon2' ELSE 'unreadable hash' END AS hash
     FROM credential c
     JOIN user_entity u ON u.id = c.user_id
     JOIN realm r ON r.id = u.realm_id
    WHERE c.type = 'password'
      AND (c.credential_data IS NULL
           OR c.credential_data ~ '"algorithm"\s*:\s*"argon2"'
           OR c.credential_data !~ '"algorithm"\s*:\s*"[^"]+"');
   SQL

   Some argon2 rows are expected: the upgrade signs in the accounts whose passwords the chart keeps, and moves them itself. Those are the administrator in the master realm (admin unless your values name another, see step 1, and temp-admin if it still exists), and in the kamiwaza realm each user listed under auth.keycloak.initUsers.users in your values (by default admin and svc-core; none when auth.keycloak.initUsers.enabled is false). The chart keeps each listed user's password in a kamiwaza-user-<username> Secret, and the upgrade moves only a listed user whose Secret exists. kubectl -n "$NS" get secret -o name | grep kamiwaza-user- lists those Secrets as a hint, but a Secret left behind for a user no longer in the list does not make that user expected. Any other argon2 row is a user who still needs to sign in or have their password reset. The upgrade also stops on an unreadable hash row, and the job's log names that account with (an unreadable hash): reset its password.

4. Remove the admin CLI's session file from the Keycloak pod:

   kubectl -n "$NS" exec "$POD" -c keycloak -- rm -f /tmp/kcadm.config

Upgrade

As the identity that installed Kamiwaza, rerun the helm upgrade --install command from your environment's install page, with the new --version and every values file you installed with, in the same order. For a Kubernetes with Istio installation with one values file:

helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <new version> \
  --namespace kamiwaza \
  --values kamiwaza-values.yaml \
  --timeout 70m

Keep --timeout 70m (4200 seconds). Core's first start on a new release can take about an hour, and the upgrade's own jobs run inside the same helm upgrade, so a shorter timeout can fail an upgrade that would have succeeded. One of those jobs, the Keycloak password migration (see Upgrade halted: Keycloak passwords (argon2)), can take up to 20 minutes on its own: it waits up to 10 minutes for Keycloak to answer (auth.keycloak.fips.migration.waitSeconds: 600), then reads every user's credentials, about one admin request per user, twice. Its limit, auth.keycloak.fips.migration.activeDeadlineSeconds: 1200, covers about 10,000 users. For a larger Keycloak, raise that limit by about 60 seconds for each further 1,000 users, and raise --timeout by the same amount.

Your environment's install page has the exact command and any additions:

• Kubernetes with Istio
• OpenShift with GreyMatter. If you use the prerequisites chart, the platform administrator first reruns it with the new version. Then repeat step 3 with the new version. Publishing replaces the configuration files, so hand edits to them, such as an edge image, are lost; set them in your values file first, as in Keep site settings across publishes. Then rerun the install command. With Helm 4, keep --server-side=false on every upgrade.
• Install Without Internet Access. Use the chart in your own registry, with the values from step 5 of that page.

Always upgrade with helm upgrade --install against the cluster, never by applying rendered manifests. The chart reads its generated keys and passwords back from the cluster on each upgrade.

If the upgrade stops with missing cluster owner trust, nothing was changed: complete Set up owner trust and run the command again.

Upgrade with your values files, as above, and never with --reuse-values. With --reuse-values, Helm renders the new chart with the previous release's values, the previous chart's defaults included, in place of the new chart's defaults. A default that changed keeps its old value, a value the new chart adds has no default, and the new chart cannot tell those old defaults apart from values you set.

The upgrade restarts each Kamiwaza PostgreSQL pod once, to add the check-pgdata-ownership init container. Some of the upgrade's jobs, such as core-upgrade-migration and core-db-init-cleanup, are kept when they fail, for kubectl -n kamiwaza describe job <job>; the next upgrade replaces them. Most of the others, the Keycloak password migration among them, are removed an hour after they finish, so read a failed one's log within the hour.

Upgrade halted: PostgreSQL data directory

Before it changes anything, the upgrade compares where each Kamiwaza PostgreSQL StatefulSet (core-postgres, keycloak-postgres, core-spicedb-postgres) keeps its data with where the new release would look for it. When they differ, PostgreSQL would start on an empty data directory, so the upgrade stops:

UPGRADE HALTED (HR-9): StatefulSet core-postgres keeps its PostgreSQL data in volume directory "/" (...), but this render would use "/postgres" (...). Proceeding would start PostgreSQL on an empty data directory. ...

The message names the values that keep the current layout, for example core.postgresql.dataMount.mountPath, .subPath, and .pgData. Add them to your values file and run the upgrade again.

To move the data on purpose, for example to let SpiceDB's PostgreSQL start empty and rebuild from Kamiwaza's database, list that StatefulSet for one upgrade, then remove the entry and upgrade again:

global:
  postgres:
    allowDataDirectoryChange: [core-spicedb-postgres]

Only the StatefulSets you list skip the check. Move any data you need yourself first, and take the pg_dumpall backup from Before you start before you list a StatefulSet whose data you cannot recreate. The check needs the cluster, so helm template and dry runs on the client do not run it.

Upgrade halted: Context TLS preflight

From 1.4.0, the Context Service verifies TLS on its embedding and LLM calls to the platform's own origins, on upgraded installations too. When the edge certificate is minted outside the chart or by a CA it does not hold (OpenShift with GreyMatter; Istio or Envoy with network.tls.mode: byo, including EZUA, or customerCA, or cert-manager with a custom issuer, or generated when the chart does not generate the certificate) and nothing publishes its CA, those calls would fail after the upgrade. The upgrade stops first, before it changes anything, with:

Context TLS preflight: Context runtime model calls verify TLS by default since 1.4.0 (core.context.embedding.verifySsl: true), but the certificate for https://<domain> is minted outside this chart (provider=<provider>, network.tls.mode=<mode>) and nothing publishes global.trust.edgeAnchor.pem<settings that do not count>, so Core cannot verify its own origin and embedding and Context LLM calls would fail after this upgrade. Fix one of: (1) set global.trust.edgeAnchor.pem to the CA chain that signs <domain><CA hint>; (2) if that certificate chains to a public CA, set global.trust.edgeAnchor.publiclyTrusted: true (verification stays on); (3) to keep the pre-1.4.0 behavior without verification, set core.context.embedding.verifySsl: false (named in NOTES and at every Core start).

A hand-made kamiwaza-edge-trust ConfigMap that Helm does not manage satisfies this check on a real upgrade. If you move its CA into global.trust.edgeAnchor.pem, as step 6 of Before you start describes, delete the ConfigMap right before you rerun, or the upgrade stops with the already exists error instead.

The check runs when the chart renders, so helm template also fails with it, including the --show-only command in GreyMatter step 3, until your values carry a fix.

<settings that do not count> is empty, or one or more of these, in parentheses and separated by semicolons:

• global.trust.edgeAnchor.publishedBySubstrate is true, but no substrate-created ConfigMap kamiwaza-edge-trust is in this namespace
• global.trust.edgeAnchor.pem is set, but network.enabled=false, so this release does not publish it
• core.scheduler.extraEnv sets <name> from valueFrom, which this chart cannot read, so it counts as verifying; use a literal value

<CA hint> names the CA to publish:

• on GreyMatter: (on GreyMatter: the ca.crt of Secret greymatter-edge-ingress);
• with a custom cert-manager issuer: (cert-manager with issuer <issuer>: that issuer's CA);
• with customerCA: (customerCA: the customer CA; one added through the ca chart's trustBundle.customerCASecret is not visible to this chart, so put the same CA here);
• otherwise nothing.

Add one fix to your values file and run the upgrade again:

1. Recommended: set global.trust.edgeAnchor.pem to the CA chain that signs the domain: on GreyMatter, the ca.crt of Secret greymatter-edge-ingress; on EZUA, the EZUA gateway's CA; with customerCA, the customer CA; with a custom cert-manager issuer, that issuer's CA.
2. If the edge certificate chains to a public CA, set global.trust.edgeAnchor.publiclyTrusted: true. This never applies on GreyMatter.
3. Only until you can publish the CA, set core.context.embedding.verifySsl: false. This turns verification off, as before 1.4.0. The install notes and Core's startup log name it.

The fixes carry different risks. The full list of conditions, including the values that already satisfy the check, is in Edge certificates minted outside the chart.

Upgrade halted: Keycloak passwords (argon2)

From 1.4.0, Keycloak runs its FIPS build, which cannot verify argon2 password hashes, and every user created with a password on 1.3 has one (see Move local users' passwords off argon2). Before the upgrade's other jobs run, while the 1.3 Keycloak still serves, its kamiwaza-keycloak-fips-migration job sets hashAlgorithm(pbkdf2-sha512) and hashIterations(210000) in the password policy of the master and kamiwaza realms and of any other realm with an argon2 password, keeping the policy's other entries. It then signs in once as each account whose password the chart keeps in a Secret, which moves those accounts to pbkdf2. If any argon2 password is left, the upgrade stops with an error that names job kamiwaza-keycloak-fips-migration failed (the exact wording depends on your Helm version):

Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* job kamiwaza-keycloak-fips-migration failed: BackoffLimitExceeded

The Kamiwaza release you run keeps running, and none of its resources has changed. Only the realm hash policies and the chart's own accounts have moved to pbkdf2, which 1.3 handles as before and which the next attempt needs. The job's log, kept for an hour, names each account as <realm>/<username>:

kubectl -n kamiwaza logs job/kamiwaza-keycloak-fips-migration -c keycloak-fips-migration

ERROR: passwords keycloak-fips cannot verify remain: kamiwaza/<username> (argon2), kamiwaza/<username> (argon2). keycloak-fips cannot verify these passwords, so these accounts could not sign in after the upgrade. ...

Fix it in one of two ways:

1. Recommended: move the listed accounts, then run the same helm upgrade again. The job has already set the pbkdf2 policy, so have each listed user sign in to Kamiwaza once, or reset their password under Local Users in Settings. Then run the same upgrade command. To check first, use the query in step 3 of Move local users' passwords off argon2. If the list names one of the chart's own accounts, the job could not sign it in: its password no longer matches the one in its Secret (keycloak-admin for the master administrator, kamiwaza-user-<username> in the kamiwaza realm), or its realm requires a one-time code. Sign it in once through the browser, or reset its password to the one in its Secret.

2. Upgrade anyway and lock the listed accounts out. Add this to your values file and run the upgrade again:

   auth:
     keycloak:
       fips:
         allowArgon2Lockout: true

   The upgrade then continues, and the job's log lists the accounts with a WARNING instead. After the upgrade, those users cannot sign in at all, with any password, until an administrator resets their password under Local Users in Settings. Users cannot reset their own unless you have set up Keycloak's email password reset, which needs a mail server the chart does not configure. When the realm requires MFA (always under the stig profile), the chart turns email reset off, so only an administrator can reset these passwords. Plan the resets and tell the users before you choose this. It never covers the chart's own administrator in the master realm: if that account is listed, the upgrade still stops.

   Do not use this while the list names one of the chart's accounts in the kamiwaza realm, such as admin or svc-core. The setting locks those out too, including the Kamiwaza administrator who resets the other passwords. Fix those accounts with option 1 first.

   Keep allowArgon2Lockout: true in your values until every locked-out account has been reset. Keycloak then runs its FIPS build, so an upgrade without the setting finds the same accounts, can no longer move them, and stops again. Its error message then gives the longer recovery, which moves Keycloak back to its standard image for one upgrade.

Upgrade from the OpenShift overlay files

The kamiwaza-openshift.yaml and kamiwaza-openshift-greymatter.yaml values files are retired, and global.platform: osgm replaces them. With them in the values file list, the upgrade stops with INSTALLATION HALTED; see Install stops with a workload identity error.

1. Remove both files from your values file list, and set global.platform: osgm in your values file. Keep the chart's default direct extension runtime. The preset replaces the files' defaults; copy over only values your site chose for itself. One set the preset does not yet supply: kamiwaza-openshift.yaml paced GreyMatter route publishing, so copy these into your values file:

   global:
     greymatter:
       routeWriter:
         postPushSettleSeconds: 30
         retainDeletedRoutes: true
         startupFlushTimeoutSeconds: 120

2. If your PostgreSQL data uses the layout those files set, set core.postgresql.dataMount and auth.keycloak.postgresql.dataMount to it. SpiceDB's PostgreSQL (core-spicedb-postgres) follows core.postgresql.dataMount too. The PostgreSQL data directory check names the values when they are missing.

3. dataMount sets only the path. Data written under the files' fixed user ID, 70 or 65532, stays owned by it, and PostgreSQL refuses a data directory it does not own. Do not change the owner before the upgrade: the running PostgreSQL pods still run as the fixed user ID, and the upgrade's database jobs use them before the pods are replaced. Change it during the upgrade instead:

   1. Run the upgrade. As it replaces each PostgreSQL pod, the new pod stops at its check-pgdata-ownership init container before PostgreSQL starts, so the data is not touched. kubectl -n kamiwaza describe pod <pod> shows an RS-16 message with the volume's owner and the user ID the pod now runs as.
   2. Have the cluster administrator change the owner of that volume's data to the user ID in the message. The init container retries on its own, or delete the pod to retry at once, and PostgreSQL starts.

   Have the administrator ready before you start: the upgrade waits for the PostgreSQL pods only until its --timeout. If it times out first, change the owners, then run the same upgrade command again. See also Troubleshooting.

External model endpoints and tlsInsecure

From 1.4, core.scheduler.tlsInsecure: true turns off certificate verification only for the platform's own origins (with one OmniParse exception, see core.scheduler.tlsInsecure): the platform's domain, in-cluster services, and paired federation clusters. Core verifies the certificate of every external model host it calls, such as in Context Service and OmniParse model calls, as Kaizen already did on 1.3 except in its reranker, which now verifies too. The Context Service and OmniParse TLS settings, including OmniParse's force insecure model SSL and a request's vision model verify_ssl: false, no longer relax an external host either; only core.scheduler.externalTlsInsecureWithDnsRetargetingRisk does. A model server addressed by a private IP address or a short hostname, such as an on-premises server at https://10.0.0.5, counts as external.

If an external model endpoint has a certificate that Kamiwaza does not trust, and it worked on 1.3 because certificate verification was off, do one of these before you upgrade:

• Recommended: add the endpoint's root CA, and any issuing CA, to global.trust.extraAnchors. After the upgrade, restart Core's pods so they load it (kubectl -n kamiwaza rollout restart deployment/core-api deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker, plus deployment/core-retrieval-streamer if you enabled it) and redeploy the extensions that call the endpoint, such as Kaizen. If you created the kamiwaza-edge-trust ConfigMap by hand, first move it into values as described in Before you start. See TLS Trust for Private Certificate Authorities.

• For a trial only: list the host in core.scheduler.externalTlsInsecureWithDnsRetargetingRisk:

  core:
    scheduler:
      externalTlsInsecureWithDnsRetargetingRisk:
        - models.example.com

  List a fully qualified hostname or an IPv4 address. A short hostname or an IPv6 address fails the install; Core covers those only with true, and Kaizen never relaxes either (see the configuration reference). The setting never covers an OmniParse translation endpoint.

  This sends the API keys Kamiwaza holds for that provider to whoever answers DNS for the host. See externalTlsInsecureWithDnsRetargetingRisk. Redeploy the extensions that call the endpoint, such as Kaizen, so they pick it up, and again when you remove it.

If you upgrade without either, Core's and Kaizen's calls to that endpoint fail (an OmniParse instance not yet re-saved behaves as described below). For its own model calls, Core logs one error for each external host that fails verification, naming the host and both fixes, and, with tlsInsecure: true, one warning that it now covers only platform origins. OmniParse and Kaizen report their verification failures in their own logs.

After the upgrade, open and save each OmniParse instance that uses any external endpoint, including a translation endpoint, so Core rebuilds its pod-wide TLS setting. Add any external translation host's CA first. Until you do, its translation calls and document downloads keep the 1.3 setting, which may leave verification off for an external host.

Extension platform gateway

Kamiwaza 1.4 gives extensions one in-cluster platform address, KAMIWAZA_PLATFORM_GATEWAY_URL, on every routing provider. See Platform Gateway for Extensions.

Core needs KAMIWAZA_ORIGIN. On Kubernetes, Core refuses to start unless KAMIWAZA_ORIGIN is an http(s) URL with a host, and core-api goes into CrashLoopBackOff with:

FATAL: KAMIWAZA_ORIGIN is unset; extensions would be told the platform lives at https://localhost. Set global.domain (or core KAMIWAZA_ORIGIN) to the public https origin.

The chart always sets it from global.domain, so only a hand-edited Core configuration is affected. Set global.domain, or Core's KAMIWAZA_ORIGIN, to the public HTTPS origin.

A KAMIWAZA_INTERNAL_GATEWAY_URL override must be an origin. If Core has one with a path, such as https://edge/api, extension deploys are refused and GET /api/kamiwaza/install-modes reports invalid, with:

KAMIWAZA_INTERNAL_GATEWAY_URL=<value> must be an origin (scheme://host[:port]); Core appends /api itself, so this value yields /api/api/... and extensions get 404.

Remove the path. Such values already made extension calls fail with 404.

GreyMatter on a custom cluster domain. The GreyMatter edge host now follows the cluster's DNS domain instead of a fixed .svc.cluster.local. Chart installs need no certificate change: the chart sets KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME on Core to global.greymatter.edge.tlsServerName, or to global.domain when that is unset, and extensions verify the edge against that name. Without the chart, when that variable is unset, Core uses <edge-service>.<namespace>.svc.<domain> as KAMIWAZA_GATEWAY_TLS_SERVER_NAME, so the edge certificate must carry that name, or set the variable to a name it does carry. Do not set it blank: Core then withholds KAMIWAZA_PLATFORM_GATEWAY_URL and says why on the deployment status. Clusters on cluster.local see no change.

Envoy on a custom cluster domain. Core now points extensions at https://kamiwaza-envoy.<namespace>.svc.<domain>, including their KAMIWAZA_API_URL. Before 1.4 that host was fixed to .svc.cluster.local, which does not resolve on such a cluster. Set global.clusterDomain to the cluster's domain. A certificate you provide must carry that name. The chart reissues its generated certificate with the name, and restarts Envoy, Core and Keycloak to load the new edge anchor. Redeploy running extensions afterward. A client outside the cluster that pinned the old self-signed certificate must fetch the new one. See TLS names for the platform gateway.

Istio on a custom cluster domain. Extensions verify the gateway against the public host, which the gateway certificate already carries. Set global.clusterDomain so that the default gateway host uses the cluster's domain.

Istio. Extensions now reach the platform in-cluster through the ingress gateway instead of through the public origin. Most chart installs need no values change. With the extension operator (not supported on public installs; see Extension operator not supported on public installs), an upgrade can stop on the install identity's rights in the gateway namespace, on a missing gateway namespace or DestinationRule CRD, or on a serviceHostname form it no longer accepts, and a namespace-admin install needs a Role from a cluster administrator; see Extension operator on Istio.

Extension pods restart once. After Core is upgraded, it applies each extension deployment again with the new variables, and each extension pod restarts once. Plan for it; no action is needed.

Remove hand-set gateway overrides. Remove any TASKS_PLATFORM_GATEWAY_URL or other gateway override you set by hand that points at the address Kamiwaza now sets.

Extension operator not supported on public installs

The extension-operator runtime is not supported on any install of the public Kamiwaza 1.4.0 release, including mirrored and air-gapped installs. No public release has published an extension-operator image, so a public install with extensionOperator.enabled: true never had a running operator. From 1.4.0 the chart refuses that value when global.imageRegistryPrefix is under ghcr.io/kamiwaza-ai/releases/, and the upgrade stops before it changes anything. On a mirrored or air-gapped install the chart cannot tell that the images are the public release's, so it does not stop you: check your values yourself.

Before you upgrade, remove extensionOperator.enabled: true from your values, and core.extensionRuntime.mode: operator if you set it. Also remove any global.namespaces.extensions or global.namespaces.sandboxes override that names a namespace other than Kamiwaza's, and any core.clusterScopedRbac.create: true or core.clusterScopedRbac.extensionManagerMode override, which the direct runtime refuses. Extensions then run through the direct runtime, the chart default. See Extension operator layout.

Workroom Manager extension removed

Kamiwaza 1.4.0 removes the standalone Workroom Manager extension (catalog name workroom-manager), and the extension value of global.workroomManager.surface with it. Workrooms themselves are not affected: workroom records, the workroom APIs, permissions and isolation stay as they are, and the data a workroom held through the extension stays in Kamiwaza.

global.workroomManager.surface accepts:

Value	Result
absent, empty or disabled (default)	Workrooms are hidden from users.
core	The built-in Workrooms page serves /workrooms.
extension	The chart refuses to render, so the install or upgrade stops, on every ingress provider.

While your values set extension, the chart refuses to render, before the upgrade changes anything:

Error: execution error at (kamiwaza/charts/...): global.workroomManager.surface=extension is retired: the standalone Workroom Manager extension is no longer supported. Remove the key to keep the default (disabled) or set it to core for the built-in Workrooms page. ...

If you used the extension:

1. Find the setting. In the values files you upgrade with, look for workroomManager: under global:. On the running release, helm get values kamiwaza -n kamiwaza | grep -A1 workroomManager shows it.

2. Remove surface: extension, or change it to core if users should keep a Workrooms page.

3. Upgrade, as described in Upgrade.

4. Delete the standalone extension. After the upgrade it is hidden and cannot be reached, but its pods keep running until you delete it. Find its name with kubectl get kamiwazaextensions -A, then, as an administrator, delete it through the platform API, which runs the normal extension teardown:

   curl -X DELETE -H "Authorization: Bearer $TOKEN" \
     "https://<your Kamiwaza domain>/api/extensions/<extension name>"

   Do not delete the KamiwazaExtension resource with kubectl: it then waits on its finalizer and is not cleaned up.

A 1.4.0 Core that is still given extension, for example by an older chart, treats it as disabled and logs an error at startup. It does not switch to the built-in page, which you did not choose.

Connectors at internal addresses

From 1.4, Kamiwaza refuses connector endpoints that resolve to private, CGNAT or other non-public addresses unless you admit them. Loopback, link-local and cloud metadata addresses are refused even when admitted. See Connector Destinations.

Existing connectors keep working without a values change: on its first start with 1.4, Kamiwaza approves each existing connector's private host:port for that connector only, shown as Grandfathered on upgrade under Settings > Connectors. Connectors Kamiwaza deployed itself are admitted anyway. After the upgrade:

• Check Settings > Connectors > Blocked connector destinations. Allow a destination for its connector there, or re-register a connector whose endpoint is a floor address or an invalid URL. See Existing connectors on upgrade for what the upgrade does not grandfather.
• To register new connectors at internal hosts, list those hosts in core.connectors.endpointAllowlist, as narrowly as you can. If you set KAMIWAZA_CONNECTOR_ENDPOINT_ALLOWLIST in core.scheduler.extraEnv (or core.scheduler.linuxNvidia.extraEnv), move it to that value, one list item per comma-separated entry, and remove the extraEnv entry; setting both fails the render, and so does an item that contains a comma.

Extension runtime policy under stig

This applies only to installs with global.securityProfile: stig. From 1.4, an unset global.extensionRuntimeCsp.mode enforces the Content-Security-Policy on App Garden extension responses under stig; before, it was report-only. The upgrade changes this with no values change. See Extension runtime policy under stig.

Upgrade Kaizen first, to the Kaizen release in the 1.4 App Garden catalog, or later, which supports an enforced extension policy. Earlier Kaizen releases cannot show artifact previews under it. Check your other extensions before you upgrade: an extension's report-only policy will be enforced as written, and the platform's floor policy blocks cross-origin framing and cross-origin form targets even for an extension that sends its own policy. See Extension runtime policy under stig for what to check. On Istio, extension pods in ambient mode get no platform policy; record them against V-204760.

If your values set global.extensionRuntimeCsp.mode: report-only, the upgrade stops and names the setting:

global.securityProfile=stig refuses global.extensionRuntimeCsp.mode=report-only (rule: V-204760 (SRG-APP-000206, enforced Content-Security-Policy)). ...

Remove the setting, so the policy enforces, or keep report-only for now by also setting global.extensionRuntimeCsp.acknowledgeReportOnly: true. The install output then warns that V-204760 is not met.

Exempt paths on the standalone Envoy edge

This applies only to installs with global.ingress.provider: envoy. From 1.4, the Envoy edge builds a route for each /api/... entry of global.authz.extAuthz.exemptPaths, and the render fails, before the upgrade changes anything, on an entry the edge cannot serve. The shipped list passes; check entries you added. Separately, on every install type, if you use SAML or certificate login, add its paths to the list before you turn it on, keeping every shipped entry (see the next link). See Public paths at the edge.

Verify

• Every Kamiwaza deployment's pods are Ready, and the upgrade's jobs have completed. Helm can return before the pods are ready; wait with the command at the start of your install page's verify step.
• helm list -n kamiwaza shows the new chart version.
• The platform answers 200 at / and 401 at /api/ without credentials.
• GET /api/kamiwaza/metadata, called as a signed-in user, reports the new version.
• A model you deploy after the upgrade runs, as described in Tenant-mode inference, step 7.

After the upgrade

Do each action that the release notes list for the releases you crossed.

Deploy your models again. A model deployment created before tenant-mode inference was on carries no tenant resources, so it cannot start again on the new release. Stop each one and deploy it again; the new deployment takes its resources from the signed profile. The platform's embedding model may need only the stop: see step 6 of Tenant-mode inference.

Running extensions do not change. Each running extension deployment keeps the version it was deployed with. A selection changes only the version that new deployments use.

From 1.3.4, the App Garden moves older selections up to the release's versions. Each release ships a version of each extension it packages. When Kamiwaza 1.3.4 or later starts after the upgrade, every App Garden selection older than the version the release ships moves up to that version, including a version an administrator chose. A selection at or above that version does not change. A selection of an extension the release does not package, such as one published after the release, does not change either. An upgrade to 1.3.3 or earlier moves nothing.

To use a version newer than the release ships, as an administrator, open the App Garden and select Check for updates, then Update on a row, Update all, or Choose version.

If Kamiwaza cannot move a selection, it keeps the older one and records why. To check, use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token:

curl --fail --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/apps/catalog" \
  | jq '{moved: [.baseline_advance.advanced[]? | "\(.name) \(.from_version) -> \(.to_version)"],
         not_moved: [.baseline_advance.blocked[]? | {name, message}]}'

moved lists the selections Kamiwaza moved. not_moved should be empty. For each entry in it, follow its message, for example by selecting the release's version with Choose version.

A new Kaizen version is a new deployment. Each Kaizen deployment has its own database and storage volumes, and no data carries over from an existing deployment to a new one. Keep your current Kaizen deployment unless you are starting fresh. If you run Kaizen v3, see Kaizen v3 on Kamiwaza 1.4 as well.

What stopping a Kaizen deployment does to its storage volume claims depends on how you stop it. On an installation with the chart's default extension runtime:

• Stop in the App Garden's list of deployments deletes the deployment's storage volume claims and, with a Delete reclaim policy, the Kubernetes default for dynamically provisioned volumes, its data. For Kaizen v3 this is permanent: see Stop a Kaizen v3 deployment without losing it.
• Stopping it through the API, with DELETE /api/apps/deployment/<id>, keeps its storage volume claims and their data, and they stay bound to the same volumes through an upgrade. DELETE /api/apps/deployment/<id>/purge deletes them.

Back up before you stop a Kaizen deployment either way.

To run a new Kaizen version alongside the existing one:

1. Back up the existing deployment, as described in Before you start.
2. Make sure the cluster has room for a second complete Kaizen deployment. Without it, the new deployment's pods stay Pending and the deployment fails. Never stop the existing deployment to make room. If the new deployment fails before anyone has used it, or opens the Kamiwaza dashboard instead of Kaizen, add capacity if it was short, then stop that new, unused deployment and deploy again. Do not stop a new deployment that anyone has added data to; contact Kamiwaza support instead.
3. If the App Garden does not already select the new Kaizen version, select it, as described above.
4. Deploy Kaizen from the App Garden.
5. In the new deployment's Admin area, under Models, select the chat and embedding models. Upload your documents again, then choose Index on each one, or Index selected for several.

Interim: Kubernetes API addresses changed by the upgrade

Interim repair

This section applies until a Kamiwaza release updates existing extensions to the configured Kubernetes API addresses on its own. That release's notes say when this section no longer applies.

Skip this section if the upgrade did not change core.extensionRuntime.kubernetesApi.egressCidrs.

Kamiwaza writes each extension deployment's access to the Kubernetes API, a NetworkPolicy whose name ends in -kube-api, when it deploys the extension, from the addresses configured at that time. An upgrade that changes egressCidrs does not rewrite the policies of extensions that are already deployed. They keep the old addresses. An extension whose policy names an address that is no longer right cannot reach the API. In Kaizen v3, new conversations then fail with 502. Kaizen v3 cannot be deployed again on 1.4, so update each policy in place instead.

1. Read the addresses now configured, as a JSON list:

   CIDRS=$(helm get values kamiwaza -n kamiwaza -a -o json \
     | jq -c '.core.extensionRuntime.kubernetesApi.egressCidrs // []')
   echo "$CIDRS"
   # Expect the addresses you set, for example ["10.50.0.4/32","10.96.0.1/32"]

   If it prints [] or nothing, this repair does not apply: contact Kamiwaza support. Step 3 refuses an empty list, because a policy with no address allows every address. If you set core.extensionRuntime.kubernetesApi.serverUrl, a policy can have a second rule that this repair does not update: contact Kamiwaza support as well.

2. List Kamiwaza's Kubernetes API policies and the addresses each one allows. With a namespace-scoped identity, replace -A with -n <extension namespace> here and in step 3:

   kubectl get networkpolicy -A -l extensions.kamiwaza.io/managed-by=kamiwaza-platform -o json \
     | jq -r '.items[] | select(.metadata.name | endswith("-kube-api"))
       | "\(.metadata.namespace) \(.metadata.name) \([.spec.egress[0].to[].ipBlock.cidr] | join(","))"'

3. Set every policy's addresses to the configured ones. A policy that already lists them is unchanged. The patch changes only the API rule's addresses, and refuses a policy whose first rule is not the API rule on port 443:

   if ! printf '%s\n' "$CIDRS" | jq -e 'type == "array" and length > 0' > /dev/null 2>&1; then
     echo "egressCidrs is empty or unreadable: nothing patched"
   else
     TO=$(echo "$CIDRS" | jq -c '[.[] | {ipBlock: {cidr: .}}]')
     PATCH=$(jq -cn --argjson to "$TO" '[
       {op: "test", path: "/spec/egress/0/ports/0/port", value: 443},
       {op: "replace", path: "/spec/egress/0/to", value: $to}]')
     kubectl get networkpolicy -A -l extensions.kamiwaza.io/managed-by=kamiwaza-platform -o json \
       | jq -r '.items[] | select(.metadata.name | endswith("-kube-api"))
         | "\(.metadata.namespace) \(.metadata.name)"' |
     while read -r ns name; do
       kubectl -n "$ns" patch networkpolicy "$name" --type=json -p "$PATCH"
     done
   fi

   If a patch reports that the test failed, leave that policy as it is and contact Kamiwaza support.

4. Run step 2 again and check that every policy lists the configured addresses. Then open a new conversation in each running Kaizen v3 deployment and check that it starts.

Kaizen v3 on Kamiwaza 1.4

On Kamiwaza 1.3 the App Garden lists two Kaizen products. Kaizen (Legacy), catalog name kaizen-legacy, is Kaizen v3. Kaizen, catalog name kaizen, is Kaizen v4. Deployments made before 1.3 may run Kaizen v3 under another name, such as kaizen or kaizen-v3. Kamiwaza recognizes Kaizen v3 by its images as well as by the name kaizen-legacy, so this section applies to those deployments too.

Kamiwaza 1.4 stops installing Kaizen v3 and keeps the Kaizen v3 deployments you already have.

New Kaizen v3 deployments are refused

From 1.4, Kamiwaza refuses every request that would create a new Kaizen v3 deployment, before it creates anything. That covers the App Garden, the API, the SDK and the CLI, and every catalog, including imported, retained and local ones. Cloning a Kaizen v3 deployment, or creating another one beside it, is refused too: a Kaizen v3 deployment you already have does not allow a new one, in its workroom or anywhere else.

The API answers 409, with code set to product_retired and this message in detail:

Kaizen v3 (kaizen-legacy) is retired on Kamiwaza 1.4 and later: new installations are refused. Existing Kaizen v3 deployments are preserved and stay manageable. Install Kaizen v4 ('kaizen') as a separate deployment; v3 data and configuration are not migrated to it.

Deploying Kaizen (Legacy) is refused

After the upgrade, the App Garden can still show a Kaizen (Legacy) card with a Deploy button. Deploying it is refused with this message:

Kaizen v3 (kaizen-legacy) is retired on Kamiwaza 1.4 and later: new installations are refused. Existing Kaizen v3 deployments are preserved and stay manageable. Install Kaizen v4 ('kaizen') as a separate deployment; v3 data and configuration are not migrated to it.

This is expected, and nothing needs repairing. The refusal comes before anything is created, so it leaves no deployment, storage or route behind. Kaizen v3 deployments you already have are not affected.

To install Kaizen, deploy Kaizen v4, the Kaizen card (catalog name kaizen), as a separate deployment, as described in After the upgrade.

Existing Kaizen v3 deployments are kept

A Kaizen v3 deployment that exists when you upgrade, whether it is running, stopped or failed, keeps its ID, data, configuration, storage, route, owner and workroom. A running deployment keeps running through the upgrade and after it.

Restore a stopped or failed deployment with POST /api/apps/deployment/<id>/restore. The App Garden has no button for it.

• The deployment's owner or an administrator can restore it. An owner who is not an administrator gets 404 when the request is made from a different workroom than the deployment's.
• A restore brings back the same deployment, with the same ID, storage volume claims, stored secrets, route, owner and workroom. It is never a new installation.
• Only a stopped or failed deployment can be restored. A request for a running one is refused with 409 and code not_restorable.
• Kamiwaza answers 202 once the launch is scheduled. Follow its progress with GET /api/apps/deployment/<id>.
• A refused restore answers 409 and changes nothing; its code says why. A 503 means the previous instance is still being cleaned up: retry after the number of seconds in its Retry-After header. If the code is reconciliation_required or launch_compose_mismatch, contact Kamiwaza support.
• A 500 means the restore failed after it started. Check the deployment's status: it may still launch. If it is FAILED, restore it again.

A restore reattaches the storage volume claims the deployment still has. It cannot bring back data that was deleted. To stop a Kaizen v3 deployment and keep its data, stop it through the API, as described in Stop a Kaizen v3 deployment without losing it.

Kaizen v4 is a separate deployment. Deploy Kaizen v4, the Kaizen row (kaizen) in the App Garden, beside your Kaizen v3 deployment, as described in After the upgrade. It has its own identity, route and storage. It does not inherit Kaizen v3 agents, conversations, data or configuration: there is no in-place upgrade from Kaizen v3 to Kaizen v4, and no importer. Keep your Kaizen v3 deployment for as long as you need its history.

App Garden updates never replace Kaizen v3. Check for updates, Update, Update all and Choose version never delete, migrate or replace a Kaizen v3 deployment.

Deleting a Kaizen v3 deployment is final. DELETE /api/apps/deployment/<id>/purge deletes the deployment and its storage volume claims, and, with a Delete reclaim policy, their data. A purged deployment cannot be restored, and you cannot install a new Kaizen v3 deployment in its place. Stop in the App Garden's list of deployments deletes a Kaizen v3 deployment in the same way: see Stop a Kaizen v3 deployment without losing it. Back up first, as described in Before you start.

Stop a Kaizen v3 deployment without losing it

Never use Stop in the App Garden's list for Kaizen v3

Stop in the App Garden's list of deployments calls DELETE /api/extensions/<name>. That deletes the Kaizen v3 deployment and its storage volume claims and, with a Delete reclaim policy, the Kubernetes default for dynamically provisioned volumes, their data. The deployment cannot be restored afterwards: a restore request answers 404. Kamiwaza 1.4 cannot install Kaizen v3 again, so the deployment is gone for good.

Stop a Kaizen v3 deployment through the API instead. That keeps the deployment and its storage volume claims, so you can restore it in place later. These commands use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token.

1. Back up the deployment, as described in Back up and restore a Kaizen v3 deployment.

2. Find the deployment's ID, its id in GET /api/apps/deployments.

3. Stop the deployment:

   ID=<deployment ID>
   curl --fail --silent --show-error --request DELETE \
     --header "Authorization: Bearer $ADMIN_TOKEN" \
     "https://kamiwaza.example.com/api/apps/deployment/$ID"

   The deployment's storage volume claims and their data stay, bound to the same volumes.

4. To bring the deployment back, restore it in place:

   curl --fail --silent --show-error --request POST \
     --header "Authorization: Bearer $ADMIN_TOKEN" \
     "https://kamiwaza.example.com/api/apps/deployment/$ID/restore"

   Kamiwaza answers 202 once the launch is scheduled. Follow its progress with GET /api/apps/deployment/<id> until it reports DEPLOYED. The rules for a restore, and what each refusal means, are in Existing Kaizen v3 deployments are kept.

Do not add /purge to the stop request: DELETE /api/apps/deployment/<id>/purge deletes the deployment and its storage volume claims, and cannot be undone.

Back up and restore a Kaizen v3 deployment

A Kaizen v3 deployment keeps its data in two places: its own PostgreSQL database, openhands_ui, and its backend storage volume, mounted at /tmp/kaizen/storage. A backup is a copy of both.

• You can restore a backup only into the same deployment it was taken from, with the same deployment ID. Its stored secrets are encrypted with a key that belongs to that deployment, so another deployment cannot read them. Kamiwaza does not install new Kaizen v3 deployments, so there is no other deployment to restore into.
• You can restore a backup taken on 1.3 after you upgrade to 1.4. The Kaizen v3 images do not change in the upgrade.
• Files in a conversation's agent workspace are not part of the backup.

These commands are for an installation with the chart's default extension runtime. On another extension runtime, contact Kamiwaza support. If the deployment sets POSTGRES_USER or POSTGRES_DB to something other than the defaults, postgres and openhands_ui, use its values.

Find the deployment's ID, its id in GET /api/apps/deployments. Then find its namespace and its postgres and backend pods:

ID=<deployment ID>
SEL="kamiwaza.io/deployment-id=$ID"
NS=$(kubectl get pods -A -l "$SEL" -o jsonpath='{.items[0].metadata.namespace}')
PG=$(kubectl -n "$NS" get pods -l "$SEL,extensions.kamiwaza.io/service=postgres" \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
BE=$(kubectl -n "$NS" get pods -l "$SEL,extensions.kamiwaza.io/service=backend" \
  --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}')
echo "$NS $PG $BE"

All three must be set. If kubectl prints an error instead, fix your access first; with a namespace-scoped identity, use kubectl -n <extension namespace> get pods -l "$SEL" and set NS to that namespace. If there is no error and a value is empty, the deployment is not running: see Restore.

Then check that the backend keeps its storage at /tmp/kaizen/storage. This command prints the backend's mount paths, and they must include it:

kubectl -n "$NS" get pod "$BE" \
  -o jsonpath='{.spec.containers[?(@.name=="backend")].volumeMounts[*].mountPath}'

If it does not, the deployment runs an older Kaizen v3 layout that these commands do not cover: contact Kamiwaza support.

Back up

Ask users to finish their conversations, then run:

kubectl -n "$NS" exec "$PG" -c postgres -- \
  pg_dump -U postgres -Fc openhands_ui > "kaizen-$ID-db.dump"
kubectl -n "$NS" exec "$BE" -c backend -- \
  tar -C /tmp/kaizen/storage -czf - . > "kaizen-$ID-storage.tgz"

Do not add -t to these commands: it corrupts the files. Keep both files together, labelled with the deployment ID and the time you took them, and store them where you keep your other Kamiwaza backups.

Restore

A restore replaces everything the deployment stored after the backup.

1. Check that the deployment ID is the one in the backup's file names.

2. If the deployment is failed, or was stopped through the API, restore it in place first, as described in Existing Kaizen v3 deployments are kept, and wait until GET /api/apps/deployment/<id> reports DEPLOYED. Then set NS, PG and BE again.

3. Check that both backup files are complete before you delete anything. The database check restores the dump into a scratch database, so the database volume needs room for a second copy of the data. Every command must finish without an error; if one fails, stop here, because the next step deletes the deployment's current data:

   tar -tzf "kaizen-$ID-storage.tgz" > /dev/null
   kubectl -n "$NS" exec "$PG" -c postgres -- \
     psql -U postgres -d postgres -c 'DROP DATABASE IF EXISTS openhands_ui_check'
   kubectl -n "$NS" exec "$PG" -c postgres -- \
     psql -U postgres -d postgres -c 'CREATE DATABASE openhands_ui_check'
   kubectl -n "$NS" exec -i "$PG" -c postgres -- \
     pg_restore -U postgres -d openhands_ui_check --exit-on-error < "kaizen-$ID-db.dump"
   kubectl -n "$NS" exec "$PG" -c postgres -- \
     psql -U postgres -d postgres -c 'DROP DATABASE openhands_ui_check'

   If the restore fails, drop openhands_ui_check with the last command before you stop.

4. Make sure nobody is using the deployment, then load the backup. The commands run in a subshell that stops at the first command that fails:

   (
     set -e
     kubectl -n "$NS" exec "$PG" -c postgres -- \
       psql -U postgres -d postgres -c 'DROP DATABASE IF EXISTS openhands_ui WITH (FORCE)'
     kubectl -n "$NS" exec "$PG" -c postgres -- \
       psql -U postgres -d postgres -c 'CREATE DATABASE openhands_ui'
     kubectl -n "$NS" exec -i "$PG" -c postgres -- \
       pg_restore -U postgres -d openhands_ui --exit-on-error < "kaizen-$ID-db.dump"
     kubectl -n "$NS" exec "$BE" -c backend -- \
       sh -c 'rm -rf /tmp/kaizen/storage/* /tmp/kaizen/storage/.[!.]* /tmp/kaizen/storage/..?*'
     kubectl -n "$NS" exec -i "$BE" -c backend -- \
       tar -C /tmp/kaizen/storage -xzf - < "kaizen-$ID-storage.tgz"
     kubectl -n "$NS" delete pod "$BE"
   )

   If a command fails, nothing after it runs. Running this step again starts the load over. If the backend pod was already deleted, wait until the new one is Ready and set PG and BE again first. If it fails a second time, contact Kamiwaza support.

5. When the new backend pod is Ready, open Kaizen. Check that your conversations and uploaded files are back, and that a new conversation starts.

Keep the Kaizen v3 images

A Kaizen v3 deployment keeps pulling images after the upgrade. Kubernetes pulls them again when it re-creates the deployment's pods, for example after a restart or a move to another node, and each new conversation starts an agent sandbox from its own image. A 1.4 release does not include Kaizen v3 images.

An installation that pulls App Garden images from Kamiwaza's public release registry, and upgrades with the published chart and its existing values, needs nothing more for Kaizen v3 deployments made on Kamiwaza 1.3: the images of the Kaizen v3 versions that 1.3 shipped stay published there. On an air-gapped installation, the images must stay in your registry. Before you upgrade one that runs Kaizen v3:

1. Keep every image your registry received from a Kamiwaza release before 1.4 for as long as you run Kaizen v3. Kaizen v3 images have had several names over its releases, and a stopped or failed deployment runs no pods that would show which ones it needs, yet a restore pulls them again. Keep each image at the same path, tag and digest, and do not remove the previous releases' copies after the upgrade. A tag does not protect an image from every kind of registry cleanup: before you garbage-collect or prune the registry, read Clean up the registry safely.

2. Check the running deployments. These commands list the images that running Kaizen v3 deployments use, and the agent sandbox image that each new conversation starts. Confirm that each one is in your registry:

   # Every Kaizen and PostgreSQL image a running pod uses
   kubectl get pods -A -o json | jq -r '
     .items[] | .spec.containers[], (.spec.initContainers // [])[] | .image' |
     grep -iE 'kaizen|postgres' | sort -u
   
   # The agent sandbox image each new conversation starts
   kubectl get deploy -A -o json | jq -r '
     .items[] | .spec.template.spec.containers[] | .env // [] | .[] |
     select(.name == "AGENT_SERVER_IMAGE") | .value' | sort -u

   The list can include Kaizen v4 and platform images as well.

3. If you are moving to a new registry, copy every image from step 1 there first, at the same path, with every platform and the same digest, as in step 3 of Install Without Internet Access.

After the upgrade, open a new conversation in each running Kaizen v3 deployment. It pulls the agent sandbox image, so a missing image shows up there. This check does not cover a stopped or failed deployment: its images are pulled only when you restore it.

Clean up the registry safely

Registry cleanup can break an image that still has its tag. Every Kamiwaza image is a multi-platform image: an index that names one manifest for each platform. Only the index has a tag. A cleanup that removes the platform manifests leaves the tag and the index in place, so the registry looks intact. Pods that are already running keep running from the copy on their node, so nothing fails at first. The failure comes later, when a pod starts on a node that has no copy: after a reschedule, on a new or rebuilt node, and in every new Kaizen v3 conversation, which pulls the agent sandbox image. This applies to the running release's images as much as to Kaizen v3's.

The commands below use crane and jq, and the SOURCE and REGISTRY values from Install Without Internet Access. If your registry requires credentials, log in with crane auth login first. Deleting from a registry:2 registry also needs its delete support turned on (storage.delete.enabled: true). Without it, every delete answers 405.

Garbage collection

• Do not run registry garbage-collect --delete-untagged on a registry built on distribution 2.x, such as the registry:2 image. Platform manifests have no tags of their own, so it deletes the platform manifests of every multi-platform image in the registry. Run registry garbage-collect without --delete-untagged, or use a registry whose garbage collection follows image indexes, such as Harbor or distribution 3.x.
• Without the flag, garbage collection frees only what you delete first. It deletes content, such as layers, that no manifest in the registry still names. It never deletes a manifest itself. Deleting only an image's tag or index frees almost nothing, because its platform manifests stay and keep their layers. To free an image's space, delete its index and its platform manifests, as in Delete an earlier release's image.
• Stop pushes during garbage collection, and restart the registry after it. Run it with the registry in read-only mode, or stopped. Then restart the registry before you copy anything into it. registry:2 caches which content it holds, and until it restarts it can accept a copy of deleted content without storing it. The copy reports success, and the image then fails to pull.

Decide what to keep

Write the images that must stay to keep-refs.txt, one reference in your registry per line. Keep at least:

• The new release's images and the images of the release you upgraded from, each from its own images.txt (step 1 of Install Without Internet Access). The release you upgraded from stays until every model and extension deployment runs on the new release and you no longer need to restore the backup, as in Before you start.
• If the installation runs Kaizen v3: the images of every release before 1.4 that you copied into the registry, as Keep the Kaizen v3 images requires. Stopped and failed Kaizen v3 deployments run no pods, but a restore pulls their images.

Use each release's full images.txt, not a list you filtered in step 1, and pick the images you delete from the same lists. If you no longer have the image list of a release you must keep, delete nothing from the registry.

Start with an empty list, then add each image list, rewritten to your registry:

: > keep-refs.txt

sed "s#^$SOURCE/#$REGISTRY/kamiwaza-ai/releases/kamiwaza/#" images.txt >> keep-refs.txt

Then add every image from your registry that a pod runs, and the agent sandbox image that Kaizen v3 deployments start. A platform administrator runs this, because it lists pods and deployments in every namespace. If it cannot, it removes keep-refs.txt, and nothing can be deleted:

if kubectl get pods -A -o json > pods.json && kubectl get deploy -A -o json > deploys.json; then
  jq -r '.items[] | .spec.containers[], (.spec.initContainers // [])[] | .image' pods.json |
    grep "^$REGISTRY/" | sort -u >> keep-refs.txt
  jq -r '.items[] | .spec.template.spec.containers[] | .env // [] | .[] |
    select(.name == "AGENT_SERVER_IMAGE") | .value' deploys.json |
    grep "^$REGISTRY/" | sort -u >> keep-refs.txt
else
  rm -f keep-refs.txt
  echo "Cannot list pods and deployments: keep-refs.txt removed, delete nothing"
fi
rm -f pods.json deploys.json

A workload with no running pod, such as one scaled to zero, is not listed. Add its images, and any image you copied into the registry yourself, to keep-refs.txt by hand.

Then record every digest those images use, indexes and platform manifests both. The command writes keep-digests.txt only if it can read every image. If it prints Cannot read for an image, run the check and repair the registry before you delete anything:

rm -f keep-digests.txt
ok=1
[ -s keep-refs.txt ] || { echo "keep-refs.txt is missing or empty"; ok=0; }
: > keep-digests.new
while read -r ref || [ -n "$ref" ]; do
  [ -z "$ref" ] && continue
  if index=$(crane manifest "$ref") && digest=$(crane digest "$ref"); then
    { echo "$digest"; printf '%s\n' "$index" | jq -r '.manifests[]?.digest'; } >> keep-digests.new
  else
    echo "Cannot read $ref"; ok=0
  fi
done < keep-refs.txt
if [ "$ok" = 1 ]; then
  sort -u keep-digests.new > keep-digests.txt
  echo "keep-digests.txt: $(wc -l < keep-digests.txt) digests"
fi
rm -f keep-digests.new

Delete an earlier release's image

Deleting a tag deletes the image under every tag. The registry deletes a manifest by its digest, so a tool that deletes a tag deletes the digest the tag names. Every other tag in that repository that names the same digest goes with it. When an image is unchanged between two releases, deleting the old release's tag, for example envoy:release-1.3.4, also deletes the new release's tag, envoy:release-1.3.5.

Delete earlier releases' images only with this command. It refuses an image that an image you keep uses, and deletes the index and every platform manifest that no kept image uses. Set old to the image, in your registry, by tag, replacing NAME:TAG. For an image listed by digest only, as in lists before 1.3.1, use NAME@sha256:… instead:

old=$REGISTRY/kamiwaza-ai/releases/kamiwaza/images/NAME:TAG
repo=${old%@*}
case ${repo##*/} in *:*) repo=${repo%:*} ;; esac
if [ ! -s keep-digests.txt ]; then
  echo "Build keep-digests.txt first: nothing deleted"
elif ! index=$(crane manifest "$old") || ! digest=$(crane digest "$old"); then
  echo "Cannot read $old: nothing deleted"
elif grep -qxF "$digest" keep-digests.txt; then
  echo "Keep $old: an image you keep uses $digest"
elif ! crane delete "$repo@$digest"; then
  echo "Could not delete $old: nothing deleted"
else
  failed=""
  for d in $(printf '%s\n' "$index" | jq -r '.manifests[]?.digest'); do
    grep -qxF "$d" keep-digests.txt || crane delete "$repo@$d" || failed="$failed $repo@$d"
  done
  if [ -z "$failed" ]; then
    echo "Deleted $old ($digest)"
  else
    echo "Deleted $old ($digest), but not:$failed"
  fi
fi

Each run prints exactly one result, and only a Deleted result deleted anything. Deleted …, but not: lists platform manifests that are left over: delete each one with crane delete <reference>. The command protects only the images in your keep list: a platform manifest shared with an image you left out of it is deleted, so build the list completely. When you have deleted what you need to, run garbage collection without --delete-untagged, restart the registry, and run the check.

Check the registry after every cleanup

Check that each image in keep-refs.txt still resolves, with each of its linux platform manifests, and that each tag still names the digest it should. The check reads manifests and does not download layers. An image copied in after garbage collection and before the registry restarted may show only as a bare MISSING line: restart the registry and copy it again.

missing=0
while read -r ref || [ -n "$ref" ]; do
  [ -z "$ref" ] && continue
  name=${ref%@*}
  repo=$name
  case ${repo##*/} in *:*) repo=${repo%:*} ;; esac
  if [ "$name" != "$ref" ] && [ "$name" != "$repo" ] &&
     [ "$(crane digest "$name" 2>/dev/null)" != "${ref##*@}" ]; then
    echo "MISSING tag $name"; missing=$((missing + 1))
  fi
  if ! index=$(crane manifest "$ref" 2>/dev/null); then
    echo "MISSING $ref"; missing=$((missing + 1)); continue
  fi
  for d in $(printf '%s\n' "$index" | jq -r '.manifests[]? | select(.platform.os == "linux") | .digest'); do
    crane manifest "$repo@$d" > /dev/null 2>&1 \
      || { echo "MISSING $repo@$d, a platform of $ref"; missing=$((missing + 1)); }
  done
done < keep-refs.txt
echo "missing: $missing"
# Expect "missing: 0"

Repair a damaged image

If the check lists anything missing, restart the registry first if garbage collection ran since its last restart. Then:

1. For each MISSING …, a platform of <reference> line, delete the damaged index, replacing REFERENCE with the reference after a platform of. Copying over a damaged image does not repair it: some copy tools, such as crane, find the index already there and skip it, and others, such as oras, fail with manifest blob unknown. Deleting the index first works with each of them:

   ref=REFERENCE
   repo=${ref%@*}
   case ${repo##*/} in *:*) repo=${repo%:*} ;; esac
   digest=$(crane digest "$ref") && crane delete "$repo@$digest" && echo "Deleted $repo@$digest"

   A bare MISSING <reference> or a MISSING tag line needs no delete.

2. Copy the images again as in step 3 of Install Without Internet Access: rerun that step's copy for every release whose image list names a damaged image. Deleting an index removes every tag in its repository that names it, and an image unchanged between releases is in more than one release's list. A 1.4 release does not include Kaizen v3 images, so copy those from the earlier release that shipped them.

3. Run the check again. If you no longer have a release's files or access to its registry, contact Kamiwaza support.

No rollback to 1.3

Kamiwaza 1.4.0 cannot be rolled back to a 1.3 release in place, 1.3.4 included. The 1.4 upgrade moves Kamiwaza's database to a schema that no 1.3 release can run. If you need to return to the release you ran before the upgrade, the supported way is to restore the backup you took in Before you start: contact Kamiwaza support, who restore the platform from those pg_dumpall files. A Kaizen v3 deployment is restored from its own backup, as described in Back up and restore a Kaizen v3 deployment.

Do not try to roll back with Helm:

• Do not run helm upgrade --install with --version 1.3.4, or any other 1.3 version, against a 1.4 installation. The 1.3 chart rolls some of Kamiwaza's workloads back to 1.3 before its database job refuses the 1.4 schema. Core then cannot start, and the platform API stays down.
• Do not use helm rollback, and do not uninstall Kamiwaza. By default, uninstalling deletes the platform databases, the object store, and every extension's data.

If a 1.3 upgrade command has already run against a 1.4 installation, stop there. Do not run further Helm commands, and do not rerun the 1.4 upgrade as a way to recover: after a failed rollback it can complete without errors while Kamiwaza is not using your data. Keep the backup from Before you start and the release history (helm history kamiwaza -n kamiwaza), and contact Kamiwaza support.

The release notes of each later 1.4 release say whether this still applies.

If you upgrade from 1.3.3 or earlier

Kamiwaza 1.3.4 made the changes below. They apply when you upgrade from 1.3.3 or earlier.

A custom-named default Global Milvus

From 1.3.4, only the Milvus VectorDB named milvus-global-workroom is the default for the Global Workroom. In 1.3.3 the default was the newest Milvus VectorDB in the Global Workroom, whatever its name. If an administrator created a Global Milvus with another name after milvus-global-workroom, or instead of it, that instance received every user's Global ingest.

On an affected installation, after the upgrade:

• The next Global ingest creates a new, empty milvus-global-workroom, unless one already exists. Global ingest and search use it.
• The custom instance's collections stay where they are, but no longer appear in default Global search or listings.
• The custom instance becomes an ordinary Global VectorDB owned by the administrator who created it. That administrator and other administrators can still reach it through the VectorDB API.

Check before you upgrade. As an administrator, list the Global Workroom's VectorDBs:

curl -sS -H "Authorization: Bearer $TOKEN" \
  -H "X-Workroom-ID: ffffffff-ffff-ffff-ffff-ffffffffffff" \
  "$KAMIWAZA_URL/api/context/vectordbs" |
  jq 'sort_by(.created_at) | .[] | {id, name, engine, created_at}'

The rows are listed oldest first. The installation is affected if the last row with engine milvus is not named milvus-global-workroom.

If you are affected, choose one:

1. Ingest again (recommended). Upgrade, then run ingestion again for the Global datasets and documents. The data lands in the new default. Then delete the custom instance if nothing else uses it.

2. Adopt the custom instance. This works only before any Global ingest runs on 1.3.4, while no milvus-global-workroom row exists. Rename the custom row. With the bundled PostgreSQL, in Kamiwaza's namespace:

   kubectl exec -i core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza <<'SQL'
   UPDATE context_vectordb_instances
      SET name = 'milvus-global-workroom'
    WHERE id = '<custom instance id>'
      AND NOT EXISTS (
        SELECT 1 FROM context_vectordb_instances
         WHERE name = 'milvus-global-workroom'
      );
   SQL

   UPDATE 1 means the row was renamed. UPDATE 0 means a milvus-global-workroom row already exists, so adopting no longer works; ingest again instead. If you renamed it after the upgrade, restart Core, and also deployment/core-api if your installation runs it:

   kubectl -n kamiwaza rollout restart \
     deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker

   On startup, Kamiwaza releases the instance's owner, and it becomes the shared default with its existing collections.

More than one tenant. In 1.3.3, the first user to ingest into the Global Workroom became the owner of milvus-global-workroom. 1.3.4 releases that owner on startup, and removes the owner's access records only in the Global Workroom's tenant, or the default tenant. That covers every single-tenant installation. If other tenants exist, Kamiwaza logs:

Canonical Global VectorDB owner release cleaned relation tuples in tenant ... only; N other tenant(s) exist.

A record left in another tenant grants nothing on the shared VectorDB. To keep the access records clean, remove it with /api/authz/resources/context_vectordb/grants.

App Garden apps no longer receive the sign-in cookies

From 1.3.4, on Kubernetes with Istio and on OpenShift with GreyMatter, an App Garden app no longer receives the Kamiwaza sign-in cookies (access_token, access_token_refresh, and the rest of that family) with each request. An app still receives its own cookies, unchanged. In earlier releases, any app a user opened could read that user's session and call Kamiwaza as that user, including as an administrator.

On OpenShift with GreyMatter, a Tool Shed tool also no longer receives any cookies, an Authorization header, or an X-App-Session-Token header, as on Kubernetes with Istio. Nothing in this section changes that for a tool. The usual upgrade applies this change, and the trust and enforce settings below; on OpenShift with GreyMatter, that includes repeating step 3 of the install page.

Apps that call Kamiwaza as the signed-in user with the sign-in cookies stop working unless an administrator trusts them:

• Kaizen and Kaizen (Legacy) are trusted when an administrator deployed them from the App Garden on Kamiwaza 1.3.3 or later. A deployment created on an earlier release, or by a user who is not an administrator, is not trusted. Its model list is empty, chat reports that no chat models are available, and documents still upload but are not indexed for semantic search. Other features that call Kamiwaza as the user, such as tasks, are affected the same way.
• Skills Library, Connector Builder, and your own apps that pass the user's cookies to Kamiwaza are not trusted by default.

To trust a deployment, add its runtime identifier, the path segment after /runtime/apps/ in the app's address, to core.auth.runtimeCookies.trustedRuntimeTargets in your values file, then rerun the upgrade command. Listing entries replaces the default list, so keep the two template: entries:

core:
  auth:
    runtimeCookies:
      trustedRuntimeTargets:
        - template:kaizen
        - template:kaizen-legacy
        - kaizen-1a2b3c4d   # an existing Kaizen deployment, by its identifier

A trusted app receives the session of every user who opens it, administrators included. Trust only apps you would trust with that session. Do not use wildcards or name prefixes: users choose deployment names.

For an existing Kaizen deployment, prefer its identifier to a redeploy. A new Kaizen deployment that an administrator deploys from the App Garden is trusted without an entry, but it does not carry over the existing deployment's data. See After the upgrade.

If an app you depend on breaks and you cannot trust it yet, set core.auth.runtimeCookies.enforce: false to restore the earlier behavior for every app. Tools still receive no cookies, Authorization header, or X-App-Session-Token header. This setting is temporary and will be removed in a later release.

The upgrade does not revoke anything an app could have obtained before it, such as an API token it created as a user. Administrators cannot see other users' API tokens. If you ran apps you do not trust, ask your users to open Settings, review the API Tokens on the My Settings tab, and revoke any they do not recognize.

Rotate the ForwardAuth signing secret

On Kubernetes with Istio and on OpenShift with GreyMatter, a user who could deploy an App Garden app or tool before 1.3.4 could read Kamiwaza's ForwardAuth signing secret through the API. Kamiwaza uses that secret to sign and verify the identity it passes to its own components. From 1.3.4 the API no longer returns it, but the secret itself is unchanged. Any signed-in user can deploy an App Garden app, so replace the secret after you upgrade, unless only administrators you trust could deploy apps and tools. Installations on Envoy ingress without Istio do not need this.

Do it after the upgrade, not before: until 1.3.4 is running, the new value could be read the same way.

What to expect. Users stay signed in. While the Ray head restarts, the console, the API, and app and model routes are unavailable, and jobs running on the Ray head are interrupted. Choose a quiet time.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

1. Optional: find the deployments that stored the secret. List App Garden deployments and look for the value Kamiwaza now withholds:

   for kind in app tool service; do
     curl --fail --silent --show-error \
       --header "Authorization: Bearer $ADMIN_TOKEN" \
       "https://kamiwaza.example.com/api/apps/deployments?workroom_id=all&template_type=$kind" \
       | jq -r '.[] | select(tostring | contains("platform ForwardAuth signing secret"))
                | "\(.id) \(.name) owner=\(.owner_user_id)"'
   done

   Each line is a deployment whose environment holds the secret, so its owner could read it. Some deployments that held it are not listed: one that no longer exists, and one that stored it under a name Kamiwaza treats as a credential, such as a name ending in _TOKEN, _SECRET, _PASSWORD, or _API_KEY, because the response leaves such variables out. So an empty result does not show that nobody read the secret.

2. Replace the value. This writes a new random value into the forwardauth-signature Secret without printing it:

   kubectl -n kamiwaza patch secret forwardauth-signature --type merge \
     --patch-file <(printf '{"data":{"secret":"%s"}}' \
       "$(openssl rand -base64 32 | tr -d '\n' | base64 | tr -d '\n')")

   To confirm the change without printing the value, run kubectl -n kamiwaza get secret forwardauth-signature -o jsonpath='{.data.secret}' | openssl dgst -sha256 before and after: the hash changes. Later upgrades keep the new value, because the chart reads the Secret back from the cluster.

   Kamiwaza signs and verifies with AUTH_FORWARD_HEADER_SECRET, which the chart reads from this Secret, and passes FORWARDAUTH_SIGNATURE_SECRET to apps when it is set. If your values files set either variable, for example in core.scheduler.extraEnv, that entry takes precedence over the Secret. Its value is the one that was exposed, and replacing the Secret does not replace it. To find such entries, run this command. It prints Deployment and variable names, never values:

   kubectl -n kamiwaza get deployments -o json \
     | jq -r '.items[] | .metadata.name as $d
         | .spec.template.spec.containers[] | (.env // [])
         | select(map(.valueFrom.secretKeyRef.name) | index("forwardauth-signature"))
         | .[] | select(.name | test("^(AUTH_FORWARD_HEADER_SECRET|FORWARDAUTH_SIGNATURE_SECRET)$"))
         | select(.valueFrom.secretKeyRef.name != "forwardauth-signature")
         | "\($d) \(.name)"'

   If it prints nothing, go on to step 3. Otherwise, remove each listed variable from your values files, so that Kamiwaza uses the Secret you just replaced. If you must keep AUTH_FORWARD_HEADER_SECRET there, give it a new random value of at least 32 characters instead, for example from openssl rand -base64 32. Then rerun the upgrade command from your environment's install page instead of step 3: the upgrade restarts the same workloads.

3. Restart every Kamiwaza workload that reads the secret, together:

   kubectl -n kamiwaza get deployments -o json \
     | jq -r '.items[]
         | select([.spec.template.spec.containers[].env[]?.valueFrom.secretKeyRef.name]
                  | index("forwardauth-signature"))
         | "deployment/" + .metadata.name' > forwardauth-readers.txt
   cat forwardauth-readers.txt
   xargs kubectl -n kamiwaza rollout restart < forwardauth-readers.txt
   xargs -n 1 kubectl -n kamiwaza rollout status --timeout 15m < forwardauth-readers.txt

   The list is core-scheduler, core-raycluster-head, the Ray worker Deployment (core-raycluster-worker), and core-api if you enabled it.

4. Verify. The pods of each listed Deployment are Ready and were created after step 2. Sign in to the console, open an App Garden app, and send a request to a model deployment. If you removed variables in step 2, its command now prints nothing.

5. OpenShift with GreyMatter: redeploy each diffusion (image generation) model deployment. Until it is redeployed, it refuses requests.

Running App Garden apps keep working and need no action. An app of your own that verifies Kamiwaza's identity signature itself rejects requests until you redeploy it.

Users disabled or deleted before the upgrade

From 1.3.4, disabling or deleting a user who is linked to a Keycloak account also disables that account, ends its sessions, and revokes its API tokens. Users you disabled or deleted on an earlier release are refused by Kamiwaza once 1.3.4 runs, but their Keycloak accounts stay enabled until you repeat the change. Repeat it for each of them. A user you disabled in Keycloak itself needs nothing.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

1. Disabled users. List them:

   curl --fail --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
     "https://kamiwaza.example.com/api/auth/users/" \
     | jq -r '.[] | select(.linked_subject_id != null and .active == false)
              | "\(.id) \(.username)"'

   For each <id>, disable the user again:

   curl --silent --show-error --request PUT \
     --header "Authorization: Bearer $ADMIN_TOKEN" \
     --header 'Content-Type: application/json' --data '{"active": false}' \
     --write-out ' %{http_code}\n' \
     "https://kamiwaza.example.com/api/auth/users/<id>"

2. Deleted users. The API does not list them, so read their IDs from the database. This query changes nothing:

   kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
     psql -U core -d kamiwaza -At -c "
       SELECT id, username FROM users
       WHERE deleted AND linked_subject_id IS NOT NULL
         AND linked_subject_id NOT IN (
           SELECT linked_subject_id FROM users
           WHERE NOT deleted AND linked_subject_id IS NOT NULL)"

   Each line is <id>|<username>. For each <id>, delete the user again:

   curl --silent --show-error --request DELETE \
     --header "Authorization: Bearer $ADMIN_TOKEN" \
     --write-out ' %{http_code}\n' \
     "https://kamiwaza.example.com/api/auth/users/<id>"

3. Check the responses. Each request prints the user and then the HTTP status. 200 means the Keycloak account is now disabled, or no longer exists. A 503 names the step that failed; repeat the same request until it returns 200.

Extension credentials from before 1.3.4

From 1.3.4, a new App Garden deployment keeps the passwords, tokens, and API keys in its environment in its own Kubernetes Secret, and App Garden deployment responses no longer return them. A deployment created before 1.3.4 still holds them as plain environment variables, in its pods and in its ConfigMap, until it is removed. Anyone who can read pods or ConfigMaps in Kamiwaza's namespace can read them there, including with the built-in view role, which cannot read Secrets.

For the extensions in the App Garden catalog, these values are the catalog's published defaults, such as dev, postgres, and minioadmin, unless an administrator entered a different value when deploying. A published default is not a secret, and a new deployment gets the same default, so these need nothing. Act only on a value that you or another administrator entered when deploying an extension before 1.3.4, such as an API key, a token, or a password.

Do not stop or replace a Kaizen or Kaizen (Legacy) deployment for this. A new deployment starts empty, and stopping a deployment in the App Garden deletes its data.

1. List the names each deployment from before 1.3.4 holds in plain text. The command prints names only, never values:

   kubectl -n kamiwaza get configmap -l kamiwaza.io/extension-runtime=direct -o json \
     | jq -r '.items[]
         | .metadata.labels as $l
         | [ (.metadata.annotations["extensions.kamiwaza.io/rendered-spec"] // "{}" | fromjson)
             | .spec.services[]?.env[]?
             | select((.value // "") != "")
             | select((.name | test("(^|_)(PASSWORD|PASSWD|PASSPHRASE|SECRET|TOKEN|APIKEY|CREDENTIALS?)$|(^|_)(API|SECRET|PRIVATE|ACCESS|ENCRYPTION|SIGNING|MASTER)_KEY$"; "i"))
                      or (.value | tostring | test("://[^/?#@ ]*:[^/?# ]+@")))
             | .name ] | unique | select(length > 0)
         | "\($l["kamiwaza.io/deployment-id"] // "-") \($l["extensions.kamiwaza.io/deployment-id"]): \(join(" "))"'

   Each line names a deployment by its ID and its runtime, then the variables it holds in plain text. A deployment created on 1.3.4 or later does not appear.

2. For each variable whose value an administrator entered, rotate the credential where it was issued: for example, revoke the API key at its provider and create a new one. If you used the same password anywhere else, change it there too. Until the deployment has the new value, whatever in it uses the credential stops working.

3. Check whether the deployment keeps data, using the runtime from step 1:

   kubectl -n kamiwaza get pvc -l extensions.kamiwaza.io/deployment-id=<runtime>

   • No volume claims, and not Kaizen or Kaizen (Legacy): deploy the extension again from the App Garden with the new value, check that the new deployment works, then stop the old deployment in the App Garden. Stopping it there also removes its plain-text copy.
   • Volume claims, or Kaizen or Kaizen (Legacy): do not stop or replace the deployment. Contact Kamiwaza support.

4. A deployment that was stopped through the API (DELETE /api/apps/deployment/<id>), rather than in the App Garden, keeps its ConfigMap and its volumes, so it still appears in step 1. When you no longer need its data, remove it with DELETE /api/apps/deployment/<id>/purge, which deletes its volumes too. <id> is the ID that step 1 prints first.

Check: run step 1 again. The line of each deployment you replaced or purged is gone, and the new deployments do not appear. A deployment that step 3 leaves to Kamiwaza support is still listed.

Workroom context settings removed by the upgrade

From 1.3.4, only administrators can set the settings of an ontology or VectorDB instance, and the settings of an OmniParse instance other than its model selection, enable, timeout, and processing settings. Before 1.3.4, any member of a workroom could set them, including where an instance connects and with what credentials. The upgrade removes such settings from OmniParse, ontology, and VectorDB instances in a workroom other than the Global Workroom, and from the record each instance is deployed from, including values an administrator set: Kamiwaza cannot tell who set them. Instances in the Global Workroom do not change.

• OmniParse: every setting is removed except the model selection, enable, timeout, and processing settings.
• Ontology and VectorDB: every setting whose name marks an endpoint, URL, or host, a credential, a TLS or authentication setting, a proxy, a trust store, an identity, or a loader path is removed. Settings such as model names, timeouts, and sizes are kept, and a VectorDB keeps the credentials Kamiwaza generated for its own Milvus.
• Graphiti endpoints: Kamiwaza restores an ontology instance's chat and embedding endpoints from its chat and embedding models on the next ontology operation, which can return a 503 while the instance restarts; retry it. If Kamiwaza cannot restore an endpoint, the operation returns a 503 with the code pre_gate_endpoint_scrubbed, and an administrator selects the instance's chat and embedding models again, as in step 4 below.

Running pods. The first time Core starts after the upgrade, it also removes these settings from the running pods of the instances the upgrade changed. On each such pod, Core changes only the removed settings and any setting built from one of them. Each one gets the value the instance would be deployed with now, or is removed if it would not be set. For an OmniParse instance, Core first restores the settings Kamiwaza writes itself, such as its model endpoints and the Kamiwaza API address. The affected pods of each changed instance restart once, and its data volumes are kept. Other instances do not restart. A stopped or failed instance has no running pod to change; if it is started again, it starts without these settings. An ontology instance's chat and embedding endpoints are restored as described above.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

1. Check what Core did. Core's scheduler logs each instance it changed, and a summary. The lines name instances and settings, never their values, and each can appear twice, once as JSON:

   kubectl -n kamiwaza logs deployment/core-scheduler -c core \
     | grep -e 'scrubbed context instance' -e 'scrubbed settings' \
       -e 'left the pod of' -e 'changed meanwhile' -e 'on its running pod'

   • Startup: removed scrubbed settings <settings> from the running pod of <type> instance <instance ID>; its pods restart once: Core removed the settings from that instance's pod.

   • Startup: scrubbed context instance pod env reconcile {'found': …, 'updated': …, 'unchanged': …, 'skipped': …, 'errors': …}: the summary, logged at every start. At a later start, it counts only the instances still listed. errors should be 0, and skipped should count only instances whose deployment is not running, described next. If no summary line appears, restart the scheduler as shown below. If it still does not appear, contact Kamiwaza support.

   • Startup: left the pod of <type> instance <instance ID> for the next start: its deployment is not running: the instance is stopped or failed, so it has no running pod to change. Core has already removed the settings from what the instance is deployed from, so if it is started again, it starts without them. Core logs this line, and counts the instance in skipped, at every start until the instance runs again or is deleted. At the first start after it runs again, Core checks its pod and stops listing it. No action is needed, unless the removed settings included a password for data the instance already holds: it would start without that password and could not open that data, so contact Kamiwaza support before you start it.

   • Startup: left the pod of <type> instance <instance ID> for the next start: <reason> with any other reason, or a line that says Core could not remove scrubbed settings from an instance's pod: the instance was deploying, stopping, or being deleted, or the change failed. Core tries again at its next start. To try again now, wait until the instance has finished deploying or stopping, then restart the scheduler and check the log again:

     kubectl -n kamiwaza rollout restart deployment/core-scheduler
     kubectl -n kamiwaza rollout status deployment/core-scheduler --timeout 15m

     If Core leaves the instance again for the same reason, contact Kamiwaza support.

   • Startup: the pod env record of <type> instance <instance ID> changed meanwhile; it is reconciled on the next start: the instance's record changed while Core worked on it, so Core checks the instance again at its next start. The summary does not count this in skipped. To check now, restart the scheduler as shown above.

   • Startup: <type> instance <instance ID> keeps <settings> on its running pod, logged as an error: Core leaves some settings on the pod, such as identity and credential settings. A new password, for example, would lock an instance out of data created with the old one. The value stays on the running pod, and Core logs that line again at every start. If the value is one an administrator set or Kamiwaza generated, you can leave it on the pod; otherwise, contact Kamiwaza support. Core does not keep the value for the instance's next deployment, though. If it is a password that protects data the instance already created, stopping and starting the instance would leave it unable to open that data. Do not stop that instance, and contact Kamiwaza support if it must keep the value.

2. List the instances the upgrade could change. This query changes nothing. It prints each instance's type, workroom ID, instance ID, and name, never its settings:

   kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
     psql -U core -d kamiwaza -At -c "
       SELECT 'omniparse', workroom_id, id, name FROM context_omniparse_instances
       WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
       UNION ALL
       SELECT 'ontology', workroom_id, id, name FROM context_ontology_instances
       WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
       UNION ALL
       SELECT 'vectordb', workroom_id, id, name FROM context_vectordb_instances
       WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
       ORDER BY 1, 2, 4"

   Each line is <type>|<workroom ID>|<instance ID>|<name>.

3. Find the values the upgrade removed. Compare against your own records of what administrators set, or read an instance's settings from before the upgrade out of the backup you took in Before you start. This command changes nothing. It prints the instance's row from the backup. The instance's settings are the row's JSON object, credentials included, so run it where no one else can see the output:

   kubectl -n kamiwaza exec -i core-postgres-0 -c postgres -- \
     pg_restore --data-only --table=context_omniparse_instances --file=- \
     < kamiwaza-before-upgrade.dump \
     | grep '^<instance ID>'

   For an ontology or VectorDB instance, use the table context_ontology_instances or context_vectordb_instances. Optionally, cross-check the upgrade's own record: the database job's log names each instance the upgrade removed settings from, with its workroom and the names of the removed settings, never their values. By default the job and its log are removed about an hour after the job finishes:

   kubectl -n kamiwaza logs -l app.kubernetes.io/name=core-db-init -c db-init --tail=-1 \
     | grep -e 'removed privileged' -e 'config not an object'

   A line that says config not an object names an instance whose settings Kamiwaza could not read. Core has no record of what the upgrade removed from it, so the values may stay on its running pod. Save it as below, or delete it. If its values must leave the running pod, contact Kamiwaza support.

4. Re-enter the values an administrator set. Save an instance only to re-enter values. A save stores them in the instance's settings and the record it is deployed from. It does not change the instance's running pod: if a value you re-enter must reach the running pod, contact Kamiwaza support. Each request prints the HTTP status; 200 means the instance was saved.

   • OmniParse: put the values in a file, so they stay off the command line, as {"config": {"<setting>": "<value>"}}. Then send it with the instance's workroom ID:

     curl --silent --show-error --request PUT \
       --header "Authorization: Bearer $ADMIN_TOKEN" \
       --header "X-Workroom-ID: <workroom ID>" \
       --header 'Content-Type: application/json' \
       --data @omniparse-settings.json \
       --write-out ' %{http_code}\n' \
       "https://kamiwaza.example.com/api/context/omniparses/<instance ID>"

   • VectorDB: send them the same way to /api/context/vectordbs/<instance ID>.

   • Ontology: Kamiwaza 1.3 has no request to set an ontology instance's settings. Kamiwaza restores the chat and embedding endpoints itself, as described above. If an operation returns pre_gate_endpoint_scrubbed, select the instance's models again with PUT /api/context/ontologies/<instance ID>/model-bindings and the body {"chat": {"deployment_id": "<chat model deployment ID>"}, "embedding": {"deployment_id": "<embedding model deployment ID>"}}. To restore another setting the upgrade removed from an ontology instance, contact Kamiwaza support.