Skip to main content

Upgrade Kamiwaza

Within a release line, upgrade Kamiwaza in place: rerun the helm upgrade --install command from your environment's install page with the new --version and the same values files. An in-place upgrade keeps Kamiwaza's models, workrooms, users, data, and storage volumes, and existing extension deployments keep running.

Supported upgrades​

You can upgrade in place from any 1.3 release to a later 1.3 release. There is no in-place upgrade from Kamiwaza 1.2 or earlier: contact your Kamiwaza representative to plan an assisted migration to a new installation.

Before you start​

  1. Read the release notes for every release after the one you run, up to and including the one you are installing. Each patch release lists any action it requires. See Release notes.

  2. Back up. Back up Kamiwaza's PostgreSQL database and, for each Kaizen deployment, its database and its object store. For Kamiwaza's database, this writes a dump to a file on your machine:

    kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
      pg_dump -U core -d kamiwaza --format=custom > kamiwaza-before-upgrade.dump

    The dump holds everything in the database, so keep it as securely as the database itself. For each Kaizen deployment, run pg_dumpall in its PostgreSQL pod, and archive the /data directory of its SeaweedFS pod. Restoring a Kaizen backup into a new deployment is not documented; contact Kamiwaza support if you need one.

    From 1.3.4, keep the Kamiwaza database backup until you are sure you will stay on the new release. After an upgrade to 1.3.4 or later, Core from 1.3.3 or earlier does not start on the upgraded database, so a helm rollback to an earlier release leaves Kamiwaza down. To go back, restore this backup and install the earlier release; contact Kamiwaza support for help with the restore.

  3. Find your values files, and the order you installed with.

  4. Air-gapped installations: copy the new release into your registry, as described in Install Without Internet Access. Keep the previous release's engine images in your registry until every model deployment has been redeployed on the new release, and its extension images while deployments of those extension versions still run.

  5. Kubernetes with Istio installations: if your values files set global.authz.extAuthz.serviceHost, it must name core-api.<namespace>.svc.<cluster domain> in Kamiwaza's namespace, for example core-api.kamiwaza.svc.cluster.local. From 1.3.3, any other value, such as core-api.kamiwaza.svc without the cluster domain, makes helm upgrade fail before it changes anything, with an error that the host does not resolve into release namespace "<namespace>". Remove the value from your values files, or correct it. Do not turn off the access check that the error message names instead: that removes authentication from model or tool routes.

  6. Execution or dataset gates that use your own class: from 1.3.4, Kamiwaza loads a gate class only from its own gate modules, from an installed gate package, or from a package listed in AUTH_GATE_CLASSPATH_PREFIXES. After the upgrade, an execution gate whose class is not allowed makes every job submission fail, local ones included, and a dataset whose gate class is not allowed is not served. Check both before you upgrade, as an administrator:

    • The execution gate: call GET /api/cluster/execution-gate.
    • Each gated dataset, one whose properties include gate or gate_type: call GET /api/catalog/datasets/<URL-encoded URN>/gate. The response names the dataset's gate class.

    A 403 with reason classpath_not_allowed means the class is not allowed. List its package in your values file before you upgrade, as described in Allow your gate's classpath.

Do not uninstall Kamiwaza before an upgrade. Uninstalling deletes the platform databases, the object store, and every extension's data.

Upgrade​

As the identity that installed Kamiwaza, rerun the helm upgrade --install command from your environment's install page, with the new --version and every values file you installed with, in the same order. For a Kubernetes with Istio installation with one values file:

helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <new version> \
  --namespace kamiwaza \
  --values kamiwaza-values.yaml \
  --timeout 70m

Your environment's install page has the exact command and any additions:

  • Kubernetes with Istio
  • OpenShift with GreyMatter. First repeat step 3 with the new version, then rerun the upgrade command. With Helm 4, keep --server-side=false on every upgrade.
  • Install Without Internet Access. Use the chart in your own registry, with the values from step 5 of that page. If you installed with the field kit, use the new release's field kit: render its values again and follow its README. Values rendered by an earlier kit refer to the earlier release's images.

Always upgrade with helm upgrade --install against the cluster, never by applying rendered manifests. The chart reads its generated keys and passwords back from the cluster on each upgrade.

Verify​

  • Every Kamiwaza deployment's pods are Ready, and the upgrade's jobs have completed. Helm can return before the pods are ready; wait with the command at the start of your install page's verify step.
  • helm list -n kamiwaza shows the new chart version.
  • The platform answers 200 at / and 401 at /api/ without credentials.
  • From 1.3.3, GET /api/kamiwaza/metadata, called as a signed-in user, reports the new version.

After the upgrade​

Do each action that the release notes list for the releases you crossed, such as redeploying models.

Running extensions do not change. Each running extension deployment keeps the version it was deployed with. A selection changes only the version that new deployments use.

From 1.3.4, the App Garden moves older selections up to the release's versions. Each release ships a version of each extension it packages. When Kamiwaza 1.3.4 or later starts after the upgrade, every App Garden selection older than the version the release ships moves up to that version, including a version an administrator chose. A selection at or above that version does not change. A selection of an extension the release does not package, such as one published after the release, does not change either. An upgrade to 1.3.3 or earlier moves nothing.

To use a version newer than the release ships, as an administrator, open the App Garden and select Check for updates, then Update on a row, Update all, or Choose version.

If Kamiwaza cannot move a selection, it keeps the older one and records why. To check, use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token:

curl --fail --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/apps/catalog" \
  | jq '{moved: [.baseline_advance.advanced[]? | "\(.name) \(.from_version) -> \(.to_version)"],
         not_moved: [.baseline_advance.blocked[]? | {name, message}]}'

moved lists the selections Kamiwaza moved. not_moved should be empty. For each entry in it, follow its message, for example by selecting the release's version with Choose version.

A new Kaizen version is a new deployment. Each Kaizen deployment has its own database and storage volumes, and no data carries over from an existing deployment to a new one. Stopping a Kaizen deployment in the App Garden deletes its storage volume claims, and with a Delete reclaim policy, the Kubernetes default for dynamically provisioned volumes, its data. Keep your current Kaizen deployment unless you are starting fresh. To run a new Kaizen version alongside it:

  1. Back up the existing deployment, as described in Before you start.
  2. Make sure the cluster has room for a second complete Kaizen deployment. Without it, the new deployment's pods stay Pending and the deployment fails. Never stop the existing deployment to make room. If the new deployment fails before anyone has used it, or opens the Kamiwaza dashboard instead of Kaizen, add capacity if it was short, then stop that new, unused deployment and deploy again. Do not stop a new deployment that anyone has added data to; contact Kamiwaza support instead.
  3. If the App Garden does not already select the new Kaizen version, select it, as described above.
  4. Deploy Kaizen from the App Garden.
  5. In the new deployment's Admin area, under Models, select the chat and embedding models. Upload your documents again, then choose Index on each one, or Index selected for several.

A custom-named default Global Milvus​

From 1.3.4, only the Milvus VectorDB named milvus-global-workroom is the default for the Global Workroom. In 1.3.3 the default was the newest Milvus VectorDB in the Global Workroom, whatever its name. If an administrator created a Global Milvus with another name after milvus-global-workroom, or instead of it, that instance received every user's Global ingest.

On an affected installation, after the upgrade:

  • The next Global ingest creates a new, empty milvus-global-workroom, unless one already exists. Global ingest and search use it.
  • The custom instance's collections stay where they are, but no longer appear in default Global search or listings.
  • The custom instance becomes an ordinary Global VectorDB owned by the administrator who created it. That administrator and other administrators can still reach it through the VectorDB API.

Check before you upgrade. As an administrator, list the Global Workroom's VectorDBs:

curl -sS -H "Authorization: Bearer $TOKEN" \
  -H "X-Workroom-ID: ffffffff-ffff-ffff-ffff-ffffffffffff" \
  "$KAMIWAZA_URL/api/context/vectordbs" |
  jq 'sort_by(.created_at) | .[] | {id, name, engine, created_at}'

The rows are listed oldest first. The installation is affected if the last row with engine milvus is not named milvus-global-workroom.

If you are affected, choose one:

  1. Ingest again (recommended). Upgrade, then run ingestion again for the Global datasets and documents. The data lands in the new default. Then delete the custom instance if nothing else uses it.

  2. Adopt the custom instance. This works only before any Global ingest runs on 1.3.4, while no milvus-global-workroom row exists. Rename the custom row. With the bundled PostgreSQL, in Kamiwaza's namespace:

    kubectl exec -i core-postgres-0 -n kamiwaza -- psql -U core -d kamiwaza <<'SQL'
    UPDATE context_vectordb_instances
       SET name = 'milvus-global-workroom'
     WHERE id = '<custom instance id>'
       AND NOT EXISTS (
         SELECT 1 FROM context_vectordb_instances
          WHERE name = 'milvus-global-workroom'
       );
    SQL

    UPDATE 1 means the row was renamed. UPDATE 0 means a milvus-global-workroom row already exists, so adopting no longer works; ingest again instead. If you renamed it after the upgrade, restart Core, and also deployment/core-api if your installation runs it:

    kubectl -n kamiwaza rollout restart \
      deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker

    On startup, Kamiwaza releases the instance's owner, and it becomes the shared default with its existing collections.

More than one tenant. In 1.3.3, the first user to ingest into the Global Workroom became the owner of milvus-global-workroom. 1.3.4 releases that owner on startup, and removes the owner's access records only in the Global Workroom's tenant, or the default tenant. That covers every single-tenant installation. If other tenants exist, Kamiwaza logs:

Canonical Global VectorDB owner release cleaned relation tuples in tenant ... only; N other tenant(s) exist.

A record left in another tenant grants nothing on the shared VectorDB. To keep the access records clean, remove it with /api/authz/resources/context_vectordb/grants.

App Garden apps no longer receive the sign-in cookies​

From 1.3.4, on Kubernetes with Istio and on OpenShift with GreyMatter, an App Garden app no longer receives the Kamiwaza sign-in cookies (access_token, access_token_refresh, and the rest of that family) with each request. An app still receives its own cookies, unchanged. In earlier releases, any app a user opened could read that user's session and call Kamiwaza as that user, including as an administrator.

On OpenShift with GreyMatter, a Tool Shed tool also no longer receives any cookies, an Authorization header, or an X-App-Session-Token header, as on Kubernetes with Istio. Nothing in this section changes that for a tool. The usual upgrade applies this change, and the trust and enforce settings below; on OpenShift with GreyMatter, that includes repeating step 3 of the install page.

Apps that call Kamiwaza as the signed-in user with the sign-in cookies stop working unless an administrator trusts them:

  • Kaizen and Kaizen (Legacy) are trusted when an administrator deployed them from the App Garden on Kamiwaza 1.3.3 or later. A deployment created on an earlier release, or by a user who is not an administrator, is not trusted. Its model list is empty, chat reports that no chat models are available, and documents still upload but are not indexed for semantic search. Other features that call Kamiwaza as the user, such as tasks, are affected the same way.
  • Skills Library, Connector Builder, and your own apps that pass the user's cookies to Kamiwaza are not trusted by default.

To trust a deployment, add its runtime identifier, the path segment after /runtime/apps/ in the app's address, to core.auth.runtimeCookies.trustedRuntimeTargets in your values file, then rerun the upgrade command. Listing entries replaces the default list, so keep the two template: entries:

core:
  auth:
    runtimeCookies:
      trustedRuntimeTargets:
        - template:kaizen
        - template:kaizen-legacy
        - kaizen-1a2b3c4d   # an existing Kaizen deployment, by its identifier

A trusted app receives the session of every user who opens it, administrators included. Trust only apps you would trust with that session. Do not use wildcards or name prefixes: users choose deployment names.

For an existing Kaizen deployment, prefer its identifier to a redeploy. A new Kaizen deployment that an administrator deploys from the App Garden is trusted without an entry, but it does not carry over the existing deployment's data. See After the upgrade.

If an app you depend on breaks and you cannot trust it yet, set core.auth.runtimeCookies.enforce: false to restore the earlier behavior for every app. Tools still receive no cookies, Authorization header, or X-App-Session-Token header. This setting is temporary and will be removed in a later release.

The upgrade does not revoke anything an app could have obtained before it, such as an API token it created as a user. Administrators cannot see other users' API tokens. If you ran apps you do not trust, ask your users to open Settings, review the API Tokens on the My Settings tab, and revoke any they do not recognize.

Rotate the ForwardAuth signing secret​

On Kubernetes with Istio and on OpenShift with GreyMatter, a user who could deploy an App Garden app or tool before 1.3.4 could read Kamiwaza's ForwardAuth signing secret through the API. Kamiwaza uses that secret to sign and verify the identity it passes to its own components. From 1.3.4 the API no longer returns it, but the secret itself is unchanged. Any signed-in user can deploy an App Garden app, so replace the secret after you upgrade, unless only administrators you trust could deploy apps and tools. Installations on Envoy ingress without Istio do not need this.

Do it after the upgrade, not before: until 1.3.4 is running, the new value could be read the same way.

What to expect. Users stay signed in. While the Ray head restarts, the console, the API, and app and model routes are unavailable, and jobs running on the Ray head are interrupted. Choose a quiet time.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

  1. Optional: find the deployments that stored the secret. List App Garden deployments and look for the value Kamiwaza now withholds:

    for kind in app tool service; do
      curl --fail --silent --show-error \
        --header "Authorization: Bearer $ADMIN_TOKEN" \
        "https://kamiwaza.example.com/api/apps/deployments?workroom_id=all&template_type=$kind" \
        | jq -r '.[] | select(tostring | contains("platform ForwardAuth signing secret"))
                 | "\(.id) \(.name) owner=\(.owner_user_id)"'
    done

    Each line is a deployment whose environment holds the secret, so its owner could read it. Some deployments that held it are not listed: one that no longer exists, and one that stored it under a name Kamiwaza treats as a credential, such as a name ending in _TOKEN, _SECRET, _PASSWORD, or _API_KEY, because the response leaves such variables out. So an empty result does not show that nobody read the secret.

  2. Replace the value. This writes a new random value into the forwardauth-signature Secret without printing it:

    kubectl -n kamiwaza patch secret forwardauth-signature --type merge \
      --patch-file <(printf '{"data":{"secret":"%s"}}' \
        "$(openssl rand -base64 32 | tr -d '\n' | base64 | tr -d '\n')")

    To confirm the change without printing the value, run kubectl -n kamiwaza get secret forwardauth-signature -o jsonpath='{.data.secret}' | openssl dgst -sha256 before and after: the hash changes. Later upgrades keep the new value, because the chart reads the Secret back from the cluster.

    Kamiwaza signs and verifies with AUTH_FORWARD_HEADER_SECRET, which the chart reads from this Secret, and passes FORWARDAUTH_SIGNATURE_SECRET to apps when it is set. If your values files set either variable, for example in core.scheduler.extraEnv, that entry takes precedence over the Secret. Its value is the one that was exposed, and replacing the Secret does not replace it. To find such entries, run this command. It prints Deployment and variable names, never values:

    kubectl -n kamiwaza get deployments -o json \
      | jq -r '.items[] | .metadata.name as $d
          | .spec.template.spec.containers[] | (.env // [])
          | select(map(.valueFrom.secretKeyRef.name) | index("forwardauth-signature"))
          | .[] | select(.name | test("^(AUTH_FORWARD_HEADER_SECRET|FORWARDAUTH_SIGNATURE_SECRET)$"))
          | select(.valueFrom.secretKeyRef.name != "forwardauth-signature")
          | "\($d) \(.name)"'

    If it prints nothing, go on to step 3. Otherwise, remove each listed variable from your values files, so that Kamiwaza uses the Secret you just replaced. If you must keep AUTH_FORWARD_HEADER_SECRET there, give it a new random value of at least 32 characters instead, for example from openssl rand -base64 32. Then rerun the upgrade command from your environment's install page instead of step 3: the upgrade restarts the same workloads.

  3. Restart every Kamiwaza workload that reads the secret, together:

    kubectl -n kamiwaza get deployments -o json \
      | jq -r '.items[]
          | select([.spec.template.spec.containers[].env[]?.valueFrom.secretKeyRef.name]
                   | index("forwardauth-signature"))
          | "deployment/" + .metadata.name' > forwardauth-readers.txt
    cat forwardauth-readers.txt
    xargs kubectl -n kamiwaza rollout restart < forwardauth-readers.txt
    xargs -n 1 kubectl -n kamiwaza rollout status --timeout 15m < forwardauth-readers.txt

    The list is core-scheduler, core-raycluster-head, the Ray worker Deployment (core-raycluster-worker), and core-api if you enabled it.

  4. Verify. The pods of each listed Deployment are Ready and were created after step 2. Sign in to the console, open an App Garden app, and send a request to a model deployment. If you removed variables in step 2, its command now prints nothing.

  5. OpenShift with GreyMatter: redeploy each diffusion (image generation) model deployment. Until it is redeployed, it refuses requests.

Running App Garden apps keep working and need no action. An app of your own that verifies Kamiwaza's identity signature itself rejects requests until you redeploy it.

Users disabled or deleted before the upgrade​

From 1.3.4, disabling or deleting a user who is linked to a Keycloak account also disables that account, ends its sessions, and revokes its API tokens. Users you disabled or deleted on an earlier release are refused by Kamiwaza once 1.3.4 runs, but their Keycloak accounts stay enabled until you repeat the change. Repeat it for each of them. A user you disabled in Keycloak itself needs nothing.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

  1. Disabled users. List them:

    curl --fail --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
      "https://kamiwaza.example.com/api/auth/users/" \
      | jq -r '.[] | select(.linked_subject_id != null and .active == false)
               | "\(.id) \(.username)"'

    For each <id>, disable the user again:

    curl --silent --show-error --request PUT \
      --header "Authorization: Bearer $ADMIN_TOKEN" \
      --header 'Content-Type: application/json' --data '{"active": false}' \
      --write-out ' %{http_code}\n' \
      "https://kamiwaza.example.com/api/auth/users/<id>"

  2. Deleted users. The API does not list them, so read their IDs from the database. This query changes nothing:

    kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
      psql -U core -d kamiwaza -At -c "
        SELECT id, username FROM users
        WHERE deleted AND linked_subject_id IS NOT NULL
          AND linked_subject_id NOT IN (
            SELECT linked_subject_id FROM users
            WHERE NOT deleted AND linked_subject_id IS NOT NULL)"

    Each line is <id>|<username>. For each <id>, delete the user again:

    curl --silent --show-error --request DELETE \
      --header "Authorization: Bearer $ADMIN_TOKEN" \
      --write-out ' %{http_code}\n' \
      "https://kamiwaza.example.com/api/auth/users/<id>"

  3. Check the responses. Each request prints the user and then the HTTP status. 200 means the Keycloak account is now disabled, or no longer exists. A 503 names the step that failed; repeat the same request until it returns 200.

Extension credentials from before 1.3.4​

From 1.3.4, a new App Garden deployment keeps the passwords, tokens, and API keys in its environment in its own Kubernetes Secret, and App Garden deployment responses no longer return them. A deployment created before 1.3.4 still holds them as plain environment variables, in its pods and in its ConfigMap, until it is removed. Anyone who can read pods or ConfigMaps in Kamiwaza's namespace can read them there, including with the built-in view role, which cannot read Secrets.

For the extensions in the App Garden catalog, these values are the catalog's published defaults, such as dev, postgres, and minioadmin, unless an administrator entered a different value when deploying. A published default is not a secret, and a new deployment gets the same default, so these need nothing. Act only on a value that you or another administrator entered when deploying an extension before 1.3.4, such as an API key, a token, or a password.

Do not stop or replace a Kaizen or Kaizen (Legacy) deployment for this. A new deployment starts empty, and stopping a deployment in the App Garden deletes its data.

  1. List the names each deployment from before 1.3.4 holds in plain text. The command prints names only, never values:

    kubectl -n kamiwaza get configmap -l kamiwaza.io/extension-runtime=direct -o json \
      | jq -r '.items[]
          | .metadata.labels as $l
          | [ (.metadata.annotations["extensions.kamiwaza.io/rendered-spec"] // "{}" | fromjson)
              | .spec.services[]?.env[]?
              | select((.value // "") != "")
              | select((.name | test("(^|_)(PASSWORD|PASSWD|PASSPHRASE|SECRET|TOKEN|APIKEY|CREDENTIALS?)$|(^|_)(API|SECRET|PRIVATE|ACCESS|ENCRYPTION|SIGNING|MASTER)_KEY$"; "i"))
                       or (.value | tostring | test("://[^/?#@ ]*:[^/?# ]+@")))
              | .name ] | unique | select(length > 0)
          | "\($l["kamiwaza.io/deployment-id"] // "-") \($l["extensions.kamiwaza.io/deployment-id"]): \(join(" "))"'

    Each line names a deployment by its ID and its runtime, then the variables it holds in plain text. A deployment created on 1.3.4 or later does not appear.

  2. For each variable whose value an administrator entered, rotate the credential where it was issued: for example, revoke the API key at its provider and create a new one. If you used the same password anywhere else, change it there too. Until the deployment has the new value, whatever in it uses the credential stops working.

  3. Check whether the deployment keeps data, using the runtime from step 1:

    kubectl -n kamiwaza get pvc -l extensions.kamiwaza.io/deployment-id=<runtime>
    • No volume claims, and not Kaizen or Kaizen (Legacy): deploy the extension again from the App Garden with the new value, check that the new deployment works, then stop the old deployment in the App Garden. Stopping it there also removes its plain-text copy.
    • Volume claims, or Kaizen or Kaizen (Legacy): do not stop or replace the deployment. Contact Kamiwaza support.

  4. A deployment that was stopped through the API (DELETE /api/apps/deployment/<id>), rather than in the App Garden, keeps its ConfigMap and its volumes, so it still appears in step 1. When you no longer need its data, remove it with DELETE /api/apps/deployment/<id>/purge, which deletes its volumes too. <id> is the ID that step 1 prints first.

Check: run step 1 again. The line of each deployment you replaced or purged is gone, and the new deployments do not appear. A deployment that step 3 leaves to Kamiwaza support is still listed.

Workroom context settings removed by the upgrade​

From 1.3.4, only administrators can set the settings of an ontology or VectorDB instance, and the settings of an OmniParse instance other than its model selection, enable, timeout, and processing settings. Before 1.3.4, any member of a workroom could set them, including where an instance connects and with what credentials. The upgrade removes such settings from OmniParse, ontology, and VectorDB instances in a workroom other than the Global Workroom, and from the record each instance is deployed from, including values an administrator set: Kamiwaza cannot tell who set them. Instances in the Global Workroom do not change.

  • OmniParse: every setting is removed except the model selection, enable, timeout, and processing settings.
  • Ontology and VectorDB: every setting whose name marks an endpoint, URL, or host, a credential, a TLS or authentication setting, a proxy, a trust store, an identity, or a loader path is removed. Settings such as model names, timeouts, and sizes are kept, and a VectorDB keeps the credentials Kamiwaza generated for its own Milvus.
  • Graphiti endpoints: Kamiwaza restores an ontology instance's chat and embedding endpoints from its chat and embedding models on the next ontology operation, which can return a 503 while the instance restarts; retry it. If Kamiwaza cannot restore an endpoint, the operation returns a 503 with the code pre_gate_endpoint_scrubbed, and an administrator selects the instance's chat and embedding models again, as in step 4 below.

Running pods. The first time Core starts after the upgrade, it also removes these settings from the running pods of the instances the upgrade changed. On each such pod, Core changes only the removed settings and any setting built from one of them. Each one gets the value the instance would be deployed with now, or is removed if it would not be set. For an OmniParse instance, Core first restores the settings Kamiwaza writes itself, such as its model endpoints and the Kamiwaza API address. The affected pods of each changed instance restart once, and its data volumes are kept. Other instances do not restart. A stopped or failed instance has no running pod to change; if it is started again, it starts without these settings. An ontology instance's chat and embedding endpoints are restored as described above.

The API calls use ADMIN_TOKEN, an access token for the admin user, as in Get an admin token, and kamiwaza.example.com for your Kamiwaza domain.

  1. Check what Core did. Core's scheduler logs each instance it changed, and a summary. The lines name instances and settings, never their values, and each can appear twice, once as JSON:

    kubectl -n kamiwaza logs deployment/core-scheduler -c core \
      | grep -e 'scrubbed context instance' -e 'scrubbed settings' \
        -e 'left the pod of' -e 'changed meanwhile' -e 'on its running pod'

    • Startup: removed scrubbed settings <settings> from the running pod of <type> instance <instance ID>; its pods restart once: Core removed the settings from that instance's pod.

    • Startup: scrubbed context instance pod env reconcile {'found': …, 'updated': …, 'unchanged': …, 'skipped': …, 'errors': …}: the summary, logged at every start. At a later start, it counts only the instances still listed. errors should be 0, and skipped should count only instances whose deployment is not running, described next. If no summary line appears, restart the scheduler as shown below. If it still does not appear, contact Kamiwaza support.

    • Startup: left the pod of <type> instance <instance ID> for the next start: its deployment is not running: the instance is stopped or failed, so it has no running pod to change. Core has already removed the settings from what the instance is deployed from, so if it is started again, it starts without them. Core logs this line, and counts the instance in skipped, at every start until the instance runs again or is deleted. At the first start after it runs again, Core checks its pod and stops listing it. No action is needed, unless the removed settings included a password for data the instance already holds: it would start without that password and could not open that data, so contact Kamiwaza support before you start it.

    • Startup: left the pod of <type> instance <instance ID> for the next start: <reason> with any other reason, or a line that says Core could not remove scrubbed settings from an instance's pod: the instance was deploying, stopping, or being deleted, or the change failed. Core tries again at its next start. To try again now, wait until the instance has finished deploying or stopping, then restart the scheduler and check the log again:

      kubectl -n kamiwaza rollout restart deployment/core-scheduler
      kubectl -n kamiwaza rollout status deployment/core-scheduler --timeout 15m

      If Core leaves the instance again for the same reason, contact Kamiwaza support.

    • Startup: the pod env record of <type> instance <instance ID> changed meanwhile; it is reconciled on the next start: the instance's record changed while Core worked on it, so Core checks the instance again at its next start. The summary does not count this in skipped. To check now, restart the scheduler as shown above.

    • Startup: <type> instance <instance ID> keeps <settings> on its running pod, logged as an error: Core leaves some settings on the pod, such as identity and credential settings. A new password, for example, would lock an instance out of data created with the old one. The value stays on the running pod, and Core logs that line again at every start. If the value is one an administrator set or Kamiwaza generated, you can leave it on the pod; otherwise, contact Kamiwaza support. Core does not keep the value for the instance's next deployment, though. If it is a password that protects data the instance already created, stopping and starting the instance would leave it unable to open that data. Do not stop that instance, and contact Kamiwaza support if it must keep the value.

  2. List the instances the upgrade could change. This query changes nothing. It prints each instance's type, workroom ID, instance ID, and name, never its settings:

    kubectl -n kamiwaza exec core-postgres-0 -c postgres -- \
      psql -U core -d kamiwaza -At -c "
        SELECT 'omniparse', workroom_id, id, name FROM context_omniparse_instances
        WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
        UNION ALL
        SELECT 'ontology', workroom_id, id, name FROM context_ontology_instances
        WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
        UNION ALL
        SELECT 'vectordb', workroom_id, id, name FROM context_vectordb_instances
        WHERE workroom_id <> 'ffffffff-ffff-ffff-ffff-ffffffffffff'
        ORDER BY 1, 2, 4"

    Each line is <type>|<workroom ID>|<instance ID>|<name>.

  3. Find the values the upgrade removed. Compare against your own records of what administrators set, or read an instance's settings from before the upgrade out of the backup you took in Before you start. This command changes nothing. It prints the instance's row from the backup. The instance's settings are the row's JSON object, credentials included, so run it where no one else can see the output:

    kubectl -n kamiwaza exec -i core-postgres-0 -c postgres -- \
      pg_restore --data-only --table=context_omniparse_instances --file=- \
      < kamiwaza-before-upgrade.dump \
      | grep '^<instance ID>'

    For an ontology or VectorDB instance, use the table context_ontology_instances or context_vectordb_instances. Optionally, cross-check the upgrade's own record: the database job's log names each instance the upgrade removed settings from, with its workroom and the names of the removed settings, never their values. By default the job and its log are removed about an hour after the job finishes:

    kubectl -n kamiwaza logs -l app.kubernetes.io/name=core-db-init -c db-init --tail=-1 \
      | grep -e 'removed privileged' -e 'config not an object'

    A line that says config not an object names an instance whose settings Kamiwaza could not read. Core has no record of what the upgrade removed from it, so the values may stay on its running pod. Save it as below, or delete it. If its values must leave the running pod, contact Kamiwaza support.

  4. Re-enter the values an administrator set. Save an instance only to re-enter values. A save stores them in the instance's settings and the record it is deployed from. It does not change the instance's running pod: if a value you re-enter must reach the running pod, contact Kamiwaza support. Each request prints the HTTP status; 200 means the instance was saved.

    • OmniParse: put the values in a file, so they stay off the command line, as {"config": {"<setting>": "<value>"}}. Then send it with the instance's workroom ID:

      curl --silent --show-error --request PUT \
        --header "Authorization: Bearer $ADMIN_TOKEN" \
        --header "X-Workroom-ID: <workroom ID>" \
        --header 'Content-Type: application/json' \
        --data @omniparse-settings.json \
        --write-out ' %{http_code}\n' \
        "https://kamiwaza.example.com/api/context/omniparses/<instance ID>"

    • VectorDB: send them the same way to /api/context/vectordbs/<instance ID>.

    • Ontology: Kamiwaza 1.3 has no request to set an ontology instance's settings. Kamiwaza restores the chat and embedding endpoints itself, as described above. If an operation returns pre_gate_endpoint_scrubbed, select the instance's models again with PUT /api/context/ontologies/<instance ID>/model-bindings and the body {"chat": {"deployment_id": "<chat model deployment ID>"}, "embedding": {"deployment_id": "<embedding model deployment ID>"}}. To restore another setting the upgrade removed from an ontology instance, contact Kamiwaza support.