Skip to main content
Version: 1.3.3 (Latest)

Release notes

Kamiwaza 1.3.1​

Kamiwaza 1.3.1 is a patch release for the 1.3 line. It replaces the 1.3.0 build that was delivered earlier as 1.3.0-rc.1. If you run that build, these notes describe what changes when you move to 1.3.1. It is mostly fixes and security updates. If your installation has no internet access, it needs the new air-gapped values profile: see Air-gapped installations.

Kaizen data is not migrated in 1.3.1

1.3.1 has no Kaizen data migration. A new Kaizen 0.7.3 deployment starts empty: conversations, documents, artifacts, collections, and model selections from an existing Kaizen deployment do not carry over. Stopping a Kaizen deployment in the App Garden deletes its data, and uninstalling Kamiwaza deletes all platform and Kaizen data. Back up before you upgrade, and keep your existing Kaizen deployment unless you are starting fresh. A data-preserving Kaizen upgrade is being addressed in a future release.

For behavior changes that 1.3 introduced over 1.2, see the per-area notes:

For the platforms and configurations this release supports, see Supported Configurations.

Highlights​

  • Kaizen 0.7.3. 1.3.1 ships Kaizen 0.7.3, which new Kaizen deployments use once an administrator selects it; see Extensions after the upgrade. Kaizen's built-in features now work on air-gapped installations without extra setup, except the Software Engineer agent's language-server tools. Hybrid search works, and several reliability and error-message fixes are included. See Kaizen.
  • Tasks are off by default in Kaizen. New Kaizen deployments no longer show Tasks. After an in-place upgrade, an existing Kaizen deployment from 1.3.0 keeps Tasks on; see Extensions after the upgrade.
  • Air-gapped installs no longer fail on the App Garden catalog. See Platform fixes.
  • vLLM on NVIDIA GB10 (DGX Spark) with tenant-mode inference. The tenant-mode engine catalog now includes an arm64 vLLM image for the GB10. See Tenant-mode inference.
  • Security updates for Kamiwaza Core base images, nginx, OmniParse, Kaizen, and the Skills Library. See Security updates.

Kaizen​

1.3.0 shipped Kaizen 0.7.1. 1.3.1 ships Kaizen 0.7.3, which also includes everything in Kaizen 0.7.2.

Kaizen 0.7.2​

  • Tasks are off by default for new deployments. See Extensions after the upgrade.
  • Mobile experience. Kaizen now has a mobile layout.
  • Scoped public web access. Administrators can grant public web access to specific uses through a new setting, which is empty by default and changes nothing until configured.
  • Federated cluster tools are available and are off by default. An administrator turns them on with FEDERATION_TOOLS_ENABLED.
  • Multipart upload limit. By default, Kaizen lets at most two multipart upload parts use a database connection at the same time, across all API replicas. Additional parts get a retryable 429 response, and the browser retries them after a short wait. Under sustained load, a large browser upload can use up its retries and fail. You can raise the limit with MULTIPART_PART_DB_CONNECTION_BUDGET, but only after you confirm that your database has spare connections, and every API replica must use the same value.

Kaizen 0.7.3​

  • Air-gapped installations. The dependencies that Kaizen's bundled agents need are now included in the Kaizen images, so built-in agent tools no longer try to download packages on first use. The one exception is listed under Known issues. The code sandbox also includes a Python data and document stack, including pandas, NumPy, SciPy, Matplotlib, openpyxl, python-docx, python-pptx, ReportLab, and pypdf, so everyday data and document work runs with no internet access. When a package that is not included is requested and the package registry cannot be reached, Kaizen now fails within seconds and says that the registry is unreachable, instead of stalling for minutes.
  • Hybrid search and document indexing. Kaizen could fail to reach the embedding model, and hybrid search and semantic document indexing then fell back to keyword-only results without a warning. Kaizen now reaches embedding models through Kamiwaza's in-cluster API and addresses them correctly, and embedding addresses saved by the earlier build correct themselves.
  • Model selection on Envoy ingress. On installations that use Envoy ingress without Istio, Kamiwaza withholds browser credentials from extensions, so Kaizen could not list the available models. Kaizen can now list them there. Chat on these installations is still affected; see Known issues.
  • Reliability.
    • When a tool call is refused, the refusal is returned to the model so the conversation continues, instead of ending the turn.
    • The Agent Builder resumes an approved draft on its own.
    • Internal requests that are safe to repeat are retried briefly when the gateway returns a temporary error.
    • Reasoning-effort settings sent to Qwen3.8 deployments now use values the model server accepts.
  • Clearer error messages.
    • When a requested file fails to build, Kaizen names the file and gives a one-line cause.
    • When a code run fails because a dependency could not be installed, Kaizen names the failed install instead of reporting an unrelated error.
    • When a Kaizen service is unavailable, Kaizen names the service.
  • Artifacts. A request that explicitly asks for an artifact now produces one, even when the content is short.

Platform fixes​

  • Reasoning effort for self-hosted models. For OpenAI-compatible model endpoints other than OpenAI itself, such as vLLM, Ollama, or a LiteLLM proxy, Kamiwaza previously dropped every reasoning-effort setting, so a request to turn reasoning off could run at the server's maximum effort. A model configuration's external_endpoint now has a supports_reasoning_effort setting, off by default. When it is on, Kamiwaza forwards the requested effort, including none, to the server. When it is off, endpoints behave as before. The registration form does not show this setting yet. To set it, read the configuration with GET /api/model_configs/{model_config_id}, add "supports_reasoning_effort": true inside its existing external_endpoint value, keeping that value's other fields and its format, and send the whole configuration back with PUT /api/model_configs/{model_config_id}. The PUT replaces the configuration, so send every field, not only the changed one.
  • Workroom changes under load. Deleting a workroom, or changing its members or owner, could fail with a 503 error when another operation held the same internal lock. Kamiwaza now waits for the lock instead of failing.
  • Air-gapped installs and the App Garden catalog. An air-gapped install or upgrade could fail because the App Garden catalog sync step tried to reach Kamiwaza's online catalog and got an HTTP 500 error (apps: unreachable). 1.3.1 adds an air-gapped values profile, kamiwaza-offline.yaml, that turns the online lookup off. Layer it on every air-gapped install and upgrade, as described in Air-gapped installations.
  • Bundled embedding model on air-gapped installations. When the default embedding model is bundled with the installation, Kamiwaza now loads it only from the bundle and reports a clear error if the bundled file is missing or does not match its checksum, instead of trying to download it from the internet. On OpenShift, deploying the embedding model stays an administrator action.
  • Model deployments on OpenShift with GreyMatter. A model deployment could start before GreyMatter was ready for it and then run without joining the GreyMatter mesh. Kamiwaza now waits for GreyMatter and restarts model pods that started too early.

Upgrade notes​

Upgrade from 1.3.0​

You can move from 1.3.0 to 1.3.1 in either of two ways. Choose Option 1 if you can redeploy your models and extensions and re-enter your data. If you have platform or Kaizen data you need to keep, use Option 2. Back up first in both cases.

Back up first. Back up Kamiwaza's Postgres database and, for each Kaizen deployment, its database and its object store. For example, run pg_dumpall in each Postgres pod, and archive the /data directory of each Kaizen deployment's SeaweedFS pod. 1.3.1 does not document restoring a Kaizen backup into a new deployment; contact Kamiwaza support if you need one.

Option 1: Uninstall and install fresh (recommended for 1.3.1)​

This is the recommended path for 1.3.1 when you don't need to keep existing data. An uninstall does not keep any data: it deletes Kamiwaza's databases and object store, and every extension's data, including Kaizen's, and the model files. After the fresh install, you download or copy in your models again, deploy your models and extensions again, and re-enter your data.

  1. Back up, as above.
  2. Uninstall 1.3.0 by following the uninstall page for your environment: Kubernetes with Istio or OpenShift with GreyMatter.
  3. Install 1.3.1 by following the install page for your environment, with --version 1.3.1 and the same values files: Kubernetes with Istio or OpenShift with GreyMatter. The uninstall pages keep the kamiwaza-license Secret unless you also removed the namespace; if it is still there, skip adding the license. Air-gapped sites first copy the 1.3.1 images into their registry, as Install Without Internet Access describes, and layer the air-gapped profile; see Air-gapped installations.
Option 2: Upgrade in place​

An in-place helm upgrade from 1.3.0-rc.1 was tested with a 1.3.1 release candidate on a connected Kubernetes + Istio install and on an air-gapped k0s + Istio install using the field kit's tag-only transfer. It keeps Kamiwaza's models, workrooms, users, data, and storage volumes, and your existing extension deployments keep running unchanged.

  1. Back up, as above.

  2. Air-gapped sites only: copy the 1.3.1 images into your registry, as described in Install Without Internet Access, and layer the air-gapped profile in the next step, as described in Air-gapped installations.

  3. Upgrade in place with Helm. As the identity that installed Kamiwaza, rerun the helm upgrade --install command from your platform's install page with --version 1.3.1 and every values file you installed with, in the same order. For an Istio installation with one values file, the command is:

    helm upgrade --install kamiwaza \
      oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
      --version 1.3.1 \
      --namespace kamiwaza \
      --values kamiwaza-values.yaml \
      --wait --wait-for-jobs --timeout 30m

    On OpenShift with Helm 4, keep --server-side=false, as Install on OpenShift with GreyMatter describes. Air-gapped sites use the chart in their own registry and add --values kamiwaza-offline.yaml before their own values files. Do not uninstall before an in-place upgrade: uninstalling deletes the platform databases and object store.

  4. Check the upgrade. Every Kamiwaza deployment's pods are Ready, the upgrade's jobs have completed, helm list shows the 1.3.1 chart, and the platform answers 200 at / and 401 at /api/ without credentials.

  5. Leave existing extension deployments as they are unless you decide otherwise, as described in Extensions after the upgrade.

Image tags​

  • Platform images that the chart pins by tag and digest are published under the tag release-1.3.1. Images that the chart pins by tag alone keep that tag. An image that would otherwise have no tag is published as release-1.3.1- followed by the first 12 hexadecimal characters of its digest, for example llamacpp-cpu:release-1.3.1-<first 12 hex characters>. App Garden extension images keep their own tags. Published tags never move, so the 1.3.0 images keep their release-1.3.0 tags.
  • The chart refers to many images by digest (@sha256:...). Copy images with a tool that keeps each image's digest unchanged, as the air-gapped install page describes.
  • If your transfer process changes image digests, for example by pulling, tagging, and pushing each image, the chart's digest references do not resolve in your registry. Install with the values that Kamiwaza's field kit generates in its tag mode (IMAGE_REF_MODE=tag, the default). Those values refer to every image by tag, with no digest, and the field kit's intake check verifies the copies in your registry. The field kit's tag mode is the supported way to install after a digest-changing transfer. If your transfer keeps every digest unchanged, you can set IMAGE_REF_MODE=digest instead.

Air-gapped installations​

On every install or upgrade of a site with no internet access, layer the air-gapped values profile, kamiwaza-offline.yaml. It contains only this setting, which turns off the App Garden's online catalog lookup:

core:
  templates:
    availability:
      enabled: false

Kamiwaza's field kit includes this file. If you don't have it, create kamiwaza-offline.yaml with exactly the content above. Pass it before your own values file, so that your values still take precedence:

helm upgrade --install kamiwaza \
  oci://<your registry host>/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version 1.3.1 \
  --namespace kamiwaza \
  --values kamiwaza-offline.yaml \
  --values kamiwaza-values.yaml \
  --wait --wait-for-jobs --timeout 30m

Without this setting, the install or upgrade fails when the catalog sync step cannot reach Kamiwaza's online catalog. If your values file already sets it, as step 5 of Install Without Internet Access shows, the profile changes nothing, and layering it is still safe. Do not layer it on a connected site, where it would turn off online App Garden discovery.

The field kit. Kamiwaza publishes a field kit with each release, together with an intake list of every image and a pickup runbook. If you install with the field kit, follow the README and runbook inside it: its install command already layers this file, and its install check reports it if online App Garden availability is still on.

Extensions after the upgrade​

  • An in-place upgrade does not change extensions you have already deployed. An existing Kaizen deployment keeps running Kaizen 0.7.1, with its data and the settings it was deployed with. The App Garden deploys the version an administrator has selected for each extension, and an existing selection stays on its earlier version after the platform upgrade. The version selection's Update all changes the version that new deployments use. It does not change a running deployment.

  • 1.3.1 has no Kaizen data migration. Each Kaizen deployment has its own database and storage volumes. A new Kaizen 0.7.3 deployment starts empty: no conversations, documents, artifacts, collections, or model selections carry over from 0.7.1. A data-preserving Kaizen upgrade is being addressed in a future release.

  • Stopping a Kaizen deployment deletes its data. Stopping a Kaizen deployment in the App Garden deletes its storage volume claims. With a Delete reclaim policy, the Kubernetes default for dynamically provisioned volumes, that permanently deletes its conversations, documents, artifacts, and collections. The confirmation dialog does not warn about this yet. Do not stop a Kaizen deployment that holds data you need.

  • Kamiwaza recommends keeping your current Kaizen deployment unless you are starting fresh. Kaizen 0.7.3 is optional. To run it alongside 0.7.1:

    1. Back up the 0.7.1 deployment's database and object store, and Kamiwaza's Postgres database.
    2. Make sure the cluster has room for a second complete Kaizen deployment. Without it, the new deployment's pods stay Pending and the deployment fails. Never stop the 0.7.1 deployment to make room. If the new 0.7.3 deployment fails before anyone has used it, or opens the Kamiwaza dashboard instead of Kaizen, add capacity if it was short, then stop that new, unused deployment and deploy again. Do not stop a 0.7.3 deployment that anyone has already added data to; contact Kamiwaza support instead.
    3. In the App Garden, as an administrator, select Check for updates, then use Update on the Kaizen row (or Update all), or Choose version to pick 0.7.3. Confirm that the row shows Selected: 0.7.3. The same steps select the new versions of the other extensions in the table below.
    4. Deploy Kaizen from the App Garden.
    5. In the new deployment's Admin area, under Models, select the chat and embedding models. Then upload your documents again and choose Index on each one, or Index selected for several.

  • Kaizen Tasks. Durable Tasks (TASKS_ENABLED) is a setting you choose when you deploy Kaizen. A new Kaizen 0.7.3 deployment has it off by default. An existing 0.7.1 deployment had it on by default and keeps Tasks on. With Tasks off, members see no Tasks pages or chat task tools.

  • New extension versions:

    Extension1.3.01.3.1
    Kaizen (kaizen)0.7.10.7.3
    OmniParse (tool-omniparse)2.2.32.2.4
    Skills Library (skills-library)0.5.20.5.3
    Hello Web (hello-web)1.0.01.0.1

Other upgrade notes​

  • Kaizen sandbox network on air-gapped installations. On an installation with no internet access, set Kaizen's Sandbox outbound scope (SANDBOX_EGRESS_MODE) to isolated when you deploy Kaizen. This is recommended, not required. You can change it later in Kaizen Admin, where it is labeled Sandbox egress mode, without a restart.
  • Upgrading from 1.2. See the per-area notes linked above, including the step to re-label existing NVIDIA nodes in the model serving notes.

Security updates​

1.3.1 includes these dependency and base-image updates:

ComponentUpdate
nginx (Hello Web sample app)Updated from nginx 1.28 to nginx 1.30.5.
OmniParse 2.2.4Updated PyJWT to 2.15.1, pypdf to 6.19.0, and urllib3 to 2.8.0, and refreshed the Chainguard base image.
Kamiwaza Core, frontend, Keycloak user setupRefreshed the Node.js 22 and Python 3.12 base images.
Kaizen 0.7.3Refreshed the Chainguard base images of all Kaizen images, and updated multidict to 6.9.1.
Skills Library 0.5.3Rebuilt with updated dependencies.
Kamiwaza Core, Kaizen, Skills LibraryUpdated source-map-js to 1.2.2 (GHSA-68fv-2mgg-jv7q).

Known issues​

  • Stopping a Kaizen deployment does not warn that its data is deleted. The App Garden's confirmation dialog for Stop says only that it terminates the running instances, but stopping a Kaizen deployment also deletes its storage volume claims and, with a Delete reclaim policy, its data. A warning in the dialog is being addressed in a future release. See Extensions after the upgrade.
  • Kaizen's embedding model note gives the wrong vector size. The Kaizen Admin Models screen says the embedding model must return 768-dimensional vectors. The 384-dimensional MiniLM model also works. The wording is being corrected in a future release.
  • Kaizen chat on Envoy ingress without Istio. On installations that use Envoy ingress without Istio (global.ingress.provider: envoy), Kaizen lists the available models, but chat with them fails. The fix, a later Kaizen version plus a configuration setting, is planned for a later release. Installations that use Istio or OpenShift with GreyMatter are not affected.
  • Tenant-mode diffusion uses the earlier engine image. Diffusion deployments under tenant-mode inference still run the diffusion CUDA engine image from before this release's base-image refresh. The refreshed image is planned for a later release.
  • Kaizen starts with no embedding model selected. Until an administrator selects an embedding model in Kaizen, documents are indexed for keyword search only. Selecting a model does not index documents automatically, including documents uploaded afterward: choose Index on each document, or Index selected for several, to add them to semantic search. The same applies to a new Kaizen deployment.
  • Model endpoints over plain HTTP need a cluster setting. Registering a model endpoint that uses plain http:// or a private network address, such as an internal vLLM, Ollama, or LiteLLM server, requires core.scheduler.allowInsecureModelEndpoints: true in your Helm values. It is off by default, and the setting is not yet covered elsewhere in these docs. Turn it on only for endpoints on a network you trust. Cloud metadata addresses stay blocked even when it is on.
  • Language-server tools in the Software Engineer agent need internet access. The agent's language-server tools install their language servers from the npm registry on first use, so they fail on an air-gapped installation. The agent's other code tools, including structural code search, work offline.
  • Extra packages need a registry. Python or npm packages that are not included in the Kaizen images still need a package registry or an internal mirror.