Skip to main content
Version: 1.3.3

Kamiwaza 1.3 release notes

Kamiwaza 1.3 installs with Helm into a Kubernetes cluster that you run. This page covers what is new in 1.3.0, what changed from 1.2, what is deprecated, the known issues for the 1.3 line, and each patch release, newest first. The latest release is 1.3.3.

New in 1.3.0​

1.3.0 was delivered as 1.3.0-rc.1. That chart and its release manifest are the 1.3.0 build.

Installation and distribution​

  • Helm install into a cluster you run. Kamiwaza installs with one helm upgrade --install into a single namespace, as a namespace-scoped identity, without cluster-admin. Kamiwaza no longer sets up Kubernetes, the service mesh, storage, or GPU drivers. A platform administrator prepares those, following the prerequisites page for each environment.
  • Two documented environments: Kubernetes with Istio, and OpenShift with GreyMatter. Each has prerequisites, install, and uninstall pages. On OpenShift, global.platform: osgm selects the OpenShift and GreyMatter settings. See Installation Environments.
  • Public distribution. The chart and every image are published at oci://ghcr.io/kamiwaza-ai/releases/kamiwaza and pull without credentials. Each release publishes a manifest that lists every image.
  • Air-gapped installation from your own registry. Copy the release into your registry and set one registry-prefix value; it re-addresses every image, including App Garden extensions and inference engines. See Install Without Internet Access.
  • Images pinned by digest, each with its third-party license notices. Images built for a release are published under the version's tag, such as release-1.3.3; an image with no tag of its own gets release-1.3.3- followed by 12 characters of its digest. Images that carry their own version, such as PostgreSQL, and App Garden extension images keep their own tags. A published tag never moves.
  • Signed license file. Every installation needs a signed license.lic, checked at startup without contacting Kamiwaza. API responses report the license state, and the UI shows a banner as the term ends. An expired term does not stop the platform. Accepting the EULA is a required install value. See License File.
  • Documented uninstall for each environment.

Platform​

  • Istio Gateway API routing. Kamiwaza attaches its routes to your Istio gateway from its own namespace and checks access at the gateway. Models, apps, and tools are served under paths on one HTTPS origin (/runtime/models|apps|tools/<id>). See Routing modes.
  • Envoy edge without Istio (early release). The chart can deploy a standalone Envoy edge for clusters that do not run Istio (global.ingress.provider: envoy). It is not yet a documented install environment.
  • Relationship-based access control on SpiceDB (early release). Access control is relationship-based by default, with SpiceDB as the backend.
  • Storage. Kamiwaza uses the block StorageClass you provide. Object storage is a built-in SeaweedFS in the namespace by default, or your own S3-compatible service.
  • Generated admin password. If you do not set one, the install generates the admin password and stores it in the kamiwaza-user-admin Secret.

App Garden and extensions​

  • Bundled extension catalog. Each release ships a fixed set of extension versions, so the App Garden works without reaching the online catalog.
  • Administrators choose extension versions. An administrator selects the version of each extension that new deployments use. Running deployments are not changed.
  • Kaizen is now the new Kaizen (v4). The kaizen entry is Kaizen 0.7.1, shown as Kaizen.Next in 1.2. Kaizen v3 continues as Kaizen (Legacy) (kaizen-legacy, 2.1.4), for existing deployments.
  • Extension versions in 1.3.0: Kaizen 0.7.1, Kaizen (Legacy) 2.1.4, OmniParse 2.2.3, Skills Library 0.5.2, Milvus 2.4.11, Graphiti 2.6.7, Kamiwaza DDE 2.5.4, Data Connector Builder and Runtime 0.2.5 (alpha), Google and Microsoft 365 connectors 0.1.1, and the Hello Web sample 1.0.0.
  • Extension developers: the SDK and kz-ext support Kamiwaza 1.3. Upgrade kz-ext, because earlier versions report Kamiwaza 1.3.0 as unsupported. kz-ext refuses build arguments that look like credentials; use BuildKit build secrets instead.

Models and GPUs​

  • Tenant-mode inference. A namespaced installation can serve local models on CPUs and NVIDIA GPUs without reading the cluster's nodes. The cluster owner describes the hardware in a signed profile, and Kamiwaza places each deployment only on that hardware: CPU, whole GPUs, MIG slices, or GPUs shared through the VRAM plugin. The owner tool ships in the Core image. See Tenant-mode inference.
  • Engine catalog. Tenant mode serves llama.cpp, vLLM, diffusion, and whisper.cpp from digest-pinned images, and picks the newest image that both the GPU and the host driver can run.
  • Image generation on NVIDIA GPUs with a first-party diffusion engine.
  • vLLM 0.28.0 (from 0.23.0).
  • Deployments report what they serve: context window, and reasoning and vision support, for local and external models.
  • Claude 5 models through Amazon Bedrock.
  • Embedding improvements, including a bundled default embedding model for installations without internet access.
  • GPU node guide with driver requirements and a check for each step. See GPU Nodes.
  • Hardware: NVIDIA GPUs. AMD GPUs are not supported in 1.3. Platform nodes are x64/amd64. NVIDIA GB10 (DGX Spark, arm64) nodes run inference engines; vLLM on GB10 is available from 1.3.1, and diffusion from 1.3.3.

Federation​

  • Delegated jobs run without KubeRay, so namespaced installations can run them.
  • The execution gate set in Helm takes effect, and the API reports which gate is in force.
  • Retrieval responses say when they are truncated.
  • Clearer pairing errors, and a private certificate authority is shared with peers during pairing.

See Federation 1.3.0 release notes.

Changes from 1.2​

  • No in-place upgrade from 1.2 or earlier. Contact your Kamiwaza representative to plan an assisted migration to a new installation.
  • The installer scripts are removed, including the online installer that set up Kubernetes on the host and the offline bundle. Kamiwaza is no longer distributed through Keygen.
  • A license is required. Kamiwaza does not start without a valid signed license file.
  • macOS is no longer supported. See System Requirements.
  • Removed components: the earlier ingress routing provider, replaced by Istio (see Routing modes), Rook/Ceph storage, KubeRay, and the bundled Grafana and Loki.
  • GPU serving needs setup. A default installation cannot see GPUs, and a GPU deployment returns 422 until the cluster owner turns on tenant-mode inference or grants Kamiwaza read access to nodes. See GPU Nodes.
  • The kaizen entry is a different product. There is no migration from Kaizen v3 to Kaizen v4 in any release. Existing v3 deployments keep running. Adopting Kaizen v4 means a new deployment. The kaizen-next name is retired.
  • Workrooms are built into Kamiwaza, and the Workroom Manager extension is retired. The Workrooms page is off by default.
  • Vespa is not offered in the 1.3 App Garden. Existing Vespa deployments and data are kept.
  • Install and upgrade with Helm only. Applying rendered manifests, for example through Argo CD or Flux, is not supported.
  • Model serving changes: see Model serving 1.3.0 release notes.

Deprecated, removed in 1.4​

  • Kaizen (Legacy) (kaizen-legacy) is removed in 1.4.0, with no migration to Kaizen.
  • The workroom classification field, replaced by level_id.
  • global.workroomManager.surface: extension.
  • A plain string for global.greymatter.proxy.image. Use the registry, repository, and tag fields, or digest.

Known issues​

Open in 1.3.3:

  • A new Kaizen version does not keep data. A new Kaizen deployment starts empty, and stopping a Kaizen deployment in the App Garden deletes its data without a warning in the dialog. See Upgrade Kamiwaza.
  • The App Garden keeps the extension versions selected before a platform upgrade. An administrator selects the new versions.
  • Kaizen starts with no embedding model selected. Until an administrator selects one, documents are indexed for keyword search only. Selecting a model does not index documents automatically, including documents uploaded afterward; choose Index on each document, or Index selected. The same applies to a new Kaizen deployment.
  • Kaizen's Models screen says the embedding model must return 768-dimensional vectors. The 384-dimensional MiniLM model also works.
  • The Software Engineer agent's language-server tools need internet access.
  • Extra Python or npm packages that the Kaizen images do not include need a package registry or an internal mirror.
  • Model endpoints over plain HTTP or private addresses require core.scheduler.allowInsecureModelEndpoints: true. Turn it on only for endpoints on a network you trust. Cloud metadata addresses stay blocked even when it is on.
  • Istio ambient mode is not supported. Istio must run in sidecar mode.
  • Tenant-mode inference is qualified on k0s with Kubernetes 1.36. A deployment across more than one GPU is not qualified, and MLX is not available in tenant mode.

Fixed:

  • Air-gapped installations failed on the App Garden catalog sync. Fixed in 1.3.1.
  • The bundled embedding model could try to download from the internet. Fixed in 1.3.1.
  • On OpenShift with GreyMatter, a model deployment could start before GreyMatter was ready and not join the mesh. Fixed in 1.3.1.
  • Workroom deletes and membership changes could fail with 503 under load. Fixed in 1.3.1.
  • Tenant-mode diffusion used the engine image from before the 1.3.1 base-image refresh. Fixed in 1.3.2.
  • Kaizen chat failed on Envoy ingress without Istio. Fixed in 1.3.3 for Kaizen that an administrator deploys from the catalog.

Patch releases​

1.3.3​

Security fixes, Kaizen 0.7.6, and OmniParse 2.2.7.

Action required after upgrading

  • Redeploy every tenant-mode model. Models deployed earlier do not get the new model-route authentication until redeployed, and diffusion deployments from earlier releases are refused through the gateway until redeployed.
  • Select Kaizen 0.7.6 and OmniParse 2.2.7 in the App Garden, then redeploy OmniParse. Kaizen 0.7.6 is a new deployment; see Upgrade Kamiwaza.
  • Air-gapped installations: keep the earlier Kaizen images in your registry while deployments of earlier Kaizen versions still run.

Security

  • On Kubernetes with Istio, model routes (/runtime/models/...) now require credentials. Before, they answered unauthenticated requests through the gateway.
  • Direct-runtime tool routes (/runtime/tools/...), including MCP tools and OmniParse, now run Kamiwaza's access check. Core updates existing tool routes when it starts.
  • Extension network policies admit only each extension's primary port, so an extension's data stores are no longer reachable from the whole namespace.
  • Kaizen 0.7.6 generates unique service credentials for each deployment and keeps its sandbox worker token in a Secret instead of the pod specification.
  • Updated OmniParse 2.2.7 (mcp 2.3.0), the CUDA diffusion engine (torchvision 0.28.0 with CUDA 12.9, without FFmpeg), and Keycloak (v26.7.4-kz.4).
  • Rebuilt the vLLM 0.28.0 CUDA engine images with security fixes (v0.28.0-kz.6, and v0.28.0-arm64.kz.5 for GB10).

Fixes and improvements

  • Kaizen chat works on Envoy ingress without Istio when an administrator deploys Kaizen from the catalog (core.auth.modelInference.trustedRuntimeTargets, default [template:kaizen]).
  • CUDA diffusion runs on NVIDIA GB10 (arm64).
  • Model staging no longer runs out of memory on nodes with slow disks.
  • Tenant-mode vLLM pods are no longer evicted while compiling a model.
  • App Garden deploy errors name the missing required variable.
  • GET /api/kamiwaza/metadata returns the running Core version and build.
  • Kaizen 0.7.6: Project members can search and cite documents shared with the Project, Kaizen says when it cannot do something instead of failing the turn, a chat that finishes in a hidden tab raises a notification, and TLS certificate errors name the cause and the fix.

1.3.2​

Extension and security updates.

  • Kaizen 0.7.4, Kaizen (Legacy) 2.1.6, OmniParse 2.2.6, Skills Library 0.5.4, Milvus 2.4.12, Graphiti 2.6.9, Kamiwaza DDE 2.5.5, and Data Connector Builder and Runtime 0.2.6.
  • The VRAM plugin chart is published with the release. See GPU Nodes.
  • Tenant-mode diffusion uses the refreshed engine image.

1.3.1​

Fixes, Kaizen 0.7.3, and security updates. 1.3.1 is the first release with a field kit for air-gapped sites: an intake list of every image, scripts to fetch and verify the release, and values templates for a site registry. If your transfer process changes image digests, install with the values the field kit generates in its tag mode (IMAGE_REF_MODE=tag, the default), which refer to every image by tag; its intake check verifies the copies in your registry.

Action required after upgrading

  • Air-gapped installations: set core.templates.availability.enabled: false, as in step 5 of Install Without Internet Access. The field kit includes it as kamiwaza-offline.yaml.
  • Kaizen on air-gapped installations: setting Sandbox outbound scope (SANDBOX_EGRESS_MODE) to isolated is recommended.

Fixes and improvements

  • Air-gapped installations no longer fail on the App Garden catalog sync.
  • The bundled embedding model loads only from the bundle, and reports a clear error if the file is missing or does not match its checksum.
  • On OpenShift with GreyMatter, model deployments wait for GreyMatter, and model pods that started too early are restarted.
  • Workroom deletes and membership changes wait for internal locks instead of failing with 503.
  • vLLM on NVIDIA GB10 (DGX Spark) in tenant-mode inference.
  • OpenAI-compatible model endpoints other than OpenAI, such as vLLM, Ollama, or LiteLLM, can receive the requested reasoning effort. Turn it on with supports_reasoning_effort in the model configuration's external_endpoint; see OpenAI-compatible chat.
  • Kaizen 0.7.3 (includes 0.7.2):
    • Built-in agent tools work without internet access, and the code sandbox includes a Python data and document stack (pandas, NumPy, SciPy, Matplotlib, openpyxl, python-docx, python-pptx, ReportLab, pypdf).
    • Hybrid search and semantic indexing reach the embedding model reliably.
    • Kaizen lists models on Envoy ingress without Istio.
    • Tasks are off by default for new deployments (Durable Tasks, TASKS_ENABLED). Existing deployments keep their setting.
    • A mobile layout, scoped public web access, and federated cluster tools (off by default; FEDERATION_TOOLS_ENABLED).
    • Multipart upload limit. By default, at most two multipart upload parts use a database connection at the same time, across all API replicas. Additional parts get a retryable 429, and a large browser upload under sustained load can use up its retries and fail. Raise MULTIPART_PART_DB_CONNECTION_BUDGET only after you confirm that your database has spare connections, and use the same value on every API replica.
    • Reliability and clearer error messages.
  • Extension versions: Kaizen 0.7.3, OmniParse 2.2.4, Skills Library 0.5.3, Hello Web 1.0.1.

Security

  • Updated the Core, frontend, and Keycloak user-setup base images (Node.js 22, Python 3.12), nginx 1.30.5 in the Hello Web sample, OmniParse 2.2.4 dependencies and base image, the Kaizen 0.7.3 base images, the Skills Library 0.5.3 dependencies, and source-map-js 1.2.2 (GHSA-68fv-2mgg-jv7q).