Skip to main content

Kamiwaza 1.4 release notes

Kamiwaza 1.4 installs with Helm into a Kubernetes cluster that you run. This page covers the changes in 1.4.0 from 1.3.5, including actions to take before upgrading, and each patch release, newest first. The latest release in this line is 1.4.1.

Kamiwaza 1.4.0​

New in 1.4.0​

Installation and distribution​

  • FIPS images by default. Core, the console, Keycloak, the inference engines and every extension that publishes a FIPS build now install as FIPS images.
  • Release documentation in the cluster. The release now ships its own frozen documentation image, and the in-cluster docs follow the installed release.

Security and identity​

  • Identity providers from Helm values. Add OIDC identity providers with auth.keycloak.identityProviders[]. They apply on fresh installs and on upgrades.
  • Audit administration role. A separate audit-admin role controls audit administration.
  • Tamper-evident audit records. Audit events carry a hash chain, and a verifier reports where the chain was changed.
  • Sign-out ends the session. Signing out now invalidates the session's access token.

Observability​

  • Metrics and authenticated export. Platform components, the managed PostgreSQL databases and the inference engines expose Prometheus metrics, and Core's OTLP export supports authentication and a private CA.
  • Wider audit coverage. Kamiwaza now audits sign-ins, privileged API actions and each connector call, and the observability chart alerts when audit records stop arriving.

Data sources and connectors​

  • Connections belong to data sources. Users see a connection only if they can view its data source.
  • Direct-mode connectors. A connector can hold its own credentials and call its data source directly.
  • Query operation. A connector can declare a query operation, which Kaizen offers to agents as a tool.
  • Content policy (alpha). Core can apply the policies attached to a data source to every connector result, using an Open Policy Agent that you run. Off by default. See Policy.
  • Your own connector images. A connector image in your own private registry is pulled exactly as you name it. See Run Your Own Connector.

Models and apps​

  • vLLM 0.32.0 on NVIDIA and AMD, and vLLM on DGX Spark nodes in tenant mode.
  • Collections remember their embedding model, and searches embed the query with that model.
  • App Garden replicas. A template can set a service's pod count with deploy.replicas.

Changes from 1.3​

Read Upgrade Kamiwaza before you upgrade from 1.3. It has the steps for each change below.

Action required​

  • Back up first. There is no rollback to 1.3. Take a pg_dumpall backup of every Kamiwaza database before you upgrade. See Before you start and No rollback to 1.3.
  • Set global.domain to the DNS name of the platform's HTTPS frontend. The chart refuses to install or upgrade without it.
  • Owner trust. 1.4.0 required the cluster owner's ownerPublicKey and clusterBindingId. From 1.4.1 they are optional. See Owner trust.
  • Inactive local accounts are disabled. With authentication enabled, Core disables local accounts that have not signed in for 35 days. Before you upgrade, list break-glass accounts in core.auth.inactivityDisable.exemptAccounts.
  • Local users' passwords. Keycloak now runs its FIPS build, which cannot verify the argon2 password hashes of users created on 1.3. See Move local users' passwords off argon2.
  • No credentials in Keycloak and OIDC URLs. Core refuses to start if one of these settings contains user:password@. See Keycloak and OIDC URL credentials.
  • Private CAs. Set the edge CA in global.trust.edgeAnchor.pem and other CAs in global.trust.extraAnchors. Delete a kamiwaza-edge-trust ConfigMap that you created by hand. With an edge certificate the chart did not issue, the upgrade stops until an edge CA is set. See TLS Trust for Private Certificate Authorities and Context TLS preflight.
  • External model endpoints verify TLS. core.scheduler.tlsInsecure no longer covers external model hosts. Add their CA before you upgrade. See External model endpoints and tlsInsecure.
  • Content Security Policy is enforced. A component in report-only mode needs acknowledgeReportOnly: true. See Content Security Policy.
  • Connectors at internal addresses need an entry in core.connectors.endpointAllowlist or an administrator's approval. Existing connectors are approved on upgrade. See Connectors at internal addresses.
  • Removed settings. The upgrade stops on global.workroomManager.surface: extension (the Workroom Manager extension is removed), on the extension operator on public installs, and on the retired OpenShift overlay files. See Workroom Manager extension removed, Extension operator not supported on public installs and Upgrade from the OpenShift overlay files.
  • Extension platform gateway. Extensions reach the platform through KAMIWAZA_PLATFORM_GATEWAY_URL on every routing provider. Installs on a custom cluster domain set global.clusterDomain. See Extension platform gateway.

Keycloak and OIDC URL credentials​

When authentication is enabled, Core refuses to start if any of these settings contains embedded credentials (user:password@): AUTH_GATEWAY_KEYCLOAK_URL, AUTH_GATEWAY_KEYCLOAK_PUBLIC_URL, AUTH_GATEWAY_KEYCLOAK_ADMIN_URL, the legacy KEYCLOAK_URL, AUTH_GATEWAY_JWKS_URL, AUTH_GATEWAY_JWT_ISSUER, AUTH_GATEWAY_OIDC_ISSUER, AUTH_GATEWAY_OIDC_AUTHORIZATION_ENDPOINT, AUTH_GATEWAY_OIDC_TOKEN_ENDPOINT, AUTH_GATEWAY_OIDC_USERINFO_ENDPOINT and AUTH_GATEWAY_OIDC_JWKS_URI. The chart never puts credentials in these settings, so only a value set by hand is affected. See Remove credentials from Keycloak and OIDC URLs.

Other changes​

  • Kaizen v3 is retired for new installs. Existing Kaizen v3 deployments are kept. Kaizen v4 installs as a separate deployment. Air-gapped sites must keep the Kaizen v3 images in their registry. See Kaizen v3 on Kamiwaza 1.4.
  • Safer upgrades. An upgrade that would move PostgreSQL data stops before it changes anything, and failed upgrade jobs are kept for inspection. See Upgrade halted: PostgreSQL data directory.
  • Logs redact credentials in URLs.
  • Brotli-compressed dataset files are refused because of a vulnerability in the bundled decoder (CVE-2026-111497). Re-encode such files with ZSTD or Snappy.

Deprecated​

  • global.trust.edgeAnchor.publishedBySubstrate, configMapName and key are deprecated. Use global.trust.edgeAnchor.pem.
  • The cluster/values/kamiwaza-fips.yaml overlay is deprecated because the FIPS profile is now the default.
  • The wrapped result format {"data": ..., "metadata": ...} for jobs with a gated target dataset is deprecated. Read the job result directly.
  • The older platform gateway variables for extensions are deprecated. Use KAMIWAZA_PLATFORM_GATEWAY_URL.

Known issues​

  • A helm upgrade fails after you connect a directory. On a FIPS install (the default), once the field kit's ldap-federation.sh apply has connected an LDAP or Active Directory directory, every later helm upgrade fails in the kamiwaza-keycloak-fips-migration hook. The running install is not affected. To upgrade, run ldap-federation.sh revert, delete the Role kamiwaza-installer-grant-check that the failed hook leaves behind, run helm upgrade, then run ldap-federation.sh apply again. Reverting unlinks directory users from their Kamiwaza accounts; the field kit's "Remove" section explains how to link them again. Found in 1.4.1.
  • Registry garbage collection can delete images. On distribution 2.x registries, garbage collection with --delete-untagged can delete images of the running release. See Clean up the registry safely.
  • Stop in the App Garden permanently deletes a Kaizen v3 deployment. To stop one so that you can bring it back, see Stop a Kaizen v3 deployment without losing it.
  • The App Garden can still show a Kaizen (Legacy) card after the upgrade. Deploying it is refused and creates nothing. See Deploying Kaizen (Legacy) is refused.
  • A recipe catalog makes every deployment it covers fail on 1.4.0. Fixed in 1.4.1. On 1.4.0, leave core.inferenceResources.gpuRecipeCatalogConfigMap and cpuRecipeCatalogConfigMap unset.

Patch releases​

1.4.1​

Easier installs in restricted namespaces and air-gapped sites, Argo CD support, offline model import, Kaizen 0.9.16, and security updates.

Action required

  • Use Helm 3.14 or later. The chart refuses any Helm older than 3.10.0.

What changed

  • Owner trust is optional. The chart installs and upgrades without a cluster owner, and local CPU models run without one. GPU models still need owner-attested mode. See Owner trust levels.
  • Namespace-admin installs. An installer with only the built-in admin role in its namespace can install and upgrade Kamiwaza, including on OpenShift with GreyMatter.
  • Argo CD. You can install and upgrade the chart through Argo CD. See Upgrade through Argo CD.
  • Offline model import. Air-gapped sites can pack a Hugging Face model as an image and import it. See Import a model without internet access.
  • Air-gapped installs. The release chart pins every image by digest, extension images come from your site registry, and the release now also publishes the standard (non-FIPS) images.
  • Resource sizing. Every container sets CPU and memory requests and limits, and sizing files are provided for namespaces with a resource quota.
  • Extension catalog. The catalog shipped with Core is authoritative on upgrade, and the online catalog is off by default.
  • Models. Newer llama.cpp engines, vLLM fixes for GeForce GPUs and tenant mode, and clearer errors when a deployment fails.
  • Security updates to the base images and libraries, including glibc, etcd, SpiceDB, gRPC and Kaizen's dependencies.
  • Bundled extensions: Kaizen 0.9.16, OmniParse 2.5.11, DDE 2.7.5, Graphiti 2.8.17, Milvus service 2.4.35, Connector Builder 0.4.5, Skills Library 0.6.5, and the Google Workspace and Microsoft 365 connectors 0.2.6.
  • Field kit. The air-gapped field kit adds the one-namespace install, a namespace-admin preflight and Argo CD support.

Known issues fixed

  • A recipe catalog no longer makes every deployment it covers fail.
  • Tools on the standalone Envoy edge now receive a signed caller identity.

New known issue

  • A helm upgrade fails after you connect a directory with the field kit. See Known issues.