Skip to main content

Kamiwaza 1.4 release notes

Kamiwaza 1.4 installs with Helm into a Kubernetes cluster that you run. This page covers the changes in 1.4.0 from 1.3.5, including actions to take before upgrading. The latest release in this line is 1.4.0.

Kamiwaza 1.4.0​

New in 1.4.0​

Installation and distribution​

  • FIPS images by default. Every product chart image that has a FIPS build now installs as its FIPS image. This covers Core, the console, Keycloak, the inference engines and the catalog extension images that publish a FIPS build. Images with no FIPS build are listed as recorded exceptions.
  • One security profile setting. global.securityProfile now sets the security profile for Core, the console and the Keycloak realm together. The stig profile requires multi-factor sign-in, refuses LDAP without TLS, requires encrypted storage classes, applies a stricter realm password policy and session limits, and takes a database snapshot before Core migrations.
  • Install mode summary. The install notes and a new admin endpoint list the extension runtime, the trust store and edge anchor state, the transport extensions use to reach the platform, and how workload identity is set. The notes also warn about every setting that turns off TLS certificate verification.
  • Support diagnostics collector. A supported collector gathers logs from every container in the install, including init and debug containers. DataHub and Kafka probes are optional.
  • Air-gapped field kit updates. Each release now ships the namespace-admin prerequisites chart, an OpenShift with GreyMatter profile, a permissions check that runs before Helm, and a scripted way to connect an LDAP or Active Directory directory. The intake list, the file bundle and the pickup runbook now ship with every release.
  • Release documentation in the cluster. The release now ships its own frozen documentation image, and the in-cluster docs follow the installed release.
  • Configurable API minimum replicas. core.api.standalone.autoscaling.minReplicas sets the minimum number of Core API pods. For example, a multi-node cluster can keep two pods running while a node is drained.

Platform​

  • Separate runtime for user jobs. Ray jobs now run in their own runtime, which holds no platform secrets. The runtime is recycled when it is idle.
  • Model trust store for model pods. Model staging and inference engine pods now trust the platform CA and every CA in global.trust.extraAnchors. A registry, mirror or OpenAI-compatible endpoint signed by a private CA now works without turning off verification.
  • Named admission refusals. When OpenShift security constraints or Pod Security refuse a model or app pod, Kamiwaza now fails the deployment right away with K8S_ADMISSION_REFUSED. The message names the refused field and the fix. A pending model deployment also shows why the scheduler cannot place it.
  • Dataset metadata conditional updates. Dataset metadata now returns an ETag and accepts an optional If-Match. A dataset update now keeps the fields the request does not name.
  • Safer user administration. Kamiwaza refuses to remove or demote the last administrator, and you can now change a user's role from Edit User.

Security and identity​

  • Identity providers from Helm values. You can now add customer OIDC identity providers with auth.keycloak.identityProviders[]. They are applied on fresh installs and on upgrades.
  • Private-CA LDAPS. Keycloak now trusts the CAs in global.trust.edgeAnchor.pem and global.trust.extraAnchors[]. An LDAP or Active Directory server that uses ldaps:// with a private CA can now be connected from values alone.
  • Directory users. User lists now show where an external user's account is managed, and LDAP users are no longer offered a password change.
  • Audit administration role. A separate audit-admin role controls audit administration, and platform administrators are kept out of it.
  • Tamper-evident audit records. Audit events now carry a hash chain, and a verifier reports where the chain was changed.
  • Sign-out ends the session. Signing out now invalidates the session's access token and shows a confirmation.
  • Inactive account control is on by default. When authentication is enabled, Core disables local realm accounts that have not signed in for 35 days. Before upgrading, add break-glass Keycloak usernames or account IDs to core.auth.inactivityDisable.exemptAccounts to keep them exempt. The period is configurable with core.auth.inactivityDisable.days. Service accounts, directory accounts and brokered accounts are excluded.
  • Stricter stig sign-in. Under the stig profile, the console redirects to Keycloak to sign in, sessions stay active only while they are in use, and minting a personal access token requires a session that used multi-factor sign-in.
  • Edge TLS policy. The Istio and standalone Envoy edges now accept TLS 1.2 or later with an explicit ECDHE AES-GCM cipher list.
  • Cloud metadata blocked. External egress now blocks the Azure WireServer and GCP IPv6 metadata endpoints.
  • Hardened workloads. Chart-owned workloads now use the RuntimeDefault seccomp profile, and the Core and Keycloak PostgreSQL databases run as non-root.
  • Password policy for local users. On Lite, SAML and certificate installs, Core now applies the platform password policy to the passwords it stores.

Observability and audit​

  • Authenticated OTLP export. Core's OTLP export now supports authentication headers from a Secret, a private CA, and a service identity for each exporting container.
  • Native metrics. Platform components, including every managed PostgreSQL instance and the inference engines, now expose Prometheus metrics behind global.observability.metrics.enabled. Monitoring CRDs are not required.
  • Observability certificates. The observability chart can now use generated certificates, your own Secrets or cert-manager.
  • Audit pipeline alerts. The observability chart now alerts when audit records stop arriving or audit storage is nearly full.
  • Wider audit coverage. Kamiwaza now audits Keycloak sign-ins, SAML, certificate and Lite sign-ins, privileged API actions, and each connector call.

Data sources and connectors​

  • Connections belong to data sources. Every connector connection now belongs to a data source. Users see a connection only if they can view its data source.
  • Direct-mode connectors. A connector can now hold its own credentials and call its data source directly, using any protocol its code supports. That includes destinations outside the cluster.
  • Query verb. A connector can now declare a query operation. Kaizen offers it to agents as a query tool.
  • Content policy (alpha). You can now connect Core to an OPA. Core then applies the policies attached to each data source to every connector result before the result leaves Core. Content policy is off by default. See Policy.
  • Connector updates apply in place. When a connector type is updated, its connections pick up the new image and settings without being recreated.
  • Your own connector images. A connector image in your own private registry, including the OpenShift built-in registry, is now pulled exactly as you name it. See Run Your Own Connector.

Models and GPUs​

  • vLLM 0.32.0. The vLLM engine images for NVIDIA (amd64 and ARM64) and AMD ROCm move to vLLM 0.32.0.
  • vLLM on DGX Spark. Tenant mode now offers vLLM on DGX Spark (GB10) nodes.
  • Collections remember their embedding model. A collection now records the embedding model that wrote it, and searches use that model to embed the query.
  • App Garden replicas. An App Garden template can now set a service's pod count with deploy.replicas. Each pod gets its own address.
  • Reasoning effort. An explicit reasoning_effort: "none" is now sent to models that accept it. When an endpoint does not declare reasoning support, the response now reports the dropped value in X-Kamiwaza-Reasoning-Effort-Dropped.
  • GPT-6 tool calls. Tool calls to GPT-6 through external chat endpoints now work.

App Garden and extensions​

  • Platform gateway for extensions. Direct-runtime extensions now get the in-cluster platform gateway in KAMIWAZA_PLATFORM_GATEWAY_URL on Envoy, GreyMatter and Istio.
  • Accurate App Garden badges. App Garden cards no longer show a governance tier as a scan badge.

Changes from 1.3​

Owner-signed inference required​

Action required for every install and upgrade, including CPU-only clusters. Kamiwaza 1.4 requires tenant-mode inference to be enabled with the cluster owner's ownerPublicKey and clusterBindingId. Without those values, Helm stops before installing or upgrading. Before you install or upgrade, complete Set up tenant-mode inference; for upgrades from 1.3, see Set up owner trust.

Removed and changed​

Platform domain required​

Action required for every install and upgrade. Set global.domain to the DNS name of the platform's HTTPS frontend, such as kamiwaza.example.com. The chart no longer supplies a placeholder domain and refuses to install or upgrade when the value is missing.

Content Security Policy enforced​

The console and API now enforce Content Security Policy in both the default and stig security profiles. Browsers block resources the policy does not allow.

Action required if your release uses report-only mode. Set core.security.csp.acknowledgeReportOnly: true or frontend.security.csp.acknowledgeReportOnly: true on each component that uses report-only mode, or return that component to enforcement. Without the acknowledgement, helm upgrade refuses the report-only setting. This also applies to KAMIWAZA_CSP_MODE overrides in extraEnv; the stig profile refuses that override entirely. See Content Security Policy for the policy and diagnostic settings.

Kaizen v3 no longer installs​

Kamiwaza 1.4 retires Kaizen v3, listed in the App Garden on 1.3 as Kaizen (Legacy) (catalog name kaizen-legacy), for new installations. Kaizen v4, listed as Kaizen (catalog name kaizen), is not affected.

  • New Kaizen v3 deployments are refused from the App Garden, the API, the SDK, the CLI and every catalog, including imported, retained and local ones. Cloning a Kaizen v3 deployment, or creating another one, is refused too. The API answers 409 with code product_retired.
  • Existing Kaizen v3 deployments are kept. Running, stopped and failed deployments keep their ID, data, configuration, storage, route, owner and workroom through the upgrade. A running one keeps running. The deployment's owner or an administrator can restore a stopped or failed one in place with POST /api/apps/deployment/<id>/restore.
  • Kaizen v4 is a separate deployment. It installs beside Kaizen v3 and does not inherit Kaizen v3 agents, conversations, data or configuration. There is no in-place upgrade from Kaizen v3 to Kaizen v4, and no importer.
  • App Garden updates never replace Kaizen v3. Check for updates, Update all and the other version selections never delete, migrate or replace a Kaizen v3 deployment.
  • Deleting a Kaizen v3 deployment is final. A purged deployment cannot be restored, and you cannot install a new Kaizen v3 deployment in its place. Stop in the App Garden's list of deployments deletes it in the same way.
  • Air-gapped installations must keep the Kaizen v3 images in their registry across the upgrade, because a 1.4 release does not include them.

For the details, including how to restore a deployment and which images to keep, see Kaizen v3 on Kamiwaza 1.4.

Workroom Manager extension removed​
  • Action required if your values set global.workroomManager.surface: extension. Kamiwaza 1.4.0 removes the standalone Workroom Manager extension (workroom-manager), deprecated in 1.3. With extension in your values, the upgrade stops before it changes anything, on every ingress provider. Remove the key to keep the default, disabled, or set it to core for the built-in Workrooms page.
  • Delete a standalone extension that still runs. After the upgrade it is hidden and cannot be reached, but its pods keep running until an administrator deletes it with DELETE /api/extensions/<name>.
  • Workrooms are not affected. Workroom records, the workroom APIs, permissions and isolation do not change, and workroom data created through the extension stays available.

See Workroom Manager extension removed.

Extension operator refused on public installs​

Action required if your values set extensionOperator.enabled: true or core.extensionRuntime.mode: operator. The public release has never published an extension-operator image, so on any public install, mirrored and air-gapped ones included, the operator pod could not pull its image. From 1.4.0 the chart refuses extensionOperator.enabled: true when global.imageRegistryPrefix is under ghcr.io/kamiwaza-ai/releases/ and stops before it changes anything. A mirrored or air-gapped install is not refused, so check its values yourself. Remove the value, and core.extensionRuntime.mode: operator if you set it, to keep the default direct runtime. See Extension operator not supported on public installs.

tlsInsecure no longer covers external model hosts​

Action required if Kamiwaza reaches an external model endpoint whose certificate is not trusted. Core now verifies the certificate of every external model host it calls, such as in Context Service and OmniParse model calls, as Kaizen already did (except in its reranker, which now verifies too). core.scheduler.tlsInsecure covers only the platform's own origins (one exception: OmniParse's pod-wide VERIFY_SSL, see core.scheduler.tlsInsecure), and the Context Service and OmniParse TLS settings no longer relax an external host.

  • Before you upgrade, add the endpoint's CA to global.trust.extraAnchors, or, for a trial, list the host in the new core.scheduler.externalTlsInsecureWithDnsRetargetingRisk (Context Service, OmniParse, and Kaizen calls only; never an OmniParse translation endpoint; External Models engines need the CA).
  • If you add or change a CA, restart Core's pods (kubectl -n kamiwaza rollout restart deployment/core-api deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker, plus deployment/core-retrieval-streamer if you enabled it) so they load it.
  • After the upgrade, open and save each OmniParse instance that uses any external endpoint, including a translation endpoint, so Core rebuilds its TLS setting.
  • When you add a CA or add or remove the trial setting, redeploy the extensions that call external models, such as Kaizen. A running extension keeps the old value.

For its own model calls, Core logs an error for each external host that fails verification, naming the host and both fixes. See External model endpoints and tlsInsecure.

Backup and rollback​

  • Action required: back up before you upgrade. Taking a pg_dumpall backup of every Kamiwaza PostgreSQL database before the upgrade is now a required step. See Before you start.
  • Rollback to 1.3 is not supported. 1.4.0 cannot be rolled back in place to any 1.3 release, 1.3.4 included: the 1.3 releases cannot run the 1.4 database schema. Do not run helm upgrade --install with a 1.3 version, or helm rollback, against a 1.4 installation; the attempt takes the platform API down. To return to the release you ran before, restore the pre-upgrade backup with Kamiwaza support. If a 1.3 upgrade command has already run, do not rerun the 1.4 upgrade to recover; contact Kamiwaza support. See No rollback to 1.3.
  • Kaizen v3 backup and restore. A Kaizen v3 deployment can be backed up before or after the upgrade, and restored into the same deployment. See Back up and restore a Kaizen v3 deployment. The earlier backup step, which archived a SeaweedFS directory, applies to Kaizen v4 only.

OpenShift with GreyMatter​

  • A fixed workload identity stops the install. The supported OpenShift security baseline is the default restricted-v2 SCC with global.platform: osgm, where OpenShift assigns each pod's user ID. The legacy restricted (v1) SCC is not supported. Values that would make chart workloads pin a user ID on OpenShift now fail the render with INSTALLATION HALTED, naming the value and the fix, before anything is installed. See Install stops with a workload identity error.
  • The OpenShift overlay files are retired. A values file list that includes kamiwaza-openshift-greymatter.yaml halts the render. Remove it and kamiwaza-openshift.yaml, and set global.platform: osgm, which replaces them; keep the GreyMatter route-publishing values that kamiwaza-openshift.yaml set. See Upgrade from the OpenShift overlay files.
  • Pod identities are cleared on redeploy. Model and app pods that Kamiwaza creates no longer request a user ID, group ID, or fsGroup when OpenShift assigns them. A deployment left with a fixed ID by an earlier install is cleared the next time it is applied, and Core logs IDENTITY_DRIFT for it. Core logs IDENTITY_MODE_MISMATCH at startup when it runs under an OpenShift-assigned user ID but is configured for fixed IDs.

Models and apps​

  • A pod refused by admission fails the deployment at once. When an OpenShift SCC or Kubernetes Pod Security refuses a model or app pod, the deployment fails on the first refusal with K8S_ADMISSION_REFUSED, the refused field, and a remedy, instead of timing out about ten minutes later. Other pod creation failures, such as quota, keep the deadline.
  • Pending models show why. core-api and core-scheduler can now list events in the Kamiwaza namespace in every extension layout, so a model whose pod cannot be scheduled reports the scheduling reason, for example Insufficient cpu.
  • Separate opt-in for unverified external model endpoints. core.scheduler.externalTlsInsecureWithDnsRetargetingRisk turns off certificate verification for external model hosts, all of them or a list of exact, fully qualified hostnames or IPv4 addresses, for trials. It is off by default and warns while it is on. See Configuration.
  • Startup permission check. core-api and core-scheduler check their own permissions on model pods at startup and log one line per missing permission, naming the service account, the namespace, the effect, and the fix. The cluster diagnose report (GET /api/cluster/diagnose, for administrators) lists each gap as an rbac_missing_* (error) or rbac_best_effort_* (warning) issue.

Keycloak and OIDC URL credentials​

  • Action required: the upgrade fails if a Keycloak or OIDC URL setting contains credentials. When authentication is enabled, Core refuses to start if one of these settings carries embedded credentials (user:password@):

    • AUTH_GATEWAY_KEYCLOAK_URL, AUTH_GATEWAY_KEYCLOAK_PUBLIC_URL, AUTH_GATEWAY_KEYCLOAK_ADMIN_URL, and the legacy KEYCLOAK_URL when AUTH_GATEWAY_KEYCLOAK_URL is not set. Any @ or %40 in these values is refused, because no supported value contains one.
    • AUTH_GATEWAY_JWKS_URL and AUTH_GATEWAY_JWT_ISSUER when set explicitly, and the external OIDC endpoints AUTH_GATEWAY_OIDC_ISSUER, AUTH_GATEWAY_OIDC_AUTHORIZATION_ENDPOINT, AUTH_GATEWAY_OIDC_TOKEN_ENDPOINT, AUTH_GATEWAY_OIDC_USERINFO_ENDPOINT and AUTH_GATEWAY_OIDC_JWKS_URI. Only credentials before the host are refused here; an @ in the path or query of a well-formed URL is accepted.

    The chart renders these settings without credentials, so only a value set by hand is affected. Every Core process that starts authentication, such as core-api, exits at startup and restarts in a loop, logging this error. It lists every offending setting, separated by commas, and never their values:

    RuntimeError: Refusing to start: <SETTING>[, <SETTING>...] contains '@' (or '%40'), which means embedded credentials (user:password@) or a mistyped address. Credentials never belong in these URLs and would be written to logs. Remove the 'user:password@' part from the setting (core configmap or env override) and restart; Core authenticates to Keycloak with its own client and admin credentials.

    Before you upgrade, remove the credentials from these settings, and set client credentials in AUTH_GATEWAY_KEYCLOAK_CLIENT_ID and AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET, or AUTH_GATEWAY_OIDC_CLIENT_ID and AUTH_GATEWAY_OIDC_CLIENT_SECRET. See Remove credentials from Keycloak and OIDC URLs. Federation peer JWKS addresses, trusted shared issuers, and connection strings that carry a password by design, such as a Redis URL, are not refused.

  • Logs redact URL credentials. Every Kamiwaza process now rewrites scheme://user:password@ to scheme://<redacted>@ in log messages, tracebacks, stack info, JSON log fields, OpenTelemetry log export, and uncaught-exception span data. For a URL that carries credentials, the host and path are kept. Side effects:

    • A connection string's password is redacted too, and a pre-masked user:***@ becomes <redacted>@, so log searches that match the old form need updating.
    • Some URLs without credentials are over-redacted: a host name without a dot, with a port, whose first path segment contains @ (http://keycloak:8080/keys@tenant is logged as http://<redacted>@tenant), and a URL with no path whose query contains @ (https://idp.example?a=b@c is logged as https://<redacted>@c).
    • Core's startup lines for the JWKS and issuer URLs show a URL that contains @ or %40 as (withheld: the URL contains '@' or '%40'; remove embedded credentials).

    Some passwords cannot be told apart from an ordinary URL with @ in its path. In free text these are redacted only in part, or not at all:

    • partly redacted: a password that contains both @ and /, or more than one x:/ sequence;
    • not always redacted: a password that contains whitespace, a double quote, a backslash, <, or >; an all-digit password followed by / after a username that contains a dot (svc.user:123/x@host), which reads exactly like idp.example:8443/keys@tenant; and credentials written without a scheme (admin:secret@keycloak:8080).

    The redaction also does not cover URL attributes that OpenTelemetry auto-instrumentation records on client spans, exceptions recorded on other spans, or a process that does not load Kamiwaza's logging. Removing credentials from the URL settings is the fix; the redaction is a backstop.

Upgrades and PostgreSQL​

  • Action required if you created kamiwaza-edge-trust by hand. The chart is now the only writer of the kamiwaza-edge-trust ConfigMap, which carries the CA that signs the Kamiwaza domain to Kamiwaza's workloads. Set that CA in global.trust.edgeAnchor.pem, and any other CA in global.trust.extraAnchors, then delete the hand-made ConfigMap right before you upgrade. While it exists, an upgrade that sets either value stops with ConfigMap kamiwaza-edge-trust already exists in namespace kamiwaza and is not managed by Helm. global.trust.edgeAnchor.publishedBySubstrate, .configMapName, and .key are deprecated and are removed in 1.5.0. See TLS Trust for Private Certificate Authorities.

  • Action required on GreyMatter and BYO edge certificates: Context model calls verify TLS. The Context Service now verifies the certificate on its embedding and LLM calls to the platform's own origins, upgraded installations included. When the edge certificate is minted outside the chart or from a CA it does not hold (OpenShift with GreyMatter; Istio or Envoy with network.tls.mode: byo, including EZUA, or customerCA, or cert-manager with a custom issuer, or generated when the chart does not generate it) and no edge CA is published, the install or upgrade stops with the Context TLS preflight error until you set one of global.trust.edgeAnchor.pem (recommended), global.trust.edgeAnchor.publiclyTrusted: true (new; the edge certificate chains to a public CA), or core.context.embedding.verifySsl: false (new; verification off, as before 1.4.0). The install notes and Core's startup log now name every Core value and setting that turns TLS verification off; extensions name theirs on their deployment status. See Upgrade halted: Context TLS preflight.

  • Action required if you created local users: Keycloak runs its FIPS build. It cannot verify the argon2 password hashes that every user created with a password on 1.3 has. Before Keycloak changes, the upgrade's kamiwaza-keycloak-fips-migration job sets a pbkdf2 hash policy, moves the chart's own accounts, and stops the upgrade, naming each account, while any argon2 password remains. Set the policy yourself on 1.3 and have each user sign in once before you upgrade, as described in Move local users' passwords off argon2. auth.keycloak.fips.allowArgon2Lockout: true (new) lets the upgrade continue and locks those users out until an administrator resets their passwords. See Upgrade halted: Keycloak passwords (argon2).

  • An upgrade that would move PostgreSQL data halts. When an upgrade would start a Kamiwaza PostgreSQL StatefulSet on a different data directory than the one it uses now, helm upgrade stops with UPGRADE HALTED (HR-9) and names the values that keep the current layout. See Upgrade halted: PostgreSQL data directory.

  • PostgreSQL ownership check. Each Kamiwaza PostgreSQL pod runs a check-pgdata-ownership init container. When the data directory belongs to another user ID, it names both IDs and the likely cause in kubectl describe pod, instead of PostgreSQL failing with data directory has wrong ownership. The new init container restarts the PostgreSQL pods once during the upgrade.

  • Failed upgrade jobs are kept. The core-upgrade-migration and core-db-init-cleanup jobs are no longer deleted when they fail, so you can inspect them with kubectl describe job. The next run of the job replaces it.

Extension platform gateway​

  • One platform gateway variable on every routing provider. Extensions get KAMIWAZA_PLATFORM_GATEWAY_URL, with KAMIWAZA_GATEWAY_CA_FILE when the platform publishes a CA and, on GreyMatter, KAMIWAZA_GATEWAY_TLS_SERVER_NAME. On Istio, extensions now reach the platform in-cluster through the ingress gateway instead of through the public origin; before 1.4, Istio installs gave extensions no in-cluster gateway. Most chart installs need no values change. With the extension operator (not supported on public installs; see Extension operator refused on public installs), an upgrade can stop on the install identity's rights in the gateway namespace, on a missing gateway namespace or DestinationRule CRD, or on a serviceHostname form it no longer accepts, and a namespace-admin install needs a Role from a cluster administrator; see Extension operator on Istio. See also Platform Gateway for Extensions.
  • Deprecated: KAMIWAZA_INTERNAL_GATEWAY_URL, KAMIWAZA_TRAEFIK_GATEWAY_HOST, and KAMIWAZA_INTERNAL_ORIGIN as a gateway. They are still set through 1.4.x and removed after it.
  • Action required if you edited Core's configuration by hand: Core refuses to start without a valid KAMIWAZA_ORIGIN. On Kubernetes, core-api goes into CrashLoopBackOff unless KAMIWAZA_ORIGIN is an http(s) URL with a host. The chart always sets it from global.domain.
  • Action required if KAMIWAZA_INTERNAL_GATEWAY_URL on Core has a path. Extension deploys are refused, and GET /api/kamiwaza/install-modes reports invalid, until you remove the path. Such values already broke extension calls with 404.
  • Action required on a custom cluster domain. In-cluster gateway hosts now follow the cluster domain instead of .svc.cluster.local. Clusters on cluster.local see no change. On another domain, set global.clusterDomain. On Envoy, a certificate you provide must also carry kamiwaza-envoy.<namespace>.svc.<domain>. On GreyMatter, chart installs keep verifying the edge against global.domain; without the chart, the edge certificate must carry <edge-service>.<namespace>.svc.<domain> unless KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME is set. See TLS names for the platform gateway.
  • Extension pods restart once after the upgrade, when Core applies them again with the new variables. No action is needed.
  • Remove hand-set gateway overrides, such as TASKS_PLATFORM_GATEWAY_URL, that point at the address Kamiwaza now sets.

See Extension platform gateway in the upgrade guide.

Security​

  • Connector endpoints cannot reach internal addresses unless you admit them. Kamiwaza checks a connector's endpoint when it is registered and on every later call, and connects only to the address it checked. Private, CGNAT and other non-public addresses need a narrow entry in the new Helm value core.connectors.endpointAllowlist, or an administrator's approval of one host:port for one connector. Loopback, link-local and cloud metadata addresses are refused even then. Registration failures no longer reveal what is listening at an address. Existing connectors are approved on upgrade ("Grandfathered on upgrade"), and connectors Kamiwaza deployed itself keep working. Administrators review refused destinations, and allow one for its connector, under Settings > Connectors > Blocked connector destinations. See Connector Destinations and Connectors at internal addresses.
  • The stig profile enforces the extension runtime Content-Security-Policy. Under global.securityProfile: stig, an unset global.extensionRuntimeCsp.mode now enforces on the extension Istio sidecars, the standalone Envoy edge and, when the chart renders its GSL bundle, the GreyMatter edge. Report-only under stig needs global.extensionRuntimeCsp.acknowledgeReportOnly: true. Upgrade Kaizen first, to the release in the 1.4 App Garden catalog. See Extension runtime policy under stig.
  • Signed-out login works on the standalone Envoy edge without a round trip to the authorization check. The edge sends the /api/... entries of global.authz.extAuthz.exemptPaths straight to the Kamiwaza API, and refuses to render an entry it cannot serve. No path becomes newly public. The edge also removes a client-sent X-User-MCP-Admission header. See Public paths at the edge.
  • Policy on connector results (alpha). Administrators can drop records and redact fields from what a connector returns, based on the user's declared attributes and the content. An Open Policy Agent you run evaluates the rules, and pulls them from Kamiwaza. Kamiwaza ships two templates, content_filter and field_needs_attribute, and you can add your own. Policy is off by default, and fails closed: when it can't be applied, the result is withheld. The API is alpha and may change. See Policy.
  • Brotli-compressed dataset files are refused. The Brotli decoder bundled with pyarrow has an out-of-bounds write (CVE-2026-111497), and no fixed pyarrow release exists yet. Retrieval therefore refuses to read a Parquet file when any of its columns uses Brotli compression, and a CSV or JSON file whose name ends in .brotli. Core checks each Parquet file's metadata first, so no Brotli data is decoded. A retrieval request is refused with HTTP 400, and the error detail starts with retrieval_brotli_refused and names the file and the reason. A refusal after a stream has started ends the stream with an error event carrying the same message and marks the retrieval job failed, and an Arrow Flight stream returns the message too. To read such a dataset, re-encode the file with ZSTD or Snappy compression outside Kamiwaza, then register or upload it again. Files that use other codecs, and text and binary datasets, are unaffected. The refusal applies to every Core image, the FIPS image included.
  • Parquet streamed over Arrow Flight is read in process, so that the same Brotli check applies. Flight now reads Parquet with the same checked reader as inline retrieval, instead of distributing it over Ray. File selection, hive-style key=value partition columns, and schema unification across files behave as before. Flight now reads a glob location, such as s3://bucket/data/*.parquet, as a literal path, as inline retrieval already does.
  • The FIPS Core image builds pyarrow with Brotli 1.2.1, which fixes CVE-2026-111497.
  • The extension operator checks each extension's trust bundle (on installs that run the operator, which public installs do not). An anchors-only, missing or invalid kamiwaza-trust-bundle sets the extension's TrustBundleReady condition to False with a message that names the fix. By default this is a warning and the extension is not failed, because installations without internet access legitimately run an anchors-only bundle. The new value extensionOperator.extensions.trust.requireCompleteBundle: true fails the extension instead. See Extension trust bundle.

Deprecated​

  • global.trust.edgeAnchor.publishedBySubstrate, configMapName and key are deprecated. Use global.trust.edgeAnchor.pem.
  • The cluster/values/kamiwaza-fips.yaml overlay is deprecated because the FIPS profile is now the default.
  • The wrapped result format {"data": ..., "metadata": ...} for jobs with a gated target dataset is deprecated. Read the job result directly.
  • The older platform gateway variables for extensions are deprecated. Use KAMIWAZA_PLATFORM_GATEWAY_URL.

Known issues​

  • No extension operator on public installs. The extension-operator runtime is not supported on public 1.4.0 installs. This includes mirrored and air-gapped installs from the public chart. Use the direct extension runtime.
  • Registry garbage collection can delete images. On distribution 2.x registries, garbage collection with --delete-untagged deletes the platform manifests of multi-platform images, including the images of the running release. Follow the safe cleanup steps in Upgrade Kamiwaza.
  • Tools on the standalone Envoy edge can rely only on edge authorization. On global.ingress.provider: envoy, Envoy authenticates the caller and authorizes the tool and workroom, but the tool does not receive a signed caller identity, so a tool that verifies the identity envelope or checks caller roles refuses every request. On Istio, tools receive the signed identity. Tracked for 1.4.1.
  • Stop in the App Garden's list of deployments permanently deletes a Kaizen v3 deployment. It deletes the deployment, its storage volume claims and, with the default Delete reclaim policy, its data. Kamiwaza 1.4 cannot install Kaizen v3 again, so the deployment cannot be restored afterwards. To stop one so that you can bring it back, use DELETE /api/apps/deployment/<id>, then restore it with POST /api/apps/deployment/<id>/restore. See Stop a Kaizen v3 deployment without losing it.
  • The App Garden can still show a Kaizen (Legacy) card after the upgrade. Deploying it is refused with "Kaizen v3 (kaizen-legacy) is retired on Kamiwaza 1.4 and later: new installations are refused…" and creates nothing. Existing Kaizen v3 deployments are not affected. Install Kaizen v4 (kaizen) as a separate deployment. See Deploying Kaizen (Legacy) is refused.