Version: 1.3.3 (Latest)

Install on Kubernetes with Istio

This page installs Kamiwaza into one namespace of a Kubernetes cluster that runs an Istio service mesh. You run a single helm upgrade --install as a namespace-scoped identity. The chart creates only namespaced objects.

Before you start

You need	Notes
A prepared cluster	Every check in Platform Prerequisites passes, and you have the facts it told you to record.
The installer kubeconfig	The namespace-scoped identity from Installer identity. Never install with a cluster-admin kubeconfig.
`kubectl`, and Helm 3.14 or later	Helm pulls the chart from an OCI registry. Helm 4 also works.
`jq` and `curl`	Used by the verification steps.
A license file	`license.lic`, issued by Kamiwaza. See License File.

No internet access? Do Install Without Internet Access first. It copies the release into your registry and gives you values to merge into your values file in step 2. In step 3, it has you check the values file before you install, and use the chart from your registry.

Point kubectl and helm at the installer identity for every command on this page:

export KUBECONFIG=<path to the installer kubeconfig>

Kamiwaza installs into the kamiwaza namespace. The examples use kamiwaza.example.com as the domain; substitute your own.

Step 1: Add the license

Create a Secret from the license file. The key must be license.lic:

kubectl -n kamiwaza create secret generic kamiwaza-license \
  --from-file=license.lic=./license.lic

The chart refuses to render until a license is configured and the EULA is accepted. You do both in the values file in the next step.

Step 2: Write your values file

Your values file carries this cluster's facts and your acceptance of the EULA. Everything else is the chart's defaults. Keep the file somewhere safe and outside any shared repository; you will need it again for upgrades.

Required values

Value	Set it to
`global.eula.accepted`	`true`, after reading the EULA. The render fails without it.
`core.license.existingSecret`	`kamiwaza-license`, the Secret from step 1. The render fails without it.
`global.domain`	The Kamiwaza domain the gateway serves.
`global.namespaces.platform`, `.extensions`, `.sandboxes`, `.observability`, `.ca`, `.registry`, `.system`	`kamiwaza`, for each one. Everything runs in that one namespace.
`global.namespaces.mesh`	The namespace istiod runs in, usually `istio-system`.
`global.storage.stateful.standard`, `.postgres`, `.etcd`, `.registry`, `.extensions`	The StorageClass you recorded, for each one.
`core.extensionRuntime.kubernetesApi.egressCidrs`	Every Kubernetes API Service address and every endpoint address you recorded, each as `/32` (IPv4) or `/128` (IPv6). Copy them exactly: a wrong address installs cleanly and then fails at runtime. The render fails without this value.
`global.ingress.gateway.serviceAccountName`	The gateway workload's service account you recorded. A gateway installed with Istio's Helm chart as `istio-ingressgateway` uses `istio-ingressgateway`; one installed with `istioctl` uses `istio-ingressgateway-service-account`. A wrong value makes Kamiwaza reject the gateway's traffic.

Values that depend on your cluster

Value	Set it when
`global.ingress.gateway.name`, `.namespace`	The Gateway resource is not `kamiwaza-gateway` in `istio-system`.
`global.ingress.gateway.selector`	The Gateway's `spec.selector` is anything but exactly `istio: ingressgateway`, the default. List all of its labels.
`global.ingress.gateway.serviceHostname`	The gateway Service is not `istio-ingressgateway.istio-system.svc.cluster.local`.
`global.istio.classicInjection`	Your mesh injects `istio-proxy` as a regular container, not a native sidecar. Set `true`. On such a mesh, leaving it `false` makes the install hang on its setup jobs.
`core.scheduler.extraEnv`	The edge certificate is self-signed. Add the `KAMIWAZA_ALLOW_INSECURE_INTERNAL_RETRY` entry exactly as shown in the example below. Omit it for a CA-issued certificate.
`global.imagePullSecrets`	Images come from a registry that needs credentials. List the pull Secret you created, for example `[{name: kamiwaza-pull}]`.
`core.inferenceResources.enabled`, `.bundleConfigMap`, `.ownerPublicKey`, `.clusterBindingId`	You serve models on GPUs. Set them after the cluster owner publishes a signed profile, as in Tenant-mode inference. Without them, a GPU deployment is refused. Prepare the nodes first; see GPU nodes.
`core.extensionRuntime.kubernetesApi.requireEgressCidrs`	Your CNI enforces NetworkPolicy before Service address translation and you choose not to list the API addresses. Set `false`. Most clusters, including k0s and OpenShift, need the addresses.

Optional values

Value	Notes
`registry.persistence.size`, `seaweedfs.persistence.size`	Volume sizes for the model registry cache (default `500Gi`) and the built-in object store (default `100Gi`). Lower them when storage is small; see Block storage. Set them before the first install; they can't be reduced later.
`core.userDefaults.adminPassword`	The password for the `admin` user. Leave it unset and Kamiwaza generates one, which you read back in step 4. Setting it here stores it in the Helm release. It applies at first install only; changing it later does not rotate the password.

Example

global:
  domain: kamiwaza.example.com
  eula:
    accepted: true
  namespaces:
    platform: kamiwaza
    extensions: kamiwaza
    sandboxes: kamiwaza
    observability: kamiwaza
    ca: kamiwaza
    registry: kamiwaza
    system: kamiwaza
    mesh: istio-system
  storage:
    stateful:
      standard: standard
      postgres: standard
      etcd: standard
      registry: standard
      extensions: standard
  ingress:
    gateway:
      serviceAccountName: istio-ingressgateway   # the service account you recorded

core:
  license:
    existingSecret: kamiwaza-license
  extensionRuntime:
    kubernetesApi:
      egressCidrs:
        - 10.96.0.1/32        # kubectl -n default get service kubernetes
        - 192.168.10.15/32    # the kubernetes EndpointSlice addresses

  # Only for a self-signed edge certificate:
  # scheduler:
  #   extraEnv:
  #     - name: KAMIWAZA_ALLOW_INSECURE_INTERNAL_RETRY
  #       value: "true"

Save it as kamiwaza-values.yaml.

Step 3: Install

Record the start time first. The verification step uses it to ignore events from before this install.

INSTALL_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)

helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <version> \
  --namespace kamiwaza \
  --values kamiwaza-values.yaml \
  --wait --wait-for-jobs --timeout 30m

<version> is the Kamiwaza version you are installing, for example 1.3.3.

Do not add --create-namespace. The namespace belongs to the platform administrator.
Helm may print Pod Security restricted warnings, EnvoyFilter exposes internal implementation details warnings, and WARNING: core cannot see GPUs on this install. until tenant-mode inference is on. All are expected.
A typical install takes 5 to 15 minutes, most of it pulling images.

Install and upgrade with Helm, not rendered manifests

Always install and upgrade with helm upgrade --install against the cluster. Rendering the chart with helm template or --dry-run and applying the output, for example through Argo CD or Flux, is not supported. On first install the chart generates the platform's encryption keys, signing keys, and database passwords, and on each upgrade it reads them back from the cluster to keep them. A rendered chart cannot read the cluster, so every render generates new ones. Applying them makes existing encrypted data and access tokens unreadable.

To upgrade to a later release in the same line, rerun the command in this step with the new --version and the same values file.

Step 4: Verify

Check what the platform does, not only its status fields.

Pods are running and ready:

kubectl -n kamiwaza get pods --no-headers \
  | awk '$3 != "Completed" { split($2, r, "/"); if ($3 != "Running" || r[1] != r[2]) print }'
# Expect no output

An Error pod whose Job later completed is a retried attempt, not a failure. Confirm with kubectl -n kamiwaza get jobs: every Job should show all completions.

No image pulls failed during this install. Older events in the namespace are ignored. Run this in the same shell as step 3, or set INSTALL_START to the time you started the install, in the same format:

kubectl -n kamiwaza get events -o json | jq --arg start "$INSTALL_START" '
  [.items[]
   | select((.lastTimestamp // .eventTime // .metadata.creationTimestamp) >= $start)
   | select((.message // "") | test("ErrImagePull|ImagePullBackOff|Failed to pull image|Back-off pulling image"))
  ] | length'
# Expect 0

The platform answers at its domain:

curl -sk -o /dev/null -w '%{http_code}\n' https://kamiwaza.example.com/       # expect 200
curl -sk -o /dev/null -w '%{http_code}\n' https://kamiwaza.example.com/api/   # expect 401

Log in. If you did not set an admin password, read the generated one:

kubectl -n kamiwaza get secret kamiwaza-user-admin \
  -o jsonpath='{.data.password}' | base64 -d; echo

Open https://kamiwaza.example.com/ and log in as admin. With a certificate your browser does not trust, accept the warning first.

admin is the Kamiwaza application administrator. It is not the installer identity and not the Keycloak management account.

The platform is installed. Deploying models and apps is covered in the Quickstart.

Troubleshooting

The install stops before creating anything. The chart checks its values before rendering. A missing EULA acceptance or license prints INSTALLATION HALTED; a missing egressCidrs value names the value it needs. Nothing was installed. Fix the values file and rerun step 3.

no Kamiwaza license is configured, with the license set. The license value belongs under core:. Check that it is core.license.existingSecret, not license.existingSecret. To see values Helm received but no chart reads, run helm get values kamiwaza -n kamiwaza.

The install hangs for about 15 minutes, then fails on a post-install job. This is almost always a cascade from core-scheduler not starting. Check its logs first: kubectl -n kamiwaza logs deploy/core-scheduler. A license problem shows here as License check failed; see License File. While the job runs, every helm upgrade fails with another operation (install/upgrade/rollback) is in progress.

Setup jobs stay at 1/2 ready and the install never finishes. The mesh uses classic sidecar injection. Set global.istio.classicInjection: true and rerun step 3.

Pods are Pending on their volumes. A StorageClass named in the values does not exist or cannot provision. Compare kubectl get storageclass with global.storage.stateful.*.

Images fail to pull with unauthorized. The pull Secret named in global.imagePullSecrets does not exist in the namespace, or has the wrong credentials. Compare it with kubectl -n kamiwaza get secret.

The platform returns 404 at its domain. The Gateway's hosts do not include global.domain, or global.ingress.gateway.* does not name your Gateway. See Istio and the ingress gateway.

Apps install but cannot reach the Kubernetes API. egressCidrs is missing the endpoint addresses. Add every address from Kubernetes API addresses and rerun step 3.

An app returns "Page Not Found", or the platform's own page. Wait about 40 seconds after its pods are ready. If the app stays deployed with no route, delete it and deploy it again.

A GPU model deployment fails with HTTP 422 and Core cannot see a GPU on this install. Tenant-mode inference is off, which is the default. Prepare the GPU nodes, then turn it on; see Tenant-mode inference.

Next steps

Quickstart: start using Kamiwaza.
License File: check and rotate the license.
Uninstall.

Before you start​

Step 1: Add the license​

Step 2: Write your values file​

Required values​

Values that depend on your cluster​

Optional values​

Example​

Step 3: Install​

Install and upgrade with Helm, not rendered manifests​

Step 4: Verify​

Troubleshooting​

Next steps​