Skip to main content

Install on Kubernetes with Istio

This page installs Kamiwaza into one namespace of a Kubernetes cluster that runs an Istio service mesh. You run a single helm upgrade --install as a namespace-scoped identity. The chart creates only namespaced objects.

Before you start​

You needNotes
A prepared clusterEvery check in Platform Prerequisites passes, and you have the facts it told you to record.
The installer kubeconfigThe namespace-scoped identity from Installer identity. Never install with a cluster-admin kubeconfig.
kubectl, and Helm 3.14 or laterHelm pulls the chart from an OCI registry. Helm 4 also works.
jq and curlUsed by the verification steps.
A license filelicense.lic, issued by Kamiwaza. See License File.

No internet access? Do Install Without Internet Access first. It copies the release into your registry and gives you values to merge into your values file in step 2. In step 3, it has you check the values file before you install, and use the chart from your registry.

Point kubectl and helm at the installer identity for every command on this page:

export KUBECONFIG=<path to the installer kubeconfig>

Kamiwaza installs into the kamiwaza namespace. The examples use kamiwaza.example.com as the domain; substitute your own.

Step 1: Add the license​

Create a Secret from the license file. The key must be license.lic:

kubectl -n kamiwaza create secret generic kamiwaza-license \
  --from-file=license.lic=./license.lic

The chart refuses to render until a license is configured and the EULA is accepted. You do both in the values file in the next step.

Step 2: Write your values file​

Your values file carries this cluster's facts and your acceptance of the EULA. Everything else is the chart's defaults. Keep the file somewhere safe and outside any shared repository; you will need it again for upgrades.

Required values​

ValueSet it to
global.eula.acceptedtrue, after reading the EULA. The render fails without it.
core.license.existingSecretkamiwaza-license, the Secret from step 1. The render fails without it.
global.domainThe Kamiwaza domain the gateway serves.
core.inferenceResources.enabled, .ownerPublicKey, .clusterBindingId, .bundleConfigMapenabled defaults to true; leave it there, because false fails the render. From 1.4.1, leave the other three unset: the install needs no cluster owner, local CPU models (including the default embedding model) run under Kamiwaza's built-in CPU baseline profile, which is not owner-attested, and GPU models are refused; see Owner trust levels. Optional hardening, and what GPU models need: owner-attested mode, with a profile the cluster owner signs; see Move to owner-attested mode. If you set ownerPublicKey and clusterBindingId, set both: with only one, an install into an existing namespace stops with owner trust is half-configured: .... On 1.4.0 both are required, and the render fails without them: leaving them unset needs 1.4.1 or later, so check the --version in your install command; see Owner trust on 1.4.0. To serve models on GPUs, prepare the nodes first; see GPU nodes.
global.namespaces.platform, .extensions, .sandboxes, .observability, .ca, .registry, .systemkamiwaza, for each one. Everything runs in that one namespace.
global.namespaces.meshThe namespace istiod runs in, usually istio-system.
global.storage.stateful.standard, .postgres, .etcd, .registry, .extensionsThe StorageClass you recorded, for each one.
core.extensionRuntime.kubernetesApi.egressCidrsEvery Kubernetes API Service address and every endpoint address you recorded, each as /32 (IPv4) or /128 (IPv6). Copy them exactly: a wrong address installs cleanly and then fails at runtime. The render fails without this value.
global.ingress.gateway.serviceAccountNameThe gateway workload's service account you recorded. A gateway installed with Istio's Helm chart as istio-ingressgateway uses istio-ingressgateway; one installed with istioctl uses istio-ingressgateway-service-account. A wrong value makes Kamiwaza reject the gateway's traffic.

Values that depend on your cluster​

ValueSet it when
global.ingress.gateway.name, .namespaceThe Gateway resource is not kamiwaza-gateway in istio-system.
global.ingress.gateway.selectorThe Gateway's spec.selector is anything but exactly istio: ingressgateway, the default. List all of its labels.
global.ingress.gateway.serviceHostnameThe gateway Service is not istio-ingressgateway.istio-system.svc.cluster.local.
global.istio.classicInjectionYour mesh injects istio-proxy as a regular container, not a native sidecar. Set true. On such a mesh, leaving it false makes the install hang on its setup jobs.
global.trust.edgeAnchor.pemThe edge certificate is not issued by a publicly trusted CA. Set it to the contents of edge-ca.crt from Edge certificate: the CA chain, or for a self-signed certificate the certificate itself. Do not create the kamiwaza-edge-trust ConfigMap yourself.
global.trust.edgeAnchor.publiclyTrustedThe edge certificate is issued by a publicly trusted CA and you leave network.tls.mode at its default, byo. Set true. Without it or edgeAnchor.pem, the install stops with the Context TLS preflight error; see Edge certificates minted outside the chart.
core.scheduler.extraEnvThe edge certificate is self-signed. Uncomment the KAMIWAZA_ALLOW_INSECURE_INTERNAL_RETRY entry in the example file. Omit it for a CA-issued certificate.
global.imagePullSecretsImages come from a registry that needs credentials. List the pull Secret you created, for example [{name: kamiwaza-pull}].
core.extensionRuntime.kubernetesApi.requireEgressCidrsYour CNI enforces NetworkPolicy before Service address translation and you choose not to list the API addresses. Set false. Most clusters, including k0s and OpenShift, need the addresses.

Optional values​

If the installer holds only the built-in admin ClusterRole, the allocationVerification row is required, not optional; see Installers that hold only admin.

ValueNotes
registry.persistence.size, seaweedfs.persistence.sizeVolume sizes for the model registry cache (default 500Gi) and the built-in object store (default 100Gi). Lower them when storage is small; see Block storage. Set them before the first install; they can't be reduced later.
core.userDefaults.adminPasswordThe password for the admin user. Leave it unset and Kamiwaza generates one, which you read back in step 4. Setting it here stores it in the Helm release. It applies at first install only; changing it later does not rotate the password.
core.inferenceResources.allocationVerificationreadinessGate (default) or label. Set label when the installer holds only the built-in admin ClusterRole and can't be granted patch on pods/status; see Installers that hold only admin.

Example​

The chart ships an example values file for this environment, examples/kubernetes-istio.yaml. Pull the chart for the version you install and copy the file:

helm pull oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza --version 1.4.1 --untar
cp kamiwaza/examples/kubernetes-istio.yaml kamiwaza-values.yaml

Without internet access, pull the chart from your registry instead.

Then edit kamiwaza-values.yaml: replace this cluster's facts as the tables above list, and set global.eula.accepted to true once you have read the EULA. The file comments out the settings that depend on your edge certificate: uncomment one of the two global.trust.edgeAnchor blocks, and for a self-signed certificate, the core.scheduler.extraEnv entry.

Step 3: Install​

Record the start time first. The verification step uses it to ignore events from before this install.

INSTALL_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)

helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version <version> \
  --namespace kamiwaza \
  --values kamiwaza-values.yaml \
  --timeout 70m

<version> is the Kamiwaza version you are installing, for example 1.4.1.

  • Do not add --create-namespace. The namespace belongs to the platform administrator.
  • Helm may print Pod Security restricted warnings and EnvoyFilter exposes internal implementation details warnings. Both are expected.
  • A typical install takes 5 to 15 minutes, most of it pulling images.

To upgrade to a later release in the same line, rerun the helm upgrade --install command above with the new --version and the same values file. To upgrade from 1.3, see Upgrade Kamiwaza.

Install with Argo CD​

From 1.4.1 you can install with Argo CD in place of the helm command above. From that release the chart generates no key, password or certificate while it renders, except those listed under Limits. A Job in your namespace, kamiwaza-namespace-prepare, creates each one once and never replaces one that exists. Argo CD runs it as a PreSync hook before every sync. Flux's Helm controller runs Helm itself, so it behaves as the helm command.

Read these first:

  • Never sync 1.4.0 or earlier with Argo CD, or with any other tool that applies a rendered chart. Those charts generated new keys and passwords in every render, and applying them made stored credentials and access tokens unreadable.
  • On an Istio mesh, an upgrade from a Helm-installed 1.3 through Argo CD is not supported in 1.4.1: upgrade with Helm first, then move the installation to Argo CD. See Upgrade through Argo CD.
  • On OpenShift with GreyMatter, an upgrade from 1.3 through Argo CD is not supported in 1.4.1 either: upgrade with Helm first. See that install page.
  • Do not install or upgrade by applying the output of helm template with kubectl apply, at any version: the chart's Jobs then run in no order, and nothing creates the keys before the workloads need them.
  • Leave DataHub (datahub.enabled: false) and the observability stack off under Argo CD in 1.4.1, in a fresh install and when you move an existing installation: their charts still generate Secrets in every render. See Limits.
  • This was tested with Argo CD 3.5.7 (annotation tracking, client-side apply) on one-node Kubernetes (k0s 1.36, arm64) with the standalone Envoy edge and no Istio. It was not tested with Istio. It was not tested on OpenShift with GreyMatter. These pages have no prerequisites or values page for the standalone Envoy edge: a site on that edge takes both from its Kamiwaza field guide and uses this section for the Argo CD steps only. The lab served the 1.4.1 chart from an HTTP Helm repository and passed the values as helm.values; the OCI registration and the valuesObject form shown below were not used with 1.4.1. The lab ran the default security profile; the hardened profile was not run under Argo CD. See Limits.

For a fresh install:

  1. Prepare as for a Helm install. Complete Platform Prerequisites, step 1 and step 2. Before the first sync, these must exist:

    • the kamiwaza namespace;
    • the resource metrics API (metrics-server) on the cluster: kubectl -n kamiwaza top pods must not answer Metrics API not available (a list, or No resources found, means it is there). Without it every sync ends Failed on the core-api autoscaler; see Limits;
    • in it, the kamiwaza-license Secret from step 1;
    • in it, your pull Secret, if global.imagePullSecrets names one;
    • the edge certificate Secret from Edge certificate, in the gateway's namespace. On the standalone Envoy edge it is the Secret that network.tls.secretName names, in the kamiwaza namespace;
    • the cluster owner's signed profile ConfigMap, the one core.inferenceResources.bundleConfigMap names. Where the cluster owner anchored the profile on the namespace, this must be the name in the anchor. Argo CD's render cannot read the namespace or the ConfigMap, so nothing checks this before the sync.

    The edge certificate's CA is a value, global.trust.edgeAnchor.pem, not an object: do not create the kamiwaza-edge-trust ConfigMap. Leave network.tls.mode at its default, byo.

  2. Confirm who applies the chart. Argo CD's application controller applies the chart and creates its Roles, with its own identity. That identity belongs to the site's platform team; it is not the installer identity of this page. Ask the platform team to confirm that it holds, in the kamiwaza namespace, the rights that Installer identity gives the installer. If it can hold only admin (and, on an Istio mesh, the Istio rights), set core.inferenceResources.allocationVerification: label, as in Installers that hold only admin. A sync that fails on a Role named kamiwaza-installer-grant-check, before any workload is applied, means that identity lacks a permission; the error lists each one. The lab's Argo CD ran with its default identity, so this refusal was not observed under Argo CD. The chart needs no cluster-scoped resource and nothing outside the namespace, so have the AppProject allow the kamiwaza namespace as its only destination and no cluster-scoped resources.

  3. Note the start time. Step 4 uses it:

    INSTALL_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)

  4. Create the Application. The Application, the repository registration and the AppProject are Argo CD objects. Whoever administers Argo CD at the site creates them, not the installer identity. Give them the YAML below and your values file.

    apiVersion: argoproj.io/v1alpha1
    kind: Application
    metadata:
      name: kamiwaza
      namespace: argocd
    spec:
      project: <your project>
      source:
        repoURL: ghcr.io/kamiwaza-ai/releases/kamiwaza/charts
        chart: kamiwaza
        targetRevision: <version>
        helm:
          releaseName: kamiwaza
          valuesObject:
            # the contents of kamiwaza-values.yaml from step 2
      destination:
        server: https://kubernetes.default.svc
        namespace: kamiwaza
      # Optional. Leave syncPolicy out to sync by hand.
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
    • Register the chart repository in Argo CD as an OCI Helm repository (enableOCI: "true"). The public registry needs no credentials. Your own registry usually needs credentials and its CA in the repository registration; the Argo CD administrator sets both.
    • Without internet access, set repoURL to <your registry host>/kamiwaza-ai/releases/kamiwaza/charts: the chart location from Install Without Internet Access without oci:// and without the chart name at the end.
    • syncPolicy.automated is optional. Add nothing under syncPolicy.syncOptions and no ignoreDifferences: the chart needs neither.
    • When you sync by hand, enable Prune for every sync that changes the version (the Prune box in the sync dialog, or argocd app sync kamiwaza --prune). The release Jobs carry a hash in their names, and without Prune the old ones keep the Application OutOfSync.
    • Do not set the CreateNamespace=true sync option. The namespace belongs to the platform administrator.

  5. Sync. With automated sync, the first sync starts by itself. Expect one sync that ends Succeeded. In the lab it took about 4.5 minutes with the images already on the node; pulling images adds to that.

  6. Check.

    • Argo CD shows the Application Synced and Healthy at the targetRevision you set. If you cannot open Argo CD, ask whoever administers it for the sync result; the two checks below are the ones you can run yourself.

    • The kamiwaza-namespace-prepare Job finished:

      kubectl -n kamiwaza logs job/kamiwaza-namespace-prepare | tail -3

      The last line is kamiwaza-namespace-prepare: done: <N> generated Secret(s) present. The log holds names and fingerprints, never a value.

    • Do step 4, with the INSTALL_START you noted. Its managed-by=Helm selectors match under Argo CD too. helm list -n kamiwaza shows nothing: Argo CD keeps no Helm release record.

After the install, a sync changes no Secret. In the lab, a repeat sync, a hard refresh followed by a sync, and 10 minutes of automated sync with prune and selfHeal changed no Secret. No automated sync started, and the Application stayed Synced and Healthy. To check, run this before and after a sync; the versions stay the same:

kubectl -n kamiwaza get secret -l kamiwaza.ai/generated-secret=true \
  -o custom-columns=NAME:.metadata.name,VERSION:.metadata.resourceVersion

The generated Secrets are not part of the Application, so Argo CD never applies, compares or prunes them. The Job labels each one kamiwaza.ai/generated-secret=true and annotates it argocd.argoproj.io/sync-options: Prune=false,Delete=false, argocd.argoproj.io/compare-options: IgnoreExtraneous and helm.sh/resource-policy: keep. It puts those marks back, at the start of every sync, on any generated Secret that lacks them.

To upgrade to a later release, change targetRevision and sync. To move an existing Helm installation to Argo CD, or to upgrade from 1.3, see Upgrade through Argo CD.

Limits​

  • Still generated when the chart renders. The edge certificate of network.tls.mode: generated is new in every render: under Argo CD use network.tls.mode: byo, the default. The Secrets of the DataHub and observability charts, DataHub's metrics exporter Secret among them, are also generated at render time, so every sync would replace them. Leave DataHub and the observability stack off under Argo CD in 1.4.1. Their charts have existingSecret values, but an install that supplies every one of their Secrets that way was not tested. The metrics exporter Secrets of Core, SpiceDB and Keycloak are not generated at render time: kamiwaza-namespace-prepare creates them.

  • A password or key pair given in values after its Secret exists. Argo CD applies the Secret rendered from the values over the existing one. Give such values (core.userDefaults.adminPassword, both seaweedfs.credentials keys) from the first install, or not at all. Setting such a value stores it in the Application, where whoever can read the Application can read it.

  • No pre-upgrade database snapshot. The snapshot the hardened profile takes before an upgrade (core.postgresql.upgradeBackup) is a Helm upgrade step and does not run under Argo CD. Take the pg_dumpall backup from Before you start before every sync that changes the version.

  • Not tested: syncPolicy.retry. Leave it out. The one deliberate stop in If Argo CD already synced 1.4.0 expects a sync that you start.

  • Label tracking. Only annotation tracking was tested. With label tracking (application.resourceTrackingMethod: label), the first sync of an installation that Helm made stops once in kamiwaza-namespace-prepare with exit code 3, as after a sync of 1.4.0. The stop and its log line are the ones shown at the start of step 1 of If Argo CD already synced 1.4.0. Sync again; the rest of that procedure does not apply.

  • No rollback to 1.4.0. Never set targetRevision back to 1.4.0 or earlier.

  • One chart. Install the kamiwaza chart. Its core, auth and seaweedfs subcharts are not supported as releases or Applications of their own in 1.4.1, and neither is the kamiwaza chart with core.enabled: false: kamiwaza-namespace-prepare belongs to the kamiwaza chart and runs from the Core image, and without it nothing creates their keys.

  • Selective sync. Do not use it for an install or an upgrade. It skips the hooks, kamiwaza-namespace-prepare among them.

  • The core-api autoscaler and metrics. Argo CD treats an autoscaler without CPU metrics as unhealthy. The chart has Argo CD create the autoscaler last, after the post-sync Jobs, and replace it on every sync, so a fresh install and an upgrade each finish in one sync on a cluster that serves the resource metrics API (metrics-server). Two consequences. Argo CD shows horizontalpodautoscaler/core-api as a hook, with no sync status, and does not prune it; that is expected. And on a cluster that does not serve the metrics API, every sync ends Failed on the autoscaler after everything else has run: the platform is installed, but Argo CD reports the operation as failed. Have the platform team install metrics-server, or ask the Argo CD administrator for a health customization for HorizontalPodAutoscaler; that setting applies to every Application of that Argo CD. The chart has no value that removes the autoscaler. This case and the customization were not run in the lab, whose cluster served the metrics API.

  • A failed database metrics Job. With global.observability.metrics.enabled: true, each PostgreSQL server has a Job named <server>-exporter-bootstrap-r1-<hash>. If one fails, it keeps its name and a later sync does not start it again, so the Application stays Degraded. Delete the failed Job and sync:

    kubectl -n kamiwaza get jobs | grep postgres-exporter-bootstrap
    kubectl -n kamiwaza delete job <name>

    This was not run in the lab.

  • No rollback of a failed sync. Argo CD does not undo a sync that fails. Fix the cause and sync again.

  • Changes made with kubectl. With selfHeal, Argo CD reverts a change made with kubectl to an object of the chart. Put the change in your values.

  • Helm commands. Do not run helm upgrade, helm rollback or helm uninstall on an installation that Argo CD manages.

  • Not tested with Istio. A fresh install on an Istio mesh through Argo CD was not run. An upgrade from 1.3 on an Istio mesh is not supported (see above).

  • Not tested on OpenShift with GreyMatter. See that install page.

  • Not tested: server-side apply. Annotation tracking with client-side apply is what was tested.

Upgrading with Helm 4: changes made with kubectl​

Helm 4 installs with server-side apply, and its upgrades keep the method of the release's last revision. Under server-side apply, Helm cannot change a field that someone else last changed, for example an image set with kubectl set image or a field changed with kubectl edit or kubectl patch. The upgrade then stops part-way and records a failed revision:

Error: UPGRADE FAILED: conflict occurred while applying object kamiwaza/core-scheduler apps/v1, Kind=Deployment:
Apply failed with 1 conflict: conflict with "kubectl-set" using apps/v1: .spec.template.spec.containers[name="core"].image

Check before upgrading, with the installer kubeconfig. First see how the release was last applied:

helm get metadata kamiwaza -n kamiwaza | grep APPLY_METHOD

If it says client-side apply, or prints nothing (Helm 3), the upgrade uses client-side apply and cannot hit this; skip the next command. If it says server-side apply:

helm get manifest kamiwaza -n kamiwaza \
  | kubectl -n kamiwaza apply --server-side --dry-run=server \
      --field-manager=helm --validate=false -f - 2>&1 >/dev/null \
  | grep -v '^Warning:'
# Expect no output. Nothing is changed in the cluster.

If it prints conflict with "kubectl-...", or an upgrade has already failed with that error, choose one:

  • Add --server-side=false to the upgrade command. Helm uses client-side apply and sets the chart's values over the changed fields. Later upgrades keep client-side apply, so you do not need the flag again. After a failed upgrade, rerun the same command with the flag; no rollback is needed.
  • Or add --force-conflicts. Helm takes over those fields and keeps server-side apply. Do not combine it with --force or --force-replace, and do not use it when the conflicting manager is not kubectl...: a controller writes its field back, and the next upgrade fails again. Use --server-side=false instead.

Either way the change made with kubectl is replaced by the chart's value. If you need to keep it, put it in your values file first.

Step 4: Verify​

Check what the platform does, not only its status fields.

After an install through Argo CD, first check in Argo CD that the Application is Synced and Healthy at the targetRevision you set, and use the INSTALL_START you noted before the sync. See Install with Argo CD.

Wait for the platform to start. Helm can return before every pod is ready and every setup job has finished:

kubectl -n kamiwaza wait deployment -l app.kubernetes.io/managed-by=Helm \
  --for=condition=Available --timeout=30m
kubectl -n kamiwaza wait job -l app.kubernetes.io/managed-by=Helm \
  --for=condition=Complete --timeout=70m

Pods are running and ready:

kubectl -n kamiwaza get pods --no-headers \
  | awk '$3 != "Completed" { split($2, r, "/"); if ($3 != "Running" || r[1] != r[2]) print }'
# Expect no output

An Error pod whose Job later completed is a retried attempt, not a failure. Confirm with kubectl -n kamiwaza get jobs: every Job should show all completions.

No image pulls failed during this install. Older events in the namespace are ignored. Run this in the same shell as step 3, or set INSTALL_START to the time you started the install, in the same format:

kubectl -n kamiwaza get events -o json | jq --arg start "$INSTALL_START" '
  [.items[]
   | select((.lastTimestamp // .eventTime // .metadata.creationTimestamp) >= $start)
   | select((.message // "") | test("ErrImagePull|ImagePullBackOff|Failed to pull image|Back-off pulling image"))
  ] | length'
# Expect 0

The platform answers at its domain:

curl -sk -o /dev/null -w '%{http_code}\n' https://kamiwaza.example.com/       # expect 200
curl -sk -o /dev/null -w '%{http_code}\n' https://kamiwaza.example.com/api/   # expect 401

Log in. If you did not set an admin password, read the generated one:

kubectl -n kamiwaza get secret kamiwaza-user-admin \
  -o jsonpath='{.data.password}' | base64 -d; echo

Open https://kamiwaza.example.com/ and log in as admin. With a certificate your browser does not trust, accept the warning first.

admin is the Kamiwaza application administrator. It is not the installer identity and not the Keycloak management account.

The platform is installed. Deploying models and apps is covered in the Quickstart.

Troubleshooting​

The install stops before creating anything. The chart checks its values before rendering. A missing EULA acceptance or license prints INSTALLATION HALTED; a missing egressCidrs value names the value it needs. Nothing was installed. Fix the values file and rerun step 3.

no Kamiwaza license is configured, with the license set. The license value belongs under core:. Check that it is core.license.existingSecret, not license.existingSecret. To see values Helm received but no chart reads, run helm get values kamiwaza -n kamiwaza.

The install hangs for about 15 minutes, then fails on a post-install job. This is almost always a cascade from core-scheduler not starting. Check its logs first: kubectl -n kamiwaza logs deploy/core-scheduler. A license problem shows here as License check failed; see License File. While the job runs, every helm upgrade fails with another operation (install/upgrade/rollback) is in progress.

Setup jobs stay at 1/2 ready and the install never finishes. The mesh uses classic sidecar injection. Set global.istio.classicInjection: true and rerun step 3.

Pods are Pending on their volumes. A StorageClass named in the values does not exist or cannot provision. Compare kubectl get storageclass with global.storage.stateful.*.

Images fail to pull with unauthorized. The pull Secret named in global.imagePullSecrets does not exist in the namespace, or has the wrong credentials. Compare it with kubectl -n kamiwaza get secret. If the pull fails with 403 Forbidden ... does not match expected scopes, the token lacks read:packages; see Registry access. Recreate the Secret with a token that has it. The pods retry the pull with back-off, so an install that has not yet timed out continues without a rerun; if the install already timed out, run the same install command again after recreating the Secret.

The platform returns 404 at its domain. The Gateway's hosts do not include global.domain, or global.ingress.gateway.* does not name your Gateway. See Istio and the ingress gateway.

Apps install but cannot reach the Kubernetes API. egressCidrs is missing the endpoint addresses. Add every address from Kubernetes API addresses and rerun step 3.

An app returns "Page Not Found", or the platform's own page. Wait about 40 seconds after its pods are ready. If the app stays deployed with no route, delete it and deploy it again.

A GPU model deployment fails with HTTP 422 and Core cannot see a GPU on this install. Core is running with tenant-mode inference off. The chart refuses to render that, so core's configuration was changed outside Helm. Upgrade the release with your values file; see Tenant-mode inference. If the core pods do not restart during the upgrade, restart them to load the restored configuration:

kubectl -n kamiwaza rollout restart deployment/core-scheduler \
  deployment/core-raycluster-head deployment/core-raycluster-worker

If your installation runs core-api, restart it separately:

kubectl -n kamiwaza rollout restart deployment/core-api

Next steps​