Uninstall from Kubernetes with Istio

This page removes a Kamiwaza installation from the kamiwaza namespace. It does not change the platform around it: the namespace, the service mesh and gateway, storage classes, DNS, and certificates belong to the platform administrator and stay in place.

This is destructive. Uninstalling deletes Kamiwaza's databases, stored objects, and model files along with the platform. Back up anything you need before you begin.

Step 1: Back up

Export or copy any data you need to keep, such as documents in workrooms, configuration, and anything in the platform databases. Step 3 deletes the platform databases and object store, and they cannot be recovered after it.

Step 2: Remove apps and models

Remove what you deployed through Kamiwaza while the platform is still running:

1. Log in as admin.
2. In the App Garden, stop every deployed app. In the Tool Shed, stop every tool server.
3. On the Models page, stop every model deployment.
4. Confirm that all three lists are empty.

If Argo CD manages the installation

Do not run helm uninstall, and skip step 3. An installation that Argo CD made has no Helm release. One that was moved from Helm still has the old release record, and running helm uninstall on it deletes live resources that Argo CD then applies again. Instead:

1. Have whoever administers Argo CD delete the Application together with its resources.

2. Argo CD may leave these behind:

   • the generated Secrets, which are marked so that Argo CD never prunes or deletes them;
   • the core-api autoscaler, which a hook creates;
   • the finished Jobs kamiwaza-namespace-prepare, core-jwt-public-key and kamiwaza-keycloak-fips-migration. If the installation was moved from a Helm release that is not named kamiwaza, the last one is named <release>-keycloak-fips-migration;
   • the Helm release record, if the installation was moved from Helm;
   • the kamiwaza-namespace-state ConfigMap;
   • the kamiwaza-trust-bundle ConfigMap, which a hook creates;
   • the kamiwaza-keycloak-brokered-idp ConfigMap, if identity providers were configured. It is marked so that Argo CD never prunes or deletes it;
   • the kamiwaza-extension-trust-store ConfigMap and the kamiwaza-user-svc-core Secret. With Helm, a pre-delete hook removes them; Argo CD does not run it;
   • the kamiwaza-svc-credentials Secret, which a Job creates.

3. Continue with step 4. Then remove the leftovers:

   kubectl -n kamiwaza delete hpa core-api --ignore-not-found
   kubectl -n kamiwaza delete job kamiwaza-namespace-prepare core-jwt-public-key kamiwaza-keycloak-fips-migration --ignore-not-found
   kubectl -n kamiwaza delete configmap kamiwaza-namespace-state kamiwaza-trust-bundle --ignore-not-found
   kubectl -n kamiwaza delete configmap kamiwaza-keycloak-brokered-idp kamiwaza-extension-trust-store --ignore-not-found
   kubectl -n kamiwaza delete secret kamiwaza-user-svc-core kamiwaza-svc-credentials --ignore-not-found

   If the installation was moved from a Helm release with another name, also delete its migration Job and its identity provider ConfigMap:

   kubectl -n kamiwaza delete job <release>-keycloak-fips-migration --ignore-not-found
   kubectl -n kamiwaza delete configmap <release>-keycloak-brokered-idp --ignore-not-found

   If it was moved from Helm, delete the old release record. Do this only after Argo CD has deleted the Application's resources. The command removes Helm's record and nothing else:

   kubectl -n kamiwaza delete secret -l owner=helm,name=<release>

   Only when the data they protect is gone too, delete the generated Secrets:

   kubectl -n kamiwaza delete secret -l kamiwaza.ai/generated-secret=true --ignore-not-found

Removing an installation through Argo CD was not run in the lab for 1.4.1.

Step 3: Uninstall the release

Run as the installer identity. This deletes the volume claims for the platform databases and the object store; step 4 describes what happens to the disks behind them:

helm uninstall kamiwaza --namespace kamiwaza --wait --timeout 15m

Step 4: Delete the remaining volumes

Two kinds of volume outlive the release: the model registry cache, which the chart keeps on purpose, and the configuration store's volumes, which Kubernetes keeps for any StatefulSet. Delete them:

kubectl -n kamiwaza get persistentvolumeclaims
kubectl -n kamiwaza delete persistentvolumeclaims --all

Whether the underlying disks are also deleted depends on the StorageClass's reclaimPolicy. With Retain, the platform administrator deletes the released PersistentVolumes.

Step 5: Verify

Run as the platform administrator, since the installer identity may not be able to list every kind:

kubectl api-resources --verbs=list --namespaced -o name \
  | grep -v -E '^events(\.events\.k8s\.io)?$' \
  | xargs -n 1 kubectl -n kamiwaza get --ignore-not-found -o name 2>/dev/null

Expect only the platform prerequisites and what Kubernetes and Istio create in every namespace:

• the installer's ServiceAccount, Roles, and RoleBindings;
• the kamiwaza-license Secret, and your pull Secret if you use one;
• from 1.4.1, the Secrets Kamiwaza generated (its keys, passwords and certificates), the finished Job kamiwaza-namespace-prepare (and kamiwaza-keycloak-fips-migration, if it ran) with its pod, and the kamiwaza-namespace-state ConfigMap. From 1.4.1 helm uninstall always leaves the generated Secrets (the keepGeneratedSecretsOnUninstall values no longer change that), so that a reinstall into the same namespace finds and reuses them. The one exception is kamiwaza-user-admin, which a pre-delete hook removes. To remove the others, run kubectl -n kamiwaza delete secret -l kamiwaza.ai/generated-secret=true. The kamiwaza-svc-credentials Secret and the kamiwaza-trust-bundle ConfigMap stay as well, and within an hour of the last install or upgrade so can the finished Job core-jwt-public-key. helm uninstall of 1.4.1 was not run in the lab; this list is from the chart;
• a kamiwaza-edge-trust ConfigMap that you created by hand for an earlier release, if it is still there. Delete it before you install again: the chart refuses to publish over it;
• the kube-root-ca.crt, istio-ca-root-cert, and (on Istio versions that publish one) istio-ca-crl ConfigMaps, and the default ServiceAccount.

If anything else remains, contact Kamiwaza support with the list before reinstalling into the same namespace.

Removing the platform prerequisites

If Kamiwaza will not be installed again, the platform administrator can remove what was prepared for it, following their own procedures for the mesh, storage, DNS, and certificates:

• the kamiwaza namespace, which removes the installer identity, license Secret, and everything else in it:

  kubectl delete namespace kamiwaza

• the Istio Gateway and edge certificate Secret, if they serve only Kamiwaza;

• the DNS records for the Kamiwaza domain, including any in-cluster DNS entry;

• the inotify settings in /etc/sysctl.d/99-kamiwaza-inotify.conf on each node, if nothing else relies on them.

To install again after deleting the namespace, start from Platform Prerequisites.