Uninstall from OpenShift with GreyMatter

This page removes a Kamiwaza installation from the kamiwaza namespace. It does not change the platform around it: the namespace, GreyMatter and Kamiwaza's configuration in your tenant configuration repository, storage classes, DNS, and certificates belong to the platform administrator and stay in place.

This is destructive. Uninstalling deletes Kamiwaza's databases, stored objects, and model files along with the platform. Back up anything you need before you begin.

Step 1: Back up

Export or copy any data you need to keep, such as documents in workrooms, configuration, and anything in the platform databases. Step 3 deletes the platform databases and object store, and they cannot be recovered after it.

Step 2: Remove apps and models

Remove what you deployed through Kamiwaza while the platform is still running:

1. Log in as admin.
2. In the App Garden, stop every deployed app. In the Tool Shed, stop every tool server.
3. On the Models page, stop every model deployment, including the embedding model Kamiwaza deploys on its own, and remove any model provider connections you added.
4. Confirm that all three lists are empty.

If Argo CD manages the installation

Do not run helm uninstall, and skip step 3. An installation that Argo CD made has no Helm release. One that was moved from Helm still has the old release record, and running helm uninstall on it deletes live resources that Argo CD then applies again. Instead:

1. Have whoever administers Argo CD delete the Application together with its resources.

2. Argo CD may leave these behind:

   • the generated Secrets, which are marked so that Argo CD never prunes or deletes them;
   • the core-api autoscaler, which a hook creates;
   • the finished Jobs kamiwaza-namespace-prepare, core-jwt-public-key and kamiwaza-keycloak-fips-migration. If the installation was moved from a Helm release that is not named kamiwaza, the last one is named <release>-keycloak-fips-migration;
   • the Helm release record, if the installation was moved from Helm;
   • the kamiwaza-namespace-state ConfigMap;
   • the kamiwaza-trust-bundle ConfigMap, which a hook creates;
   • the kamiwaza-keycloak-brokered-idp ConfigMap, if identity providers were configured. It is marked so that Argo CD never prunes or deletes it;
   • the kamiwaza-extension-trust-store ConfigMap and the kamiwaza-user-svc-core Secret. With Helm, a pre-delete hook removes them; Argo CD does not run it;
   • the kamiwaza-svc-credentials Secret, which a Job creates.

3. Continue with step 4. Then remove the leftovers:

   kubectl -n kamiwaza delete hpa core-api --ignore-not-found
   kubectl -n kamiwaza delete job kamiwaza-namespace-prepare core-jwt-public-key kamiwaza-keycloak-fips-migration --ignore-not-found
   kubectl -n kamiwaza delete configmap kamiwaza-namespace-state kamiwaza-trust-bundle --ignore-not-found
   kubectl -n kamiwaza delete configmap kamiwaza-keycloak-brokered-idp kamiwaza-extension-trust-store --ignore-not-found
   kubectl -n kamiwaza delete secret kamiwaza-user-svc-core kamiwaza-svc-credentials --ignore-not-found

   If the installation was moved from a Helm release with another name, also delete its migration Job and its identity provider ConfigMap:

   kubectl -n kamiwaza delete job <release>-keycloak-fips-migration --ignore-not-found
   kubectl -n kamiwaza delete configmap <release>-keycloak-brokered-idp --ignore-not-found

   If it was moved from Helm, delete the old release record. Do this only after Argo CD has deleted the Application's resources. The command removes Helm's record and nothing else:

   kubectl -n kamiwaza delete secret -l owner=helm,name=<release>

   Only when the data they protect is gone too, delete the generated Secrets:

   kubectl -n kamiwaza delete secret -l kamiwaza.ai/generated-secret=true --ignore-not-found

Removing an installation through Argo CD was not run in the lab for 1.4.1.

Step 3: Uninstall the release

Run as the installer identity. This deletes the volume claims for the platform databases and the object store; step 4 describes what happens to the disks behind them:

helm uninstall kamiwaza --namespace kamiwaza --wait --timeout 15m

Step 4: Delete the remaining volumes

Two kinds of volume outlive the release: the model registry cache, which the chart keeps on purpose, and the configuration store's volumes, which Kubernetes keeps for any StatefulSet. GreyMatter shares this namespace, so delete Kamiwaza's claims by name rather than every claim in it:

kubectl -n kamiwaza get persistentvolumeclaims
kubectl -n kamiwaza delete persistentvolumeclaims --ignore-not-found \
  kamiwaza-registry-data core-etcd-data-core-etcd-0 core-etcd-data-core-etcd-1 core-etcd-data-core-etcd-2

Whether the underlying disks are also deleted depends on the StorageClass's reclaimPolicy. With Retain, the platform administrator deletes the released PersistentVolumes.

Step 5: Verify

Run as the platform administrator, since the installer identity may not be able to list every kind:

kubectl api-resources --verbs=list --namespaced -o name \
  | grep -v -E '^events(\.events\.k8s\.io)?$' \
  | xargs -n 1 kubectl -n kamiwaza get --ignore-not-found -o name 2>/dev/null \
  | grep -v -E '^(packagemanifest|podmetrics)\.'

The last line drops kinds that OpenShift lists in every namespace: operator catalog entries and pod metrics.

Expect only the platform prerequisites, what OpenShift creates in every project, and what GreyMatter runs for the namespace:

• the installer's ServiceAccount, Roles, and RoleBindings;
• the kamiwaza-license Secret, and your pull Secret if you use one;
• from 1.4.1, the Secrets Kamiwaza generated (its keys, passwords and certificates), the finished Job kamiwaza-namespace-prepare (and kamiwaza-keycloak-fips-migration, if it ran) with its pod, and the kamiwaza-namespace-state ConfigMap. From 1.4.1 helm uninstall always leaves the generated Secrets (the keepGeneratedSecretsOnUninstall values no longer change that), so that a reinstall into the same namespace finds and reuses them. The one exception is kamiwaza-user-admin, which a pre-delete hook removes. To remove the others, run kubectl -n kamiwaza delete secret -l kamiwaza.ai/generated-secret=true. The kamiwaza-svc-credentials Secret and the kamiwaza-trust-bundle ConfigMap stay as well, and within an hour of the last install or upgrade so can the finished Job core-jwt-public-key. helm uninstall of 1.4.1 was not run in the lab; this list is from the chart;
• the greymatter-tenant-repo, greymatter-edge-ingress, and greymatter-image-pull Secrets;
• a kamiwaza-edge-trust ConfigMap that you created by hand for an earlier release, if it is still there. Delete it before you install again: the chart refuses to publish over it;
• the kube-root-ca.crt and openshift-service-ca.crt ConfigMaps, and the service accounts, Secrets, and RoleBindings OpenShift adds to every project;
• GreyMatter's tenant controllers, the edge and its Route, and their GreyMatter resources. GreyMatter keeps these while Kamiwaza's configuration is in the tenant configuration repository. They include the edgeresources and sidecarresources objects, the greymatter-sidecar-injection-certs Secret, the greymatter-sidecar-injection-service Service, the greymatter-cni-greymatter NetworkAttachmentDefinition, and the controllers' leases.

If anything else remains, contact Kamiwaza support with the list before reinstalling into the same namespace.

Removing the platform prerequisites

If Kamiwaza will not be installed again, the platform administrator can remove what was prepared for it, following their own procedures for GreyMatter, storage, DNS, and certificates. Work in this order:

1. Remove Kamiwaza's configuration from the tenant configuration repository (_tenants/kamiwaza), and let GreyMatter Sync apply the change. GreyMatter then removes the edge and its Route.

2. Remove the namespace from tenant_namespaces (and ftr_namespaces, if you added it) in your GreyMatter Core configuration.

3. Delete the kamiwaza namespace, which removes the installer identity, the Secrets, and everything else in it:

   oc delete project kamiwaza

4. Remove the DNS records for the Kamiwaza domain. If nothing else relies on the inotify limits, remove them the way you set them: delete the 99-worker-kamiwaza-inotify MachineConfig, which restarts the pool's nodes, or, where you couldn't use a MachineConfig, undo the setting the way you applied it.

To install again after deleting the namespace, start from Platform Prerequisites.