Skip to main content

Platform Prerequisites for Kubernetes with Istio

Kamiwaza installs into a single namespace on a Kubernetes cluster that you already run. Before the install, a platform administrator (someone with cluster-admin rights) prepares the cluster so it provides everything on this page. The Kamiwaza chart does not create or repair any of it: it creates no cluster-scoped objects, installs no operators, and does not configure your service mesh. If a prerequisite is missing, the install fails or the platform comes up unable to serve traffic.

Work through each section and run its check. Each section also says what to record; the install needs those facts.

Hardware sizing and supported operating systems are covered separately, in System Requirements.

Summary​

PrerequisiteWhat the cluster must provide
NamespaceA pre-created namespace, labeled for Istio sidecar injection
Installer identityA namespace-scoped identity to run Helm as. Never cluster-admin
Pod security guardAn admission policy that enforces Pod Security baseline in the kamiwaza and kamiwaza-extensions namespaces. Needs Kubernetes 1.30 or later
Istio and the ingress gatewayAn Istio mesh in sidecar mode, with an ingress gateway serving the Kamiwaza domain on HTTPS
Edge certificateA TLS certificate for the domain on that gateway
DNSThe domain resolves to the gateway, from clients and from inside the cluster
Block storageAn RWO StorageClass whose volumes keep their data when pods are replaced
Object storageA decision: the built-in object store, or your own S3-compatible service
Kubernetes API addressesThe API server's Service and endpoint addresses, recorded for network policy
Registry accessNodes can pull Kamiwaza images
Node kernel limitsfs.inotify limits at or above the floor on every node
Outbound network accessHTTPS to the image registry and to any model sources you use

GPU inference and multi-node clusters add requirements; see GPU nodes and Multi-node clusters.

Kamiwaza installs into a namespace named kamiwaza; use that name. The examples use kamiwaza.example.com as the domain; substitute your own. Run the checks with the administrator's kubeconfig unless a check says otherwise.

Namespace​

Create the kamiwaza namespace. The installer does not create it, and the install never uses --create-namespace.

Label it for Istio sidecar injection, and to warn and audit against the Pod Security restricted profile. Do not enforce restricted: some Kamiwaza components, and Istio's init container, do not meet it yet.

Do not add an enforce label for baseline either. Istio's istio-init container adds the NET_ADMIN and NET_RAW capabilities, which baseline forbids, and Pod Security admission can exempt a whole pod but never one container. So enforce=baseline rejects every pod Istio injects. The pod security guard enforces baseline instead.

kubectl create namespace kamiwaza
kubectl label namespace kamiwaza \
  istio-injection=enabled \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

If your mesh uses revision-based injection (istio.io/rev=<revision>), use that label instead of istio-injection=enabled.

Check:

kubectl get namespace kamiwaza --show-labels
# Expect istio-injection=enabled (or istio.io/rev=...), and the pod-security labels

Installer identity​

Helm runs as a namespace-scoped identity, never as cluster-admin. It needs:

  • the built-in admin ClusterRole, bound in the namespace only with a RoleBinding;
  • create and manage rights on the Istio resources Kamiwaza writes into its namespace (VirtualService, DestinationRule, EnvoyFilter, AuthorizationPolicy, PeerAuthentication);
  • rights to manage endpoints and patch pods/status. The chart grants these to its own service accounts, and Kubernetes only lets an identity grant permissions it holds itself.

This manifest creates a service account with those rights. You can bind the same Roles to a user or group from your identity provider instead.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kamiwaza-installer
  namespace: kamiwaza
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kamiwaza-installer-admin
  namespace: kamiwaza
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
  - kind: ServiceAccount
    name: kamiwaza-installer
    namespace: kamiwaza
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kamiwaza-installer-istio
  namespace: kamiwaza
rules:
  - apiGroups: ["networking.istio.io"]
    resources: ["destinationrules", "envoyfilters", "virtualservices"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["security.istio.io"]
    resources: ["authorizationpolicies", "peerauthentications"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kamiwaza-installer-istio
  namespace: kamiwaza
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kamiwaza-installer-istio
subjects:
  - kind: ServiceAccount
    name: kamiwaza-installer
    namespace: kamiwaza
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kamiwaza-installer-delegation
  namespace: kamiwaza
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["create", "delete", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/status"]
    verbs: ["patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kamiwaza-installer-delegation
  namespace: kamiwaza
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kamiwaza-installer-delegation
subjects:
  - kind: ServiceAccount
    name: kamiwaza-installer
    namespace: kamiwaza

Save it as kamiwaza-installer.yaml and apply it as the administrator:

kubectl apply -f kamiwaza-installer.yaml

Give the installer a kubeconfig for this identity, following your cluster's normal practice for issuing credentials. For example, with a service account token, run as the administrator:

(
  set -e
  SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
  kubectl config view --minify --raw \
    -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > cluster-ca.crt
  TOKEN=$(kubectl -n kamiwaza create token kamiwaza-installer --duration=24h)

  export KUBECONFIG=kamiwaza-installer.kubeconfig
  kubectl config set-cluster kamiwaza --server="$SERVER" \
    --certificate-authority=cluster-ca.crt --embed-certs
  kubectl config set-credentials kamiwaza-installer --token="$TOKEN"
  kubectl config set-context kamiwaza --cluster=kamiwaza --user=kamiwaza-installer --namespace=kamiwaza
  kubectl config use-context kamiwaza
)

The parentheses keep these commands from changing your own shell's KUBECONFIG. If your kubeconfig names a certificate-authority file instead of embedding certificate-authority-data, copy that file to cluster-ca.crt instead. The token expires, and the API server may cap the duration you ask for. Issue a new one before each later upgrade or uninstall.

Check that the identity can create namespaced resources and cannot create cluster-scoped ones:

SA=system:serviceaccount:kamiwaza:kamiwaza-installer
kubectl auth can-i create deployments -n kamiwaza --as=$SA            # yes
kubectl auth can-i create clusterroles --as=$SA                       # no
kubectl auth can-i create customresourcedefinitions --as=$SA          # no
kubectl auth can-i list nodes --as=$SA                                # no

The installer's Istio rights are checked in the next section, once Istio's resource types exist.

Pod security guard​

The installer identity holds admin in the namespace, so it can create pods, and so can the service accounts Kamiwaza installs. Without an admission policy, anyone holding one of those identities can start a privileged pod, or one that mounts the node's file system, and take over the node. The namespace labels above only warn.

The guard closes that. It is a ValidatingAdmissionPolicy and binding that enforce the Kubernetes Pod Security baseline controls on every pod in the kamiwaza and kamiwaza-extensions namespaces, including init and ephemeral containers. That includes the control Kubernetes 1.34 added to baseline: a probe or lifecycle hook may not name a host.

The guard makes one exception: an istio-init container may add NET_ADMIN and NET_RAW, but only in the shape Istio injects. The exception requires Istio's own proxy image and entrypoint, and the istio-iptables command. The only volume it may mount is the service account token volume that Kubernetes adds to the pod (kube-api-access-*), read-only at the token path. The guard checks that volume's contents, not only where it is mounted: it must project just the token, the cluster CA certificate and the pod's namespace. The container's environment variables must have literal values: Istio copies the mesh's proxyMetadata settings, such as DNS capture, into istio-init that way. A variable set with valueFrom or envFrom is refused, and so are names that change which code runs: PATH, GCONV_PATH, the dynamic loader's LD_*, and the iptables module paths XTABLES_LIBDIR, IPTABLES_LIB_DIR and IP6TABLES_LIB_DIR. A container merely named istio-init gets nothing.

A second policy, kamiwaza-tenant-exec-guard, refuses kubectl exec and kubectl attach into any container named istio-init in the same namespaces. The installer's admin role includes exec, and istio-iptables has flags that keep it running, so without this policy the installer could start a pod whose istio-init stays up and open a root shell in it with NET_ADMIN and NET_RAW. Exec and attach into every other container work as before.

Both policies bind cluster administrators too, and a namespace admin cannot read, change or remove them.

What remains is inherent to classic Istio injection: istio-init must run with NET_ADMIN and NET_RAW to write the pod's iptables rules. The guard keeps tenants from forging that container and from exec'ing or attaching into it. It pins the image, the entrypoint and the istio-iptables command, not every flag, so a tenant can still choose how istio-iptables configures its own pod's network, but cannot run code of its own with those capabilities. Istio's ambient mode has no istio-init container at all.

The guard needs the admissionregistration.k8s.io/v1 ValidatingAdmissionPolicy API, which Kubernetes 1.30 and later serve. For older clusters, see Kubernetes 1.28 and 1.29 below.

Save this as kamiwaza-tenant-pod-guard.yaml:

# Kamiwaza tenant pod guard.
#
# Enforces the Kubernetes Pod Security "baseline" controls, including the
# probe and lifecycle-hook host control added in Kubernetes 1.34, on every pod
# in the Kamiwaza namespaces. There is one exception: Istio's classic
# `istio-init` init container may add NET_ADMIN and NET_RAW, and only in the
# exact shape istiod injects. A second policy refuses exec and attach into
# istio-init. Requires Kubernetes 1.30 or later. Apply as a cluster
# administrator.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: kamiwaza-tenant-pod-guard
  labels:
    app.kubernetes.io/name: kamiwaza-tenant-pod-guard
    app.kubernetes.io/part-of: kamiwaza
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods", "pods/ephemeralcontainers"]
  matchConditions:
    # Pod specs are immutable apart from images, tolerations, deadlines and
    # ephemeral containers. Skipping metadata-only updates keeps pods admitted
    # before the guard existed patchable (labels, ownerReferences) until they
    # are replaced. Every create and every spec change is checked.
    - name: create-or-spec-change
      expression: "request.operation == 'CREATE' || object.spec != oldObject.spec"
  variables:
    - name: podSC
      expression: "has(object.spec.securityContext) ? object.spec.securityContext : {}"
    - name: initContainers
      expression: "has(object.spec.initContainers) ? object.spec.initContainers : []"
    - name: ephemeralContainers
      expression: "has(object.spec.ephemeralContainers) ? object.spec.ephemeralContainers : []"
    - name: allContainers
      expression: "object.spec.containers + variables.initContainers + variables.ephemeralContainers"
    - name: baselineCapabilities
      expression: >-
        ['AUDIT_WRITE', 'CHOWN', 'DAC_OVERRIDE', 'FOWNER', 'FSETID', 'KILL', 'MKNOD',
        'NET_BIND_SERVICE', 'SETFCAP', 'SETGID', 'SETPCAP', 'SETUID', 'SYS_CHROOT']
    # Where the mesh pulls its proxy image from. Each entry ends in ':' or '@'
    # so a look-alike repository name cannot match. Edit for a mirrored registry.
    - name: istioInitImagePrefixes
      expression: >-
        ['registry.istio.io/release/proxyv2:', 'registry.istio.io/release/proxyv2@',
        'docker.io/istio/proxyv2:', 'docker.io/istio/proxyv2@',
        'ghcr.io/kamiwaza-internal/containers/images/istio/proxyv2:',
        'ghcr.io/kamiwaza-internal/containers/images/istio/proxyv2@']
    # Environment names istio-init may not set, because they steer what code it
    # loads: the dynamic loader (LD_*, GCONV_PATH), the command search path, and
    # the directory iptables loads its extension modules from. Exact names, so
    # istio-iptables' own settings (IPTABLES_PROBE_PORT, for example) still pass.
    - name: istioInitLoaderEnvPrefixes
      expression: "['LD_']"
    - name: istioInitLoaderEnvNames
      expression: >-
        ['PATH', 'GCONV_PATH', 'XTABLES_LIBDIR', 'IPTABLES_LIB_DIR',
        'IP6TABLES_LIB_DIR']
    # Volumes in exactly the shape the ServiceAccount admission plugin injects
    # (kube-api-access-*): a projected token, the cluster CA from the
    # kube-root-ca.crt ConfigMap, and the pod's namespace. They hold no file a
    # tenant chooses, so they are the only volumes istio-init may mount.
    - name: serviceAccountTokenVolumes
      expression: >-
        (has(object.spec.volumes) ? object.spec.volumes : []).filter(v,
        has(v.projected) && has(v.projected.sources) &&
        size(v.projected.sources) == 3 &&
        v.projected.sources.exists(s, has(s.serviceAccountToken) &&
        s.serviceAccountToken.path == 'token' && !has(s.serviceAccountToken.audience)) &&
        v.projected.sources.exists(s, has(s.configMap) &&
        s.configMap.name == 'kube-root-ca.crt' && has(s.configMap.items) &&
        size(s.configMap.items) == 1 && s.configMap.items[0].key == 'ca.crt' &&
        s.configMap.items[0].path == 'ca.crt') &&
        v.projected.sources.exists(s, has(s.downwardAPI) && has(s.downwardAPI.items) &&
        size(s.downwardAPI.items) == 1 && s.downwardAPI.items[0].path == 'namespace' &&
        has(s.downwardAPI.items[0].fieldRef) &&
        s.downwardAPI.items[0].fieldRef.fieldPath == 'metadata.namespace')
        ).map(v, v.name)
    - name: safeSysctls
      expression: >-
        ['kernel.shm_rmid_forced', 'net.ipv4.ip_local_port_range',
        'net.ipv4.ip_unprivileged_port_start', 'net.ipv4.tcp_syncookies',
        'net.ipv4.ping_group_range', 'net.ipv4.ip_local_reserved_ports',
        'net.ipv4.tcp_keepalive_time', 'net.ipv4.tcp_fin_timeout',
        'net.ipv4.tcp_keepalive_intvl', 'net.ipv4.tcp_keepalive_probes']
    - name: selinuxTypes
      expression: "['', 'container_t', 'container_init_t', 'container_kvm_t', 'container_engine_t']"
    # Containers whose probe or lifecycle hook names a host. The kubelet sends
    # these requests from the node, so a host could aim them at a node-local or
    # cloud metadata endpoint. Baseline allows the field unset or empty.
    - name: handlerHostContainers
      expression: >-
        variables.allContainers.filter(c,
        ((has(c.livenessProbe) ? [c.livenessProbe] : []) +
        (has(c.readinessProbe) ? [c.readinessProbe] : []) +
        (has(c.startupProbe) ? [c.startupProbe] : [])).exists(h,
        (has(h.httpGet) && has(h.httpGet.host) && h.httpGet.host != '') ||
        (has(h.tcpSocket) && has(h.tcpSocket.host) && h.tcpSocket.host != '')) ||
        (has(c.lifecycle) &&
        ((has(c.lifecycle.postStart) ? [c.lifecycle.postStart] : []) +
        (has(c.lifecycle.preStop) ? [c.lifecycle.preStop] : [])).exists(h,
        (has(h.httpGet) && has(h.httpGet.host) && h.httpGet.host != '') ||
        (has(h.tcpSocket) && has(h.tcpSocket.host) && h.tcpSocket.host != '')))
        ).map(c, c.name)
  validations:
    - expression: >-
        !(has(object.spec.hostNetwork) && object.spec.hostNetwork) &&
        !(has(object.spec.hostPID) && object.spec.hostPID) &&
        !(has(object.spec.hostIPC) && object.spec.hostIPC)
      message: "hostNetwork, hostPID and hostIPC are not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        variables.allContainers.all(c, !has(c.securityContext) ||
        !has(c.securityContext.privileged) || !c.securityContext.privileged)
      messageExpression: >-
        'privileged containers are not allowed in the Kamiwaza tenant namespace: ' +
        variables.allContainers.filter(c, has(c.securityContext) &&
        has(c.securityContext.privileged) && c.securityContext.privileged).map(c, c.name).join(', ')
      message: "privileged containers are not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: "!has(object.spec.volumes) || object.spec.volumes.all(v, !has(v.hostPath))"
      messageExpression: >-
        'hostPath volumes are not allowed in the Kamiwaza tenant namespace: ' +
        object.spec.volumes.filter(v, has(v.hostPath)).map(v, v.name).join(', ')
      message: "hostPath volumes are not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    # Regular and ephemeral containers: the baseline capability set only.
    - expression: >-
        (object.spec.containers + variables.ephemeralContainers).all(c,
        !has(c.securityContext) || !has(c.securityContext.capabilities) ||
        !has(c.securityContext.capabilities.add) ||
        c.securityContext.capabilities.add.all(cap, cap in variables.baselineCapabilities))
      message: "containers may add only the Pod Security baseline capabilities in the Kamiwaza tenant namespace."
      reason: Forbidden
    # Init containers: the baseline set, or NET_ADMIN/NET_RAW for an injected
    # istio-init. The exception pins the mesh's own image, its entrypoint (no
    # command), the istio-iptables subcommand, and no restart policy (a plain
    # init container, not a long-running sidecar). Its only mounts may be the
    # service-account token volume (serviceAccountTokenVolumes), read-only at
    # the token path. Environment entries must be literal values: istiod copies
    # the mesh's proxyMetadata (DNS capture, for example) into istio-init as
    # name/value pairs. valueFrom, envFrom and the loader names above are
    # refused.
    - expression: >-
        variables.initContainers.all(c,
        !has(c.securityContext) || !has(c.securityContext.capabilities) ||
        !has(c.securityContext.capabilities.add) ||
        c.securityContext.capabilities.add.all(cap, cap in variables.baselineCapabilities) ||
        (c.name == 'istio-init' &&
        c.securityContext.capabilities.add.all(cap,
        cap in variables.baselineCapabilities || cap in ['NET_ADMIN', 'NET_RAW']) &&
        variables.istioInitImagePrefixes.exists(p, c.image.startsWith(p)) &&
        !has(c.command) &&
        has(c.args) && size(c.args) > 0 && c.args[0] == 'istio-iptables' &&
        (!has(c.env) || c.env.all(e, !has(e.valueFrom) &&
        !variables.istioInitLoaderEnvPrefixes.exists(p, e.name.startsWith(p)) &&
        !(e.name in variables.istioInitLoaderEnvNames))) && !has(c.envFrom) &&
        !has(c.volumeDevices) && !has(c.restartPolicy) &&
        (!has(c.volumeMounts) || c.volumeMounts.all(m,
        m.mountPath == '/var/run/secrets/kubernetes.io/serviceaccount' &&
        has(m.readOnly) && m.readOnly && !has(m.subPath) && !has(m.subPathExpr) &&
        m.name in variables.serviceAccountTokenVolumes)) &&
        has(c.securityContext.allowPrivilegeEscalation) &&
        !c.securityContext.allowPrivilegeEscalation &&
        has(c.securityContext.capabilities.drop) &&
        'ALL' in c.securityContext.capabilities.drop))
      message: >-
        init containers may add only the Pod Security baseline capabilities in the
        Kamiwaza tenant namespace; NET_ADMIN and NET_RAW are allowed only for the
        Istio-injected istio-init container.
      reason: Forbidden
    - expression: >-
        variables.allContainers.all(c, !has(c.ports) ||
        c.ports.all(p, !has(p.hostPort) || p.hostPort == 0))
      message: "hostPort is not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: "size(variables.handlerHostContainers) == 0"
      messageExpression: >-
        'probes and lifecycle hooks may not set a host in the Kamiwaza tenant namespace: ' +
        variables.handlerHostContainers.join(', ')
      message: "probes and lifecycle hooks may not set a host in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        !(has(variables.podSC.seccompProfile) && variables.podSC.seccompProfile.type == 'Unconfined') &&
        variables.allContainers.all(c, !has(c.securityContext) ||
        !has(c.securityContext.seccompProfile) ||
        c.securityContext.seccompProfile.type != 'Unconfined')
      message: "seccompProfile Unconfined is not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        !(has(variables.podSC.appArmorProfile) && variables.podSC.appArmorProfile.type == 'Unconfined') &&
        variables.allContainers.all(c, !has(c.securityContext) ||
        !has(c.securityContext.appArmorProfile) ||
        c.securityContext.appArmorProfile.type != 'Unconfined') &&
        (!has(object.metadata.annotations) || object.metadata.annotations.all(k,
        !k.startsWith('container.apparmor.security.beta.kubernetes.io/') ||
        object.metadata.annotations[k] == 'runtime/default' ||
        object.metadata.annotations[k].startsWith('localhost/')))
      message: "AppArmor Unconfined is not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        (!has(variables.podSC.seLinuxOptions) ||
        ((!has(variables.podSC.seLinuxOptions.type) || variables.podSC.seLinuxOptions.type in variables.selinuxTypes) &&
        (!has(variables.podSC.seLinuxOptions.user) || variables.podSC.seLinuxOptions.user == '') &&
        (!has(variables.podSC.seLinuxOptions.role) || variables.podSC.seLinuxOptions.role == ''))) &&
        variables.allContainers.all(c, !has(c.securityContext) || !has(c.securityContext.seLinuxOptions) ||
        ((!has(c.securityContext.seLinuxOptions.type) || c.securityContext.seLinuxOptions.type in variables.selinuxTypes) &&
        (!has(c.securityContext.seLinuxOptions.user) || c.securityContext.seLinuxOptions.user == '') &&
        (!has(c.securityContext.seLinuxOptions.role) || c.securityContext.seLinuxOptions.role == '')))
      message: "custom SELinux user, role or type is not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        variables.allContainers.all(c, !has(c.securityContext) ||
        !has(c.securityContext.procMount) || c.securityContext.procMount == 'Default')
      message: "procMount Unmasked is not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        !has(variables.podSC.sysctls) ||
        variables.podSC.sysctls.all(s, s.name in variables.safeSysctls)
      message: "only the Pod Security baseline safe sysctls are allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
    - expression: >-
        !(has(variables.podSC.windowsOptions) && has(variables.podSC.windowsOptions.hostProcess) &&
        variables.podSC.windowsOptions.hostProcess) &&
        variables.allContainers.all(c, !has(c.securityContext) ||
        !has(c.securityContext.windowsOptions) ||
        !has(c.securityContext.windowsOptions.hostProcess) ||
        !c.securityContext.windowsOptions.hostProcess)
      message: "Windows HostProcess pods are not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: kamiwaza-tenant-pod-guard
  labels:
    app.kubernetes.io/name: kamiwaza-tenant-pod-guard
    app.kubernetes.io/part-of: kamiwaza
spec:
  policyName: kamiwaza-tenant-pod-guard
  # [Warn, Audit] reports what would be denied without denying it. Use it for a
  # staged rollout on an existing install, then switch to [Deny, Audit].
  validationActions: [Deny, Audit]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [kamiwaza, kamiwaza-extensions]
---
# The exception above gives NET_ADMIN and NET_RAW to istio-init, and a
# namespace admin can create a pod whose istio-init keeps running. Exec or
# attach into it would give the tenant a root shell with those capabilities in
# the pod's network namespace, so this policy refuses both.
#
# Exec and attach reach admission as CONNECT on pods/exec and pods/attach, and
# `object` is the request's PodExecOptions or PodAttachOptions. Kubernetes
# v1.30.0 source: the ValidatingAdmissionPolicy plugin handles Connect
# (staging/src/k8s.io/apiserver/pkg/admission/plugin/policy/validating/plugin.go:79);
# CONNECT is a valid policy operation
# (pkg/apis/admissionregistration/validation/validation.go:473-479); the
# connect handler admits the parsed options as the object
# (staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:211).
# `container` is absent when the client names none. The API server then picks
# the pod's only regular container, never an init container
# (pkg/registry/core/pod/strategy.go:668-691), so that request is allowed.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: kamiwaza-tenant-exec-guard
  labels:
    app.kubernetes.io/name: kamiwaza-tenant-exec-guard
    app.kubernetes.io/part-of: kamiwaza
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CONNECT"]
        resources: ["pods/exec", "pods/attach"]
  validations:
    - expression: "!has(object.container) || object.container != 'istio-init'"
      message: "exec and attach into istio-init are not allowed in the Kamiwaza tenant namespace."
      reason: Forbidden
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: kamiwaza-tenant-exec-guard
  labels:
    app.kubernetes.io/name: kamiwaza-tenant-exec-guard
    app.kubernetes.io/part-of: kamiwaza
spec:
  policyName: kamiwaza-tenant-exec-guard
  # Binds the same namespaces as kamiwaza-tenant-pod-guard, in the same mode.
  validationActions: [Deny, Audit]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values: [kamiwaza, kamiwaza-extensions]

Mirrored proxy image. The guard recognizes Istio's proxy image by its repository, listed in istioInitImagePrefixes. If your mesh pulls the proxy from another registry or under another name, for example a mirror such as registry.example.com/istio/proxyv2, add that repository before you apply the guard. End each entry with : and add a second entry ending in @, as the existing entries do. Otherwise the guard denies every pod Istio injects. A registry mirror that keeps the original image name, such as a containerd mirror, needs no change.

Apply it as the administrator:

kubectl apply -f kamiwaza-tenant-pod-guard.yaml

Check that the four objects exist, and that the installer is denied a privileged pod. The server-side dry run creates nothing:

kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding \
  kamiwaza-tenant-pod-guard kamiwaza-tenant-exec-guard

SA=system:serviceaccount:kamiwaza:kamiwaza-installer
kubectl create --dry-run=server -f - --as=$SA \
  --as-group=system:serviceaccounts \
  --as-group=system:serviceaccounts:kamiwaza \
  --as-group=system:authenticated <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: kamiwaza-pod-guard-probe
  namespace: kamiwaza
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  hostPID: true
  containers:
    - name: probe
      image: busybox:1.37
      securityContext:
        privileged: true
EOF
# Expect Forbidden, with a message that names
# ValidatingAdmissionPolicy 'kamiwaza-tenant-pod-guard'

Check that the installer is denied exec into istio-init. Admission runs before the API server looks the pod up, so the request needs no pod:

kubectl create --raw \
  "/api/v1/namespaces/kamiwaza/pods/kamiwaza-exec-guard-probe/exec?container=istio-init&command=true&stdout=true" \
  -f /dev/null --as=$SA \
  --as-group=system:serviceaccounts \
  --as-group=system:serviceaccounts:kamiwaza \
  --as-group=system:authenticated
# Expect Forbidden, with a message that names
# ValidatingAdmissionPolicy 'kamiwaza-tenant-exec-guard'.
# NotFound means the request got past the guard.

Existing installs. On a cluster that already runs Kamiwaza, roll the guard out in stages, so that it never denies a pod the platform needs:

  1. Apply it with validationActions: [Warn, Audit] in both bindings. The guard then reports what it would deny without denying it.

  2. Check every running pod in both namespaces the guard covers against it. Most pods are created by controllers, so a helm upgrade does not show the guard's warnings; submitting each pod's spec again as a server-side dry run does, and creates nothing:

    for ns in kamiwaza kamiwaza-extensions; do
      for pod in $(kubectl -n "$ns" get pods -o name); do
        kubectl -n "$ns" get "$pod" -o json \
          | jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.ownerReferences,
                    .metadata.creationTimestamp, .metadata.generateName, .status,
                    .spec.nodeName, .spec.ephemeralContainers)
                | .metadata.name += "-guard-check"' \
          | kubectl create --dry-run=server -f - 2>&1 >/dev/null \
          | grep -E "kamiwaza-tenant-pod-guard|^Error" && echo "  from $ns/$pod"
      done
    done
    # Expect no output, apart from "No resources found" for a namespace
    # with no pods

    Any Error line means that pod's spec could not be checked; resolve it before you continue.

    A warning about a hostPath volume means the install uses the single-node shared model cache (core.modelsStorage.sharedHostPath.enabled), a developer setting; turn it off and upgrade first. A warning about istio-init means the injected container does not match the guard's exception. Most often its proxy image is not in istioInitImagePrefixes. Otherwise, compare the pod's istio-init container with the exception in the manifest: the guard refuses an environment variable set with valueFrom or envFrom or named for a loader path, and any mount other than the service account token volume.

  3. When the check prints no warning and no Error line, change both bindings to validationActions: [Deny, Audit] and apply the file again.

Pods that were running before the guard keep running. The guard checks a pod when it is created or its spec changes.

How a denial shows up. A pod created directly, or an ephemeral container from kubectl debug, is refused with Forbidden and the guard's message. A Deployment, StatefulSet or Job is accepted, but its controller cannot create the pod: kubectl -n kamiwaza get events --field-selector reason=FailedCreate shows the guard's message, and the install fails on a setup job or the wait in Verify times out. The guard also denies node debug pods (kubectl debug node/...) in the kamiwaza namespace; start them in another namespace. kubectl exec or kubectl attach into istio-init is refused with Forbidden and the exec guard's message.

To remove the guard, delete the bindings first; that takes effect at once:

kubectl delete validatingadmissionpolicybinding kamiwaza-tenant-pod-guard kamiwaza-tenant-exec-guard
kubectl delete validatingadmissionpolicy kamiwaza-tenant-pod-guard kamiwaza-tenant-exec-guard

Kubernetes 1.28 and 1.29​

These versions do not serve the v1 ValidatingAdmissionPolicy API, so applying the guard fails with no matches for kind "ValidatingAdmissionPolicy" in version "admissionregistration.k8s.io/v1". Either upgrade the cluster to Kubernetes 1.30 or later and apply the guard, or enforce the same rules with the policy engine your cluster already runs, such as Kyverno or OPA Gatekeeper: Pod Security baseline in the Kamiwaza namespaces, with the istio-init exception exactly as the manifest above defines it, and a rule that refuses exec and attach into istio-init.

On OpenShift, the restricted-v2 SecurityContextConstraints already deny privileged and hostPath pods to a namespace admin, so the guard is optional there.

Istio and the ingress gateway​

Kamiwaza routes all traffic through an Istio mesh that the platform owns. Install and operate Istio the way your organization does; Kamiwaza does not install or configure it. The mesh must provide:

  • istiod, in sidecar mode. Ambient mode is not supported.
  • An ingress gateway exposed to your users on port 443, for example through a LoadBalancer Service or your existing load balancer.
  • An Istio Gateway resource (networking.istio.io) that selects that gateway, serves the Kamiwaza domain on HTTPS port 443 with the edge certificate, and serves or redirects port 80. Its hosts must include the Kamiwaza domain: Istio programs a route only where the Gateway's hosts match, so a mismatch leaves the platform unreachable while the install reports success.

Kamiwaza attaches its routes to this Gateway from its own namespace. It writes nothing into the gateway's namespace.

A cluster without Istio. If the cluster has no mesh yet, for example a fresh k0s node with no load balancer, this example sets up one that meets the requirements above. It installs Istio 1.30.3 with Istio's Helm charts, and exposes the gateway on ports 80 and 443 of a node address through the Service's externalIPs instead of a LoadBalancer. Replace <node-ip> with the node address your users reach:

helm repo add istio https://blob.istio.io/istio-release/charts
helm repo update
helm upgrade --install istio-base istio/base --version 1.30.3 \
  --namespace istio-system --create-namespace --wait
helm upgrade --install istiod istio/istiod --version 1.30.3 \
  --namespace istio-system --wait \
  --set global.hub=docker.io/istio
helm upgrade --install istio-ingressgateway istio/gateway --version 1.30.3 \
  --namespace istio-system --wait \
  --set service.type=ClusterIP --set 'service.externalIPs={<node-ip>}'

global.hub sets the registry for istiod and for every proxy it injects, the ingress gateway's included. Keep it: the 1.30 chart's default registry, registry.istio.io, does not serve anonymous pulls. If you mirror Istio's images, set global.hub to your mirror, add its proxy repository to istioInitImagePrefixes in the pod guard (see Mirrored proxy image), and apply the guard again.

Kubernetes deprecates externalIPs as of 1.36, so expect the API server to warn when the gateway Service is created. The kube-proxy that k0s runs by default still routes it. If the cluster has a load balancer, leave out both --set options: the chart then creates a LoadBalancer Service.

Then create the Gateway. Save this as kamiwaza-gateway.yaml, with your domain in both hosts lists, and apply it with kubectl apply -f kamiwaza-gateway.yaml:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: kamiwaza-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - kamiwaza.example.com
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - kamiwaza.example.com
      tls:
        mode: SIMPLE
        credentialName: kamiwaza-edge-tls

kamiwaza-edge-tls is the edge certificate Secret, which you create in istio-system. The release name istio-ingressgateway and this Gateway match the gateway defaults in the Record table below: name, namespace, selector, service account, and Service. In DNS, point your domain at <node-ip>.

Check:

kubectl -n istio-system get deploy istiod
kubectl get crd virtualservices.networking.istio.io authorizationpolicies.security.istio.io
kubectl auth can-i create virtualservices.networking.istio.io -n kamiwaza \
  --as=system:serviceaccount:kamiwaza:kamiwaza-installer   # yes
kubectl get gateways.networking.istio.io -A
kubectl -n <gateway-namespace> get gateways.networking.istio.io <gateway-name> -o yaml
# Expect a server on port 443, protocol HTTPS, whose hosts include your domain,
# and a tls.credentialName naming your edge certificate Secret

Record the gateway's coordinates. The install uses the defaults in the right column unless you record something different.

FactHow to find itDefault
Gateway name and namespacekubectl get gateways.networking.istio.io -Akamiwaza-gateway in istio-system
Mesh namespace (where istiod runs)kubectl get deploy -A | grep istiodistio-system
Gateway pod selectorspec.selector of the Gateway resource. Record every label as-is.istio: ingressgateway
Gateway workload's service accountkubectl -n <ns> get pod -l <selector> -o jsonpath='{.items[0].spec.serviceAccountName}'. A gateway installed with istioctl uses istio-ingressgateway-service-account.istio-ingressgateway
Gateway Service's in-cluster hostname<service-name>.<namespace>.svc.cluster.localistio-ingressgateway.istio-system.svc.cluster.local
Sidecar styleSee belowNative sidecars

Sidecar style. Kamiwaza expects Istio to inject istio-proxy as a Kubernetes native sidecar (an init container with restartPolicy: Always). If your mesh injects it as a regular container, record that: the install sets a value for it. To tell which, start a short-lived pod in the kamiwaza namespace and look for istio-proxy:

kubectl -n kamiwaza run kamiwaza-sidecar-check --image=busybox:1.37 --restart=Never -- sleep 300
kubectl -n kamiwaza wait --for=condition=Ready pod/kamiwaza-sidecar-check --timeout=120s
kubectl -n kamiwaza get pod kamiwaza-sidecar-check \
  -o jsonpath='{range .spec.initContainers[*]}init {.name} {.restartPolicy}{"\n"}{end}{range .spec.containers[*]}container {.name}{"\n"}{end}'
# "init istio-proxy Always": native sidecars.
# "container istio-proxy": classic injection.
# No istio-proxy line at all: the pod was not injected; check the namespace label.
kubectl -n kamiwaza delete pod kamiwaza-sidecar-check

A Pod Security restricted warning when this pod starts is expected; the namespace only warns and audits.

Edge certificate​

The gateway serves the Kamiwaza domain with a TLS certificate that you provide and rotate. Create it as a kubernetes.io/tls Secret in the gateway's namespace and name it in the Gateway's tls.credentialName. The certificate must cover the Kamiwaza domain in its subject alternative names.

If the certificate is not issued by a publicly trusted CA (a private CA, or self-signed), Kamiwaza workloads need its CA to verify the gateway. Publish the CA certificate, or for a self-signed certificate the certificate itself, in the Kamiwaza namespace:

kubectl -n kamiwaza create configmap kamiwaza-edge-trust \
  --from-file=ca-certificates.crt=./edge-ca.crt

This ConfigMap holds public certificates only. Never put a private key in it. Kamiwaza reads it by name when it exists; no install value refers to it.

Check:

kubectl -n <gateway-namespace> get secret <edge-secret> -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -subject -issuer -ext subjectAltName -enddate
# Expect your domain under subjectAltName and an end date in the future

Record: whether the certificate is self-signed (subject and issuer are the same). The install sets a value for a self-signed certificate.

DNS​

The Kamiwaza domain must resolve to the ingress gateway in two places:

  • For users: your DNS points the domain at the gateway's external address.
  • Inside the cluster: Kamiwaza services and apps call the platform at its public domain. Pods must resolve the domain to an address that reaches the gateway. If your cluster DNS forwards to a resolver that already answers for the domain, and pods can reach that address, nothing more is needed. Otherwise, add a cluster DNS entry that maps the domain to the gateway Service's in-cluster hostname.

With CoreDNS, add a rewrite line to the Corefile, above the kubernetes plugin:

rewrite name exact kamiwaza.example.com istio-ingressgateway.istio-system.svc.cluster.local

Some distributions, k0s among them, manage the CoreDNS configuration and can put it back on restart or upgrade. On those, make the change through the distribution's own configuration. On k0s, add a patch under spec.network.coreDNS.patches in the k0s configuration that replaces the coredns ConfigMap's Corefile with your edited copy (see k0s component patches), then restart k0s so it reads the configuration again. Make the same change on every controller node.

First find the node's configuration file and the datastore k0s runs on. A restart with a configuration that names a different datastore brings the cluster up empty:

systemctl cat k0scontroller | grep ExecStart
# --config=<file> names the configuration file. Without --config, k0s reads
# /etc/k0s/k0s.yaml if it exists, and otherwise runs on built-in defaults
sudo grep -E '^ +type: (etcd|kine)$' /run/k0s/k0s.yaml
# The datastore the running k0s uses: etcd or kine
  • If the configuration file exists, add the patch to it and leave spec.storage as it is.
  • If it does not, create /etc/k0s/k0s.yaml with the patch and a spec.storage.type equal to the running datastore; k0s fills in defaults for everything else. Do not generate the file with k0s config create. Its output sets type: etcd, while a node started with --single runs on kine, and restarting that node with type: etcd brings it up on a new, empty etcd datastore.

Start from the current Corefile (kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}'), add your line, and put the whole file in the patch. This is a complete file for a node that has none; to an existing file, add only spec.network.coreDNS:

# /etc/k0s/k0s.yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  storage:
    type: kine    # the datastore the check above printed
  network:
    coreDNS:
      patches:
        - target:
            kind: ConfigMap
            name: coredns
            namespace: kube-system
          patch:
            type: StrategicMergePatch
            content: |
              data:
                Corefile: |
                  .:53 {
                      # your complete edited Corefile
                  }

type: kine on its own selects k0s's default SQLite database under /var/lib/k0s, which is what --single uses. If ExecStart also sets --data-dir, copy kine.dataSource from /run/k0s/k0s.yaml into the file too.

Validate the file, take a backup, and restart k0s. k0s backup copies the configuration file, so it fails on a node that has none until you create it. The archive holds the datastore and the cluster's private keys: keep it off the node, somewhere safe. If the restart goes wrong, restore it as described in k0s backup and restore.

sudo "$(command -v k0s)" config validate --config /etc/k0s/k0s.yaml   # or the file --config names
sudo "$(command -v k0s)" backup --save-path "$HOME"
sudo systemctl restart k0scontroller

After any cluster DNS change, allow a minute or two for CoreDNS to reload before running the check.

Check from your workstation, and from a pod:

curl -sk -o /dev/null -w '%{http_code}\n' https://kamiwaza.example.com/
# Any HTTP status proves DNS and TLS reach the gateway. Before Kamiwaza is
# installed, expect 404.

kubectl -n default run kamiwaza-dns-check --restart=Never \
  --image=curlimages/curl --annotations=sidecar.istio.io/inject=false \
  --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":100,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"kamiwaza-dns-check","image":"curlimages/curl","args":["-sk","-o","/dev/null","-w","%{http_code}\\n","https://kamiwaza.example.com/"],"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}'
kubectl -n default wait --for=jsonpath='{.status.phase}'=Succeeded \
  pod/kamiwaza-dns-check --timeout=120s
kubectl -n default logs kamiwaza-dns-check
# Expect the same status from inside the cluster
kubectl -n default delete pod kamiwaza-dns-check

Record: the domain.

Block storage​

Kamiwaza's databases and caches use ReadWriteOnce persistent volumes. The cluster must have a StorageClass that provisions them. Record its name; the install names it explicitly.

Size the storage provider for Kamiwaza's volume requests plus your provider's replication and free-space reserves. A StorageClass existing does not prove its volumes can be provisioned; the write test below does. Node-local storage, such as local-path, works on a single node but does not replicate data.

Any reclaimPolicy works, including Delete, the local-path default. The policy decides only what happens to a volume once its claim is deleted, as at uninstall.

Kamiwaza requests these volumes by default, about 624 GiB in total. Apps you deploy later add their own.

VolumeDefault requestHoldsChange it with
kamiwaza-registry-data500 GiBModel and inference-engine cacheregistry.persistence.size
seaweedfs-data100 GiBBuilt-in object storeseaweedfs.persistence.size
core-postgres-data10 GiBPlatform database—
keycloak-postgres-data10 GiBIdentity database—
core-spicedb-postgres-data1 GiBAuthorization database—
core-etcd-data-core-etcd-0 to -21 GiB eachConfiguration store—

The model registry cache holds inference-engine images and the model files your users deploy. If you serve only external models (for example, through a cloud provider), or your storage is small, set registry.persistence.size lower, for example 100Gi, in your install values before the first install. An existing volume keeps its size on upgrade.

Check the classes, then run a write test in the Kamiwaza namespace: write a marker, replace the pod, and read the marker back.

kubectl get storageclass
# Expect one class marked (default), or note the class you will name at install

cat > kamiwaza-storage-check.yaml <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kamiwaza-storage-check
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
  # storageClassName: <class>    # uncomment if you will not use the default
---
apiVersion: v1
kind: Pod
metadata:
  name: kamiwaza-storage-check
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: check
      image: busybox:1.37
      command: ["sh", "-c", "sleep 3600"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: kamiwaza-storage-check
EOF

kubectl -n kamiwaza apply -f kamiwaza-storage-check.yaml
kubectl -n kamiwaza wait --for=condition=Ready pod/kamiwaza-storage-check --timeout=5m
kubectl -n kamiwaza exec kamiwaza-storage-check -- sh -c 'echo ok > /data/marker'

# Replace the pod; the claim and its volume stay
kubectl -n kamiwaza delete pod kamiwaza-storage-check
kubectl -n kamiwaza apply -f kamiwaza-storage-check.yaml
kubectl -n kamiwaza wait --for=condition=Ready pod/kamiwaza-storage-check --timeout=5m
kubectl -n kamiwaza exec kamiwaza-storage-check -- cat /data/marker     # expect: ok

kubectl -n kamiwaza delete -f kamiwaza-storage-check.yaml

A class with volumeBindingMode: WaitForFirstConsumer binds the claim only once the pod is scheduled, so a Pending claim before the pod starts is normal.

Record: the StorageClass name.

Object storage​

Workrooms, the Skills Library, and other features store objects. Choose one:

  • The built-in object store (default). Kamiwaza runs SeaweedFS in its namespace, backed by a ReadWriteOnce volume from your block storage. Nothing to prepare.
  • Your own S3-compatible service. See S3 workroom storage.

Kubernetes API addresses​

Some Kamiwaza apps call the Kubernetes API from inside the namespace, and Kamiwaza allows that traffic with NetworkPolicy. Many CNIs, including the ones k0s and OpenShift use, evaluate NetworkPolicy after a Service address is translated to the API server's real address. So the policy needs both the kubernetes Service address and every API server endpoint address. With only the first, apps install cleanly and then fail at runtime.

Record both sets of addresses, as /32 for IPv4 or /128 for IPv6:

kubectl -n default get service kubernetes -o jsonpath='{.spec.clusterIPs[*]}{"\n"}'
kubectl -n default get endpointslices -l kubernetes.io/service-name=kubernetes \
  -o jsonpath='{.items[*].endpoints[*].addresses[*]}{"\n"}'

Copy the addresses exactly. A wrong address still installs cleanly, and the apps then fail at runtime. If the API server's addresses change (for example, a control-plane node is replaced), update them in the install values and upgrade.

If your CNI enforces NetworkPolicy before Service address translation, the endpoint addresses are not needed; the Service address alone works. Record it anyway.

Registry access​

Kamiwaza's charts and images are published to a public registry and pull without credentials. Nodes must be able to pull from it; see Outbound network access.

If you mirror images into your own registry and it requires credentials, create a kubernetes.io/dockerconfigjson pull Secret in the Kamiwaza namespace. Kamiwaza uses only namespaced pull Secrets, never node-wide registry credentials. Build the Secret from a file, so the password never appears in a command line or your shell history:

read -rs REGISTRY_PASSWORD      # typed, not echoed
(umask 077; d=$(mktemp -d)
 printf '{"auths":{"%s":{"auth":"%s"}}}' '<registry-host>' \
   "$(printf '%s:%s' '<username>' "$REGISTRY_PASSWORD" | base64 | tr -d '\n')" > "$d/config.json"
 kubectl -n kamiwaza create secret generic kamiwaza-pull \
   --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson="$d/config.json"
 rm -rf "$d")
unset REGISTRY_PASSWORD

<registry-host> is the registry's host name, with its port if it has one, as it appears in the image references.

If the registry is GitHub Container Registry (ghcr.io), the password is a GitHub token that carries the read:packages scope for the organization that owns the images. A token without it, such as one from gh auth login with its default scopes, is refused, and every pod that uses a private image goes to ImagePullBackOff with 403 Forbidden ... The token provided does not match expected scopes while the install keeps waiting.

A build that has not been published to the public registry yet, such as a pre-release Kamiwaza provided by your Kamiwaza contact, pulls its images from a private registry and needs this Secret. Create it before you run the install. If your Kubernetes substrate was prepared with Kamiwaza's setup automation, its hand-off values name the Secret in global.imagePullSecrets (kamiwaza-ghcr-pull); create the Secret under that name instead of kamiwaza-pull.

Record: the pull Secret's name, kamiwaza-pull above. The install lists it in global.imagePullSecrets. That list does not reach the engine pods of tenant-mode inference: they carry only the staging credential, which defaults to the first entry in global.imagePullSecrets.

Node kernel limits​

Every Istio sidecar watches files with inotify. Distribution defaults are too low for the number of pods Kamiwaza runs, and sidecars then crash with too many open files. Set these floors on every node, persistently:

cat <<'EOF' | sudo tee /etc/sysctl.d/99-kamiwaza-inotify.conf
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 1048576
fs.inotify.max_queued_events = 16384
EOF
sudo sysctl --system

Check on each node:

sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches fs.inotify.max_queued_events
# Expect values at or above 8192, 1048576, and 16384

Managed platforms that do not give you node access usually expose these limits through a node configuration resource; use your platform's mechanism.

Outbound network access​

Nodes and pods need outbound HTTPS (port 443) to:

  • ghcr.io and pkg-containers.githubusercontent.com, for Kamiwaza charts and images;
  • the model sources you use, for example huggingface.co and its content hosts, or your cloud model provider's endpoint.

The checks on this page use the public busybox and curlimages/curl images from Docker Hub. If your cluster cannot reach Docker Hub, mirror them to your registry.

For installs with no outbound access, see Install Without Internet Access.

GPU nodes​

GPU inference is optional. To serve models on NVIDIA GPUs, prepare each GPU node as described in GPU Nodes: a driver that meets the engine images' floor, persistence mode, the NVIDIA container runtime and the nvidia RuntimeClass, the NVIDIA GPU Operator, and no NoSchedule or NoExecute taint on the node. Kamiwaza installs none of these.

Prepared nodes are not enough on their own: a default install cannot use GPUs. After the install, the cluster owner turns GPU serving on with Tenant-mode inference.

Check:

kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'
# Expect a count on each GPU node
kubectl get runtimeclass nvidia
# Expect a RuntimeClass named nvidia, with handler nvidia

Record: which nodes have GPUs, and the GPU model on each. The cluster owner needs them to publish a signed profile.

Multi-node clusters​

On a cluster with more than one node, also provide:

  • Storage that survives a node loss. Use a StorageClass whose volumes are replicated or network-attached, such as a CSI driver for your SAN or cloud block storage, or a replicated provider like Longhorn. Node-local storage ties each database to one node: if that node is lost, so is the data.
  • Every GPU node prepared. Prepare each GPU node as in GPU nodes, including nodes added later.
  • A CNI that enforces NetworkPolicy. Kamiwaza isolates apps and jobs from each other with NetworkPolicy. A CNI that accepts policies but does not enforce them leaves that isolation off without any error.
  • The inotify floor on every node, including nodes added later.
  • The API endpoint addresses of every control-plane node in the recorded Kubernetes API addresses.

Next step​

When every check passes, hand the recorded facts and the installer kubeconfig to whoever runs the install, and continue to Install on Kubernetes with Istio.