Prepare GPU nodes

A platform administrator prepares GPU nodes before Kamiwaza serves models on them. Kamiwaza installs nothing on a node: no driver, no container runtime configuration, and no GPU operator. This page covers each node-side requirement for NVIDIA GPUs, with a check and its expected output.

A default install cannot use GPUs even on prepared nodes. It installs without node reads and with tenant-mode inference off, and the install prints WARNING: core cannot see GPUs on this install. Prepare the nodes here, then turn on GPU serving with Tenant-mode inference, which uses a profile that the cluster owner signs.

Before you start

You need	Notes
Root access on each GPU node	The commands are for Ubuntu 22.04 and 24.04 nodes. On another distribution, use its packages; the checks are the same.
Cluster-admin access	To install the GPU Operator and create the nvidia RuntimeClass.
Packages and charts from inside your network	The node packages come from your mirror of the Ubuntu archive and of NVIDIA's container toolkit repository, and the GPU Operator chart and its images come from your registry.
kubectl, helm, and jq	On the machine you run the cluster checks from.
Two public test images	The checks run docker.io/library/ubuntu:24.04 and docker.io/nvidia/cuda:13.0.3-base-ubuntu24.04. With internet access, use them as named. In an air gap, copy them into your registry first; the checks write them as <YOUR_REGISTRY>/library/ubuntu:24.04 and <YOUR_REGISTRY>/nvidia/cuda:13.0.3-base-ubuntu24.04. If your registry needs credentials, the check pods run in the default namespace: create a pull Secret there and add imagePullSecrets: [{name: <SECRET_NAME>}] to each check pod's spec.

These steps were verified on a four-GPU NVIDIA A100 80GB host running k0s 1.36, the R580 server driver, and GPU Operator v26.7.1, except Refuse GPU requests by environment and the checks for it in step 4, step 5, and the VRAM plugin section. Its two settings were verified on an NVIDIA GB10 (arm64) host running k0s 1.36, NVIDIA Container Toolkit 1.19.1 installed on the host, GPU Operator v25.10.1, and a VRAM plugin that passes the device-list mount: with both set, an unprivileged pod that asks for every GPU by environment sees none, and a VRAM plugin share still gets its GPU. Those checks test the same behavior, but were not run as written. The GPU Operator-managed toolkit variant was not verified. Where a step differs on k0s, the step says so.

Summary

Requirement	Check
Supported GPU and driver	nvidia-smi reports the driver and CUDA version the engines need
Driver installed	The driver loads, with Secure Boot on if the node uses it
Persistence mode	nvidia-smi answers in well under a second
Container runtime	The node advertises an nvidia runtime handler
GPU requests by environment refused (required for tenant-mode GPU serving)	Both toolkit settings are set, and an unprivileged pod that asks for every GPU by environment sees none
nvidia RuntimeClass	A privileged pod with runtimeClassName: nvidia sees the GPUs
GPU Operator	Nodes advertise nvidia.com/gpu, and nvidia.com/mig-* for MIG
MIG layout (optional)	Nodes advertise the MIG resources you configured
No taints on GPU nodes	No GPU node carries a NoSchedule or NoExecute taint
Pinned GPU stack	Driver, toolkit, and kernel packages are held
VRAM plugin (optional)	Nodes advertise kamiwaza.ai/vram-gb-gpu-<N> for each shared GPU

Supported GPUs and driver floors

Kamiwaza serves models on NVIDIA GPUs. Each engine image is built against one CUDA major, and the host driver must support that major. nvidia-smi prints the highest CUDA version the installed driver supports in its header, as CUDA Version: 13.0.

Engine image	Engines	Minimum host driver	Lowest GPU compute capability
CUDA 13	llama.cpp, vLLM	580.65.06 (R580)	7.5 for llama.cpp; 8.0 for vLLM on amd64 and 12.1 (GB10) on arm64, and vLLM requires bf16
CUDA 12	llama.cpp	525.60.13; R570 or later for Blackwell GPUs	7.0
CUDA 12, built with CUDA 12.8	Diffusion	R570 or later recommended; not enforced	8.0, and the engine requires bf16

A GPU older than Volta (compute capability 7.0) has no engine image. Diffusion ships for amd64 only. vLLM ships for amd64, and for arm64 on GB10 only. llama.cpp ships for amd64 and arm64. See Engines for what each engine serves.

Check on each GPU node:

nvidia-smi --query-gpu=index,name,driver_version,compute_cap --format=csv,noheader
nvidia-smi | grep -o 'CUDA Version: [0-9.]*'

0, NVIDIA A100 80GB PCIe, 580.178.04, 8.0
1, NVIDIA A100 80GB PCIe, 580.178.04, 8.0
CUDA Version: 13.0

Expect every GPU on a node to report the same name and compute capability. A profile binding cannot single out one kind of GPU on a node that has two (see Tenant-mode inference), so keep such a node out of tenant-mode GPU serving.

Keep a node with two kinds of GPU out

A binding selects nodes by architecture only, so on a cluster with other GPU nodes a tenant-mode GPU pod could land on this one. After you install the GPU Operator in step 5, turn off the operator's components on the node:

kubectl label nodes <NODE_NAME> nvidia.com/gpu.deploy.operands=false --overwrite

The operator then removes its device plugin, validator, GPU feature discovery, DCGM exporter and mig-manager from the node, so the scheduler can no longer give any pod one of its GPUs. That holds for every pod, not only tenant-mode ones; pods already running keep the GPUs they have. The host's driver and container toolkit stay, so the step 4 check still passes there. In the step 5 check, the node shows 0 for its NVIDIA resources, or none at all. To undo it, remove the label: kubectl label nodes <NODE_NAME> nvidia.com/gpu.deploy.operands-.

A cluster whose only GPU node has two kinds of GPU, such as a single-node cluster, needs none of this: leave tenant-mode GPU serving off there, and publish no GPU binding for it.

Step 1: Install the driver

Upgrade the node and reboot before you install the driver. The kernel meta-package can be ahead of the kernel the node booted, and the driver's kernel modules track the meta-package:

export DEBIAN_FRONTEND=noninteractive
sudo -E apt-get update
sudo -E apt-get -y full-upgrade
sudo systemctl reboot

List the driver branches the archive offers for the node's GPUs:

sudo apt-get install -y ubuntu-drivers-common
sudo ubuntu-drivers list --gpgpu

nvidia-driver-580-server, (kernel modules provided by linux-modules-nvidia-580-server-generic)

Install the driver together with the prebuilt kernel module package that the list names. The flavor at the end of its name, -generic above, matches your kernel. Naming the module package stops apt from building the module with DKMS:

sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
  linux-modules-nvidia-580-server-generic nvidia-driver-580-server
sudo systemctl reboot

Check:

nvidia-smi -L

GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-8d898926-...)

Expect one line per GPU. Right after a driver load, nvidia-smi can print couldn't communicate with the NVIDIA driver for a few seconds while each GPU initializes.

Secure Boot

With UEFI Secure Boot on, the kernel loads only signed modules. Check the state:

mokutil --sb-state

SecureBoot enabled

The prebuilt module packages from step 1 are signed by Canonical and load without further steps. A module that DKMS builds is signed with a Machine Owner Key (MOK) on the node, and the firmware must trust that key. Enroll it once per node, from a console that reaches the pre-boot screen (physical, or a BMC or serial console); SSH cannot do the enrollment:

1. Create the key, if the node has none, and stage its enrollment. mokutil asks for a one-time password twice:

   sudo update-secureboot-policy --new-key
   sudo mokutil --import /var/lib/shim-signed/mok/MOK.der

2. Reboot. On the blue MOK manager screen, which waits about 10 seconds for a key press, select Enroll MOK, Continue, Yes, enter the one-time password, and select Reboot.

3. Rebuild the module so DKMS signs it with the enrolled key. Name the DKMS package of your driver branch, nvidia-dkms-580-server for the R580 server driver:

   sudo dpkg-reconfigure nvidia-dkms-580-server
   sudo systemctl reboot

Check:

mokutil --test-key /var/lib/shim-signed/mok/MOK.der
modinfo -F signer nvidia
nvidia-smi -L

Expect is already enrolled, a signer name, and one line per GPU. If the enrollment screen timed out, the request stays pending; reboot again. If the password is lost, clear the request with sudo mokutil --revoke-import and repeat step 1.

Step 2: Turn on persistence mode

Without persistence mode, every nvidia-smi call initializes each GPU again. On a four-GPU host that took about 13 seconds per call, long enough for the GPU Operator's validator to exceed the kubelet's 2-minute container start timeout (RunContainerError: context deadline exceeded), and the operator never became ready. Ubuntu's nvidia-persistenced service starts with --no-persistence-mode; override it:

sudo mkdir -p /etc/systemd/system/nvidia-persistenced.service.d
sudo tee /etc/systemd/system/nvidia-persistenced.service.d/10-persistence-mode.conf >/dev/null <<'EOF'
[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --persistence-mode --verbose
EOF
sudo systemctl daemon-reload
sudo systemctl restart nvidia-persistenced

Check:

nvidia-smi --query-gpu=index,persistence_mode --format=csv,noheader
/usr/bin/time -f "%es" nvidia-smi -L >/dev/null

Expect Enabled for every GPU, and a time well under a second (0.04 seconds on the verification host).

Step 3: Configure the container runtime

Install the NVIDIA Container Toolkit from your mirror of NVIDIA's container toolkit repository:

sudo DEBIAN_FRONTEND=noninteractive apt-get install -y nvidia-container-toolkit
nvidia-ctk --version

NVIDIA Container Toolkit CLI version 1.20.1

Then add an nvidia runtime to containerd. Keep runc as the default runtime: only pods that name the nvidia RuntimeClass use the NVIDIA runtime.

On a node whose containerd configuration you manage, let the toolkit write it, then restart containerd:

sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd

On k0s, k0s owns /etc/k0s/containerd.toml, so write the runtime as a drop-in by hand instead. k0s 1.36 runs containerd 2, whose drop-ins use config version 3:

sudo tee /etc/k0s/containerd.d/nvidia.toml >/dev/null <<'EOF'
version = 3

[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
  privileged_without_host_devices = false
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options]
  BinaryName = "/usr/bin/nvidia-container-runtime"
EOF
sudo systemctl restart k0sworker   # k0scontroller on a controller that runs workloads

Check from the cluster, for each GPU node. The node also lists an unnamed default handler, which the check leaves out:

kubectl get node <NODE_NAME> -o json | jq -r '[.status.runtimeHandlers[]?.name | select(. != "")] | join(" ")'

nvidia runc

If your nodes do not report runtime handlers, this prints nothing; the pod check below is then the proof.

Refuse GPU requests by environment

Required on every node that serves tenant-mode GPU deployments.

By default the NVIDIA runtime gives a container on the nvidia RuntimeClass every GPU that its NVIDIA_VISIBLE_DEVICES variable names, whether the pod, a ConfigMap, or the image sets it. CUDA images, Kamiwaza's engine images among them, set it to all. So any pod that can use the nvidia RuntimeClass sees every GPU on the node without requesting one, and a quota on GPU resources does not limit it. Two NVIDIA Container Toolkit settings close this. With both set, the runtime gives an unprivileged container only the GPUs a device plugin allocated to it:

Setting	Effect
accept-nvidia-visible-devices-as-volume-mounts = true	The runtime accepts the device list a device plugin passes as volume mounts.
accept-nvidia-visible-devices-envvar-when-unprivileged = false	The runtime ignores NVIDIA_VISIBLE_DEVICES in an unprivileged container.

The order matters: accept volume mounts first, then make sure no device plugin on the node passes GPUs by environment, and refuse the environment last. A device plugin that passes volume mounts to a runtime that does not accept them hands out no usable GPU, and refusing the environment while a device plugin still uses it leaves every new GPU pod without a GPU.

On a node that runs no GPU pods yet, set both now. The GPU Operator's device plugin from step 5 passes GPUs through CDI by default, not by environment:

sudo nvidia-ctk config --in-place --set accept-nvidia-visible-devices-as-volume-mounts=true
sudo nvidia-ctk config --in-place --set accept-nvidia-visible-devices-envvar-when-unprivileged=false

On a node that already serves GPUs, follow Turn the setting on for a node that serves GPUs instead.

With the toolkit installed on the host, the runtime reads its configuration for every new container, so neither setting needs a restart. Privileged containers, such as the GPU Operator's own pods, are not affected. The pod security guard keeps tenant pods from being privileged, adding capabilities, or mounting host paths, which are the ways around the setting.

If the GPU Operator manages the toolkit (toolkit.enabled: true), the commands above do not reach it. Its runtime reads /usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml, under the operator's default toolkit.installDir, and takes both settings from the operator's toolkit.env. Add them to gpu-operator-values.yaml as strings, under the toolkit: key the file already has, beside any toolkit.env entries you already have:

toolkit:
  enabled: true
  env:
    - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS
      value: "true"
    - name: ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
      value: "false"

Keep one toolkit: key. If a values file has two, Helm keeps the last one without a warning, and drops either these entries or enabled: true.

Apply the values with the helm upgrade --install command from step 5. Helm changes only the operator's ClusterPolicy, and the operator updates the toolkit DaemonSet afterwards, so rollout status run straight after Helm can return before anything changed. Wait until the DaemonSet carries the new value, then for its rollout. Name the DaemonSet, the variable, and its new value:

DS=nvidia-container-toolkit-daemonset
NAME=ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED
VALUE=false

Then wait. Later steps run this same wait with other values:

for i in $(seq 60); do   # up to five minutes for the operator to update the DaemonSet
  CURRENT=$(kubectl -n gpu-operator get daemonset "$DS" \
    -o jsonpath="{.spec.template.spec.containers[*].env[?(@.name==\"$NAME\")].value}")
  [ "$CURRENT" = "$VALUE" ] && break
  sleep 5
done
echo "$NAME=$CURRENT"
kubectl -n gpu-operator rollout status "daemonset/$DS" --timeout=10m

Expect ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED=false, then successfully rolled out. Any other value on the first line means the operator has not applied the change yet, and the rollout result does not cover it. The DaemonSet replaces one node's pod at a time; on a large cluster, raise the timeout. Each toolkit pod it replaces removes the nvidia runtime from containerd and restarts containerd, and its replacement adds the runtime back and restarts containerd again. Running containers keep running, but a pod on the nvidia RuntimeClass that starts on that node meanwhile fails until the runtime is back, and is retried.

The setting applies to the whole node

Every other workload on the node that selects GPUs by environment stops seeing them: unprivileged pods that set NVIDIA_VISIBLE_DEVICES instead of requesting nvidia.com/gpu, and docker run --gpus or Podman with the NVIDIA runtime without --privileged. Dedicate tenant-mode GPU nodes to Kamiwaza, or move those workloads to device-plugin requests before you turn the setting on.

Check on each GPU node:

grep -E '^accept-nvidia-visible-devices' /etc/nvidia-container-runtime/config.toml

accept-nvidia-visible-devices-as-volume-mounts = true
accept-nvidia-visible-devices-envvar-when-unprivileged = false

With a GPU Operator-managed toolkit, check the file its runtime reads instead, and expect the same two lines:

grep -E '^accept-nvidia-visible-devices' /usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml

The step 4 check proves it from the cluster.

The setting does not cover the variables that add other devices to a container that already has a GPU: NVIDIA_MOFED, NVIDIA_GDS, NVIDIA_GDRCOPY, and NVIDIA_NVSWITCH. On a node with InfiniBand or RDMA devices, a pod that has a GPU can add them with NVIDIA_MOFED=enabled, and RDMA traffic is not subject to Kubernetes network policy. No toolkit setting turns these off.

To turn the setting off again, for new containers at once:

sudo nvidia-ctk config --in-place --set accept-nvidia-visible-devices-envvar-when-unprivileged=true

Leave volume mounts accepted, so a device plugin that passes them keeps working. With the environment accepted again, any pod on the nvidia RuntimeClass can see every GPU on the node.

With a GPU Operator-managed toolkit, set ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED to "true" in toolkit.env instead, apply the values, and run the wait above with DS=nvidia-container-toolkit-daemonset, NAME=ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED, and VALUE=true.

Turn the setting on for a node that serves GPUs

1. If the VRAM plugin shares GPUs on the node, first upgrade it to the plugin from Kamiwaza 1.3.4 or later; see Share GPUs with the VRAM plugin. The plugin from Kamiwaza 1.3.3 and earlier passes each share's GPU by environment only. Then run the VRAM plugin's share check, and expect its device-list mount line: it proves the plugin passes the mount while the node still accepts the environment.

2. Accept volume mounts:

   sudo nvidia-ctk config --in-place --set accept-nvidia-visible-devices-as-volume-mounts=true

3. Read how the GPU Operator's device plugin passes GPUs, as in step 5. With CDI, the default, change nothing. If it prints envvar, or nothing, set volume-mounts as step 5 describes, and apply the values. Then name the device plugin's DaemonSet, variable, and value, and run the wait from Refuse GPU requests by environment:

   DS=nvidia-device-plugin-daemonset
   NAME=DEVICE_LIST_STRATEGY
   VALUE=volume-mounts

4. Stop every GPU deployment in Kamiwaza and deploy it again, and check that each one serves. Do it now, after steps 1 to 3 and before step 5: a pod created now gets its GPU through CDI or a device-list mount, which the runtime still honors once it refuses the environment. A pod created before step 3 can hold its GPU by environment only, and when its container restarts after step 5, it starts without a GPU.

5. Refuse the environment:

   sudo nvidia-ctk config --in-place --set accept-nvidia-visible-devices-envvar-when-unprivileged=false

6. Run the checks above, the checks in step 4 and step 5, and the VRAM plugin's share check. In the step 5 allocation check, set COUNT to no more than the GPUs the node has free.

With a GPU Operator-managed toolkit, steps 2 and 5 add the matching toolkit.env entry instead: ACCEPT_NVIDIA_VISIBLE_DEVICES_AS_VOLUME_MOUNTS with "true", then ACCEPT_NVIDIA_VISIBLE_DEVICES_ENVVAR_WHEN_UNPRIVILEGED with "false". Apply each with the step 5 helm upgrade --install command, then wait as in Refuse GPU requests by environment, with DS=nvidia-container-toolkit-daemonset and NAME and VALUE set to the entry you added, before the next step.

Step 4: Create the nvidia RuntimeClass

cat <<'EOF' | kubectl apply -f -
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: nvidia
EOF

Check that a privileged pod on the nvidia RuntimeClass sees the GPUs. It is privileged because the node refuses GPU requests by environment from unprivileged pods (step 3), and it runs in default, which the pod security guard does not cover. The runtime adds nvidia-smi to the container, so a plain Ubuntu image from your registry is enough. The pod is pinned to one node; set GPU_NODE to its name, and run the check once for each GPU node. The pod has no tolerations, so a NoSchedule or NoExecute taint on the node keeps it Pending. If your GPU nodes carry either, do step 7 first, or rerun this check after it.

If the API server refuses the pod with violates PodSecurity, your cluster enforces Pod Security on default, through a label on the namespace or a cluster-wide default. Then run this page's checks in a namespace of their own: create one, label it pod-security.kubernetes.io/enforce=privileged, use it in place of default in each check, and delete it when you are done.

GPU_NODE="<NODE_NAME>"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-runtime-check
  namespace: default
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: ${GPU_NODE}
  containers:
  - name: smi
    image: <YOUR_REGISTRY>/library/ubuntu:24.04
    command: ["nvidia-smi", "-L"]
    securityContext:
      privileged: true
    env:
    - {name: NVIDIA_VISIBLE_DEVICES, value: all}
    - {name: NVIDIA_DRIVER_CAPABILITIES, value: utility}
EOF
kubectl -n default wait --for=jsonpath='{.status.phase}'=Succeeded pod/nvidia-runtime-check --timeout=5m
kubectl -n default logs nvidia-runtime-check
kubectl -n default delete pod nvidia-runtime-check

Expect one GPU n: ... line per GPU on the node.

Check that the same request from an unprivileged pod gets no GPU. It runs on the node the check above set in GPU_NODE, and stops with an error if GPU_NODE is not set:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-env-refusal-check
  namespace: default
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: ${GPU_NODE:?set GPU_NODE to the node name, as in the check above}
  containers:
  - name: env
    image: <YOUR_REGISTRY>/library/ubuntu:24.04
    command: ["sh", "-c", "ls /dev/nvidia[0-9]* 2>/dev/null || echo 'no GPU'"]
    env:
    - {name: NVIDIA_VISIBLE_DEVICES, value: all}
EOF
kubectl -n default wait --for=jsonpath='{.status.phase}'=Succeeded pod/nvidia-env-refusal-check --timeout=5m
kubectl -n default logs nvidia-env-refusal-check
kubectl -n default delete pod nvidia-env-refusal-check

Expect no GPU. A list of /dev/nvidia<N> devices means the runtime still honors NVIDIA_VISIBLE_DEVICES; check the settings in step 3.

Use only the nvidia RuntimeClass. The GPU Operator in step 5 also creates nvidia-cdi and nvidia-legacy RuntimeClasses; on a node where the operator does not manage the toolkit, those have no containerd handler, and pods that name them fail to start. The operator takes ownership of the nvidia RuntimeClass, and uninstalling the operator deletes it.

Step 5: Install the GPU Operator

The NVIDIA GPU Operator's device plugin advertises GPUs to Kubernetes as nvidia.com/gpu, and MIG partitions as nvidia.com/mig-<profile>. With the driver and toolkit on the host, the operator must not install its own.

Save these values as gpu-operator-values.yaml. The mig and migManager blocks are needed only for MIG; keep them anyway, since they are harmless without it:

driver:
  enabled: false        # the host driver from step 1
toolkit:
  enabled: false        # the host toolkit and runtime from step 3
mig:
  strategy: mixed       # whole GPUs as nvidia.com/gpu, partitions as nvidia.com/mig-<profile>
# On k0s with its default kubelet root, uncomment the next two lines. Leave them
# commented on a k0s started with --kubelet-root-dir=/var/lib/kubelet, as
# Kamiwaza's setup tooling starts it and as the VRAM plugin needs.
# hostPaths:
#   kubeletRootDir: /var/lib/k0s/kubelet
migManager:
  # Persistence mode (step 2) keeps the GPUs open, so mig-manager must stop
  # nvidia-persistenced before a MIG change and start it again afterwards.
  gpuClientsConfig:
    name: gpu-clients
extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: gpu-clients
    data:
      clients.yaml: |
        version: v1
        systemd-services:
          - nvidia-persistenced.service
          - nvidia-dcgm.service
          - dcgm.service
          - dcgm-exporter.service

Install the operator chart from your registry, with its images mirrored there. With the values above, GPU Operator v26.7.1 runs these images; for another version, read the tags from helm show values:

Image	Values key to point at your copy
nvcr.io/nvidia/gpu-operator:v26.7.1	operator.repository and validator.repository
nvcr.io/nvidia/k8s-device-plugin:v0.20.1	devicePlugin.repository and gfd.repository
nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.15.1	migManager.repository
nvcr.io/nvidia/k8s/dcgm-exporter:4.6.1-4.8.4-distroless	dcgmExporter.repository
registry.k8s.io/nfd/node-feature-discovery:v0.19.0	node-feature-discovery.image.repository

Copy each image to your registry under the same path, e.g. <YOUR_REGISTRY>/nvidia/gpu-operator:v26.7.1. Then point the chart at your copies: save these values as gpu-operator-images.yaml. Each repository value is the image's path without its last segment, except node-feature-discovery.image.repository, which is the full image path. The chart supplies the tags in the table:

operator:
  repository: <YOUR_REGISTRY>/nvidia
validator:
  repository: <YOUR_REGISTRY>/nvidia
devicePlugin:
  repository: <YOUR_REGISTRY>/nvidia
gfd:
  repository: <YOUR_REGISTRY>/nvidia
migManager:
  repository: <YOUR_REGISTRY>/nvidia/cloud-native
dcgmExporter:
  repository: <YOUR_REGISTRY>/nvidia/k8s
node-feature-discovery:
  image:
    repository: <YOUR_REGISTRY>/nfd/node-feature-discovery

If your registry needs credentials, create a pull Secret in the gpu-operator namespace and list it under each component's imagePullSecrets.

Install with both values files. Helm merges them, so the migManager keys in each file apply together:

helm upgrade --install gpu-operator <GPU_OPERATOR_CHART> --version <GPU_OPERATOR_VERSION> \
  -n gpu-operator --create-namespace -f gpu-operator-values.yaml -f gpu-operator-images.yaml \
  --wait --timeout 10m

Check:

kubectl get clusterpolicy -o jsonpath='{.items[0].status.state}'; echo
kubectl get node <NODE_NAME> -o jsonpath='{.status.allocatable}' | tr ',' '\n' | grep nvidia

ready
"nvidia.com/gpu":"4"
"nvidia.com/mig-3g.40gb":"0"

With mig.strategy: mixed, a node whose GPUs support MIG can also list MIG resources with a count of 0 until you partition a GPU in step 6. A node that has never had MIG instances may not list them at all.

The device plugin must not pass GPUs by environment, which the node refuses (step 3). Leave cdi.enabled at the chart's default, true: the device plugin then passes GPUs through CDI and needs no change. If you set cdi.enabled: false, also add this to gpu-operator-values.yaml, never envvar:

devicePlugin:
  env:
    - name: DEVICE_LIST_STRATEGY
      value: volume-mounts

Check how the device plugin passes GPUs:

kubectl -n gpu-operator get daemonset nvidia-device-plugin-daemonset \
  -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="DEVICE_LIST_STRATEGY")].value}'; echo

Expect a CDI strategy, such as cdi-annotations,cdi-cri, or volume-mounts. envvar, or no output, which means the plugin's default of envvar, gives every new GPU pod no GPU on a node that refuses the environment: set volume-mounts as above.

Check that a pod gets exactly the GPUs it requests. Set COUNT to the number of GPUs to request. Use a CUDA base image from your registry whose CUDA version is at or below the CUDA Version from the driver check; the container refuses to start on an older driver. Like the step 4 check, the pod cannot run on a node with a NoSchedule or NoExecute taint. If your GPU nodes carry either, do step 7 first, or rerun this check after it.

COUNT=2
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-allocation-check
  namespace: default
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  containers:
  - name: cuda
    image: <YOUR_REGISTRY>/nvidia/cuda:13.0.3-base-ubuntu24.04
    command: ["bash", "-c", "nvidia-smi -L; nvidia-smi topo -p2p r"]
    resources:
      limits:
        nvidia.com/gpu: $COUNT
EOF
kubectl -n default wait --for=jsonpath='{.status.phase}'=Succeeded pod/gpu-allocation-check --timeout=5m
kubectl -n default logs gpu-allocation-check
kubectl -n default delete pod gpu-allocation-check

Expect COUNT GPU lines, even though CUDA images ask for every GPU.

Step 6: Partition GPUs with MIG (optional)

MIG divides a supported GPU into hardware-isolated instances. Describe the layouts the node can take in the mig-manager configuration, then select one with a node label. Add a configuration to gpu-operator-values.yaml under migManager, e.g., one that leaves GPUs 0 and 1 whole and partitions GPUs 2 and 3 into two 3g.40gb instances each. On a host bridged in pairs, keeping one bridged pair whole is what lets the whole GPUs serve tensor parallelism; see MIG and tensor parallelism.

migManager:
  config:
    create: true
    name: custom-mig-config
    default: all-disabled
    data:
      config.yaml: |-
        version: v1
        mig-configs:
          all-disabled:
            - devices: all
              mig-enabled: false
          gpu23-2x3g40gb:
            - devices: [0, 1]
              mig-enabled: false
            - devices: [2, 3]
              mig-enabled: true
              mig-devices:
                "3g.40gb": 2
  gpuClientsConfig:
    name: gpu-clients

Rerun the step 5 install with the updated values. mig-manager reads the new configuration only after the ConfigMap reaches its pod, and it acts only when the node label changes, so a label set too early fails with selected mig-config not present and is not retried. Confirm the ConfigMap holds the layout, then restart mig-manager and wait for it to be ready:

kubectl -n gpu-operator get configmap custom-mig-config \
  -o jsonpath='{.data.config\.yaml}' | grep -c 'gpu23-2x3g40gb:'

1

kubectl wait fails with no matching resources found until the DaemonSet has created the new pod, so the restart first waits for the pod to exist:

kubectl -n gpu-operator delete pod -l app=nvidia-mig-manager
for i in $(seq 30); do   # up to a minute for the new pod to appear
  kubectl -n gpu-operator get pod -l app=nvidia-mig-manager -o name | grep -q . && break
  sleep 2
done
kubectl -n gpu-operator wait --for=condition=Ready pod -l app=nvidia-mig-manager --timeout=5m

Select the layout and wait for mig-manager. It evicts the GPU workloads on the node, changes the MIG mode, and creates the instances; on the verification host this took about 4 minutes:

kubectl label node <NODE_NAME> nvidia.com/mig.config=gpu23-2x3g40gb --overwrite
sleep 30   # mig-manager marks the new layout pending before it applies it
while :; do
  state=$(kubectl get node <NODE_NAME> -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}')
  echo "mig.config.state=$state"
  case "$state" in success|failed) break ;; esac
  sleep 15
done

Expect mig.config.state=success. On failed, read why, then restart mig-manager, which applies the labeled layout again when it starts:

kubectl -n gpu-operator logs -l app=nvidia-mig-manager --tail=30
kubectl -n gpu-operator delete pod -l app=nvidia-mig-manager
for i in $(seq 30); do   # up to a minute for the new pod to appear
  kubectl -n gpu-operator get pod -l app=nvidia-mig-manager -o name | grep -q . && break
  sleep 2
done
kubectl -n gpu-operator wait --for=condition=Ready pod -l app=nvidia-mig-manager --timeout=5m

Then rerun the block that selects the layout, including its sleep 30. The label reads failed until the new pod updates it, so the loop alone stops at once.

Check:

nvidia-smi --query-gpu=index,mig.mode.current --format=csv,noheader
kubectl get node <NODE_NAME> -o jsonpath='{.status.allocatable}' | tr ',' '\n' | grep nvidia

0, Disabled
1, Disabled
2, Enabled
3, Enabled
"nvidia.com/gpu":"2"
"nvidia.com/mig-3g.40gb":"4"

MIG instances do not survive a reboot, and on a virtual machine stopped and deallocated the GPUs' MIG mode is reset too. mig-manager applies the labeled layout again when it starts after boot, and pods that request a MIG resource stay Pending until it finishes. After each boot, check that nvidia.com/mig.config.state is success; if it is failed, see Troubleshooting.

MIG and tensor parallelism

A deployment that spans several GPUs needs a peer-to-peer path between every pair of GPUs the device plugin can give it. Check the paths between GPUs:

nvidia-smi topo -p2p r

        GPU0    GPU1    GPU2    GPU3
GPU0    X       OK      NS      NS
GPU1    OK      X       NS      NS
GPU2    NS      NS      X       OK
GPU3    NS      NS      OK      X

Only OK is a peer path. On this host the GPUs are bridged in pairs, 0 with 1 and 2 with 3, and have no path between the pairs. A multi-GPU request can land on any whole GPUs, so this node can offer tensor parallelism only if its whole GPUs are exactly one bridged pair. Partition the GPUs outside that pair with MIG, here GPUs 2 and 3, as the step 6 layout does. The owner's detector claims the p2p feature only when every pair of whole GPUs reports OK; see One replica across several devices.

Step 7: Remove taints from GPU nodes

Tenant-mode engine pods carry no tolerations, so they never schedule on a node with a NoSchedule or NoExecute taint. GPU nodes used for tenant-mode inference must carry neither.

Check every node that advertises an NVIDIA resource:

kubectl get nodes -o json | jq -r '.items[]
  | select(.status.allocatable | keys | any(startswith("nvidia.com/")))
  | [.metadata.name, ([.spec.taints[]? | select(.effect == "NoSchedule" or .effect == "NoExecute")
      | .key + ":" + .effect] | join(","))]
  | @tsv'

gpu-node-1

Expect each GPU node's name with nothing after it. For a node listed with a taint, remove it, repeating the key and effect the check printed, e.g., nvidia.com/gpu:NoSchedule:

kubectl taint nodes <NODE_NAME> <TAINT_KEY>:<EFFECT>-

A taint that your cluster needs, such as one that reserves GPU nodes for other workloads, cannot stay on the nodes Kamiwaza serves from.

Step 8: Pin the GPU stack

An unattended upgrade of the NVIDIA libraries while the old kernel module is still loaded breaks the driver until a reboot, and nvidia-smi then fails with Failed to initialize NVML: Driver/library version mismatch. Hold the driver, toolkit, and kernel packages, and upgrade them together in a maintenance window:

PKGS=$(dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' | awk '$1=="ii"{print $2}' \
  | grep -E '^(nvidia-|libnvidia-|linux-modules-nvidia-|linux-objects-nvidia-|linux-signatures-nvidia-|linux-image-|linux-generic)')
sudo apt-mark hold $PKGS

Check:

apt-mark showhold | grep -E '^(nvidia-driver|nvidia-container-toolkit|linux-modules-nvidia)'

Expect the driver, the container toolkit, and the kernel module package. Upgrade with sudo apt-mark unhold $PKGS, upgrade, reboot, and hold again.

Share GPUs with the VRAM plugin (optional)

The VRAM plugin lets several deployments share one GPU by its memory. Install it only if the cluster owner's signed profile has a binding with the vram-plugin allocator; see Tenant-mode inference. It accounts for memory and does not isolate it.

The plugin is a DaemonSet, kamiwaza-vram-plugin, with a ServiceAccount and a ClusterRole. It advertises each GPU it shares as kamiwaza.ai/vram-gb-gpu-<N>, where <N> is the GPU's index and the count is its memory in decimal GB. A cluster administrator installs it once per cluster; the Kamiwaza release does not. Its chart is published with the release from Kamiwaza 1.3.2; 1.3.0 and 1.3.1 publish no chart for it. The chart is for Kubernetes clusters: it ships no OpenShift SecurityContextConstraints.

The plugin runs in its own namespace, kamiwaza-vram-plugin, never in kamiwaza. Its ServiceAccount may get, watch, and patch every Node, and anyone who can create pods, workloads, or ServiceAccount tokens in its namespace can act as it. Give no tenant identity, including Kamiwaza's installer and service accounts, any rights in kamiwaza-vram-plugin. From Kamiwaza 1.3.4 the chart refuses to install into kamiwaza or kamiwaza-extensions.

Create the namespace and label it. The plugin mounts host paths, which needs the privileged Pod Security level, and it serves no mesh traffic:

kubectl create namespace kamiwaza-vram-plugin
kubectl label namespace kamiwaza-vram-plugin --overwrite \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=privileged \
  pod-security.kubernetes.io/warn=privileged \
  istio-injection=disabled

Install it from the release:

helm upgrade --install vram-plugin oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/vram-plugin \
  --version 1.3.4 -n kamiwaza-vram-plugin

The chart pulls its image from the release, ghcr.io/kamiwaza-ai/releases/kamiwaza/images/vram-plugin, without credentials. In an air gap, install it from your registry instead; see Install Without Internet Access.

By default the plugin runs on every node that GPU feature discovery from step 5 labels nvidia.com/gpu.present=true, tolerates the nvidia.com/gpu:NoSchedule taint, sizes each GPU from the nvidia.com/gpu.memory label, and, unless you set gpuIndices, shares every GPU, MIG-enabled ones included. It sees neither MIG slices nor whole-device allocations, so keep two kinds of GPU out of it:

• A MIG-enabled GPU. The plugin advertises the card's full memory, and a deployment it places there finds no CUDA device and fails to start.
• A GPU that something allocates whole. A GPU the plugin shares is still offered as nvidia.com/gpu, and a whole-device binding covers every node of its architecture, so it can take that GPU while accounted deployments use it. Do not publish a whole-device binding for an architecture whose nodes the plugin shares GPUs on, and run nothing else there that requests nvidia.com/gpu.

List the GPUs to share, by their indices as nvidia-smi prints them, in gpuIndices. For example, on a node whose GPU 0 is MIG-partitioned and GPU 1 is whole, --set 'gpuIndices={1}' shares GPU 1, and GPU 0's slices stay available to MIG bindings. The step 6 example node partitions GPUs 2 and 3 and keeps GPUs 0 and 1 for whole-device bindings, so the plugin has no GPU to share there. Every node the plugin runs on must have each listed index; on a node without one, the plugin exits.

A node you kept out still carries nvidia.com/gpu.present, so the default placement would run the plugin there too. Keep it off such a node: label the node kamiwaza.ai/gpu-vendor=none, which the default placement excludes, or set affinity yourself.

The plugin registers with the kubelet only through /var/lib/kubelet/device-plugins, and the chart has no value to move it. On a node whose kubelet root is elsewhere, such as k0s's default /var/lib/k0s/kubelet, its pod does not become ready: it waits on the missing directory, or restarts when it cannot reach the kubelet. On k0s, install the plugin only on nodes whose k0s runs with --kubelet-root-dir=/var/lib/kubelet, as Kamiwaza's own setup tooling installs it.

Install it as release vram-plugin in the kamiwaza-vram-plugin namespace, as above: the same release name and namespace Kamiwaza's setup tooling uses, so the two do not contend for it.

Check:

kubectl -n kamiwaza-vram-plugin rollout status daemonset/kamiwaza-vram-plugin --timeout=5m
kubectl get node <NODE_NAME> -o jsonpath='{.status.allocatable}' | tr ',' '\n' | grep vram-gb

Expect successfully rolled out, then one kamiwaza.ai/vram-gb-gpu-<N> line for each GPU the plugin shares on the node.

A node that refuses GPU requests by environment (step 3) needs the VRAM plugin from Kamiwaza 1.3.4 or later. It passes each share's GPU as a device-list mount as well as by environment. The plugin from Kamiwaza 1.3.3 and earlier passes it by environment only, and on such a node the deployments it places start without a GPU. The share check below shows which of the two a node runs, whether or not the node refuses the environment yet.

The VRAM plugin is supported only with the toolkit installed on the host, as in step 3, with toolkit.enabled: false. With toolkit.enabled: true and the chart's default cdi.enabled: true, the GPU Operator runs the NVIDIA runtime in CDI mode, where the GPU index the plugin passes does not name a CDI device. The deployments the plugin places then fail to start, whether or not the node refuses GPU requests by environment. From Kamiwaza 1.3.4, the chart refuses to install on such a cluster. Its error names the operator's ClusterPolicy and contains vram-plugin: the GPU Operator on this cluster manages the NVIDIA container toolkit in CDI mode. Set allowGpuOperatorCdiToolkit=true only if the nodes the plugin runs on use a host toolkit anyway, such as nodes labeled nvidia.com/gpu.deploy.container-toolkit=false.

Check that a share gets its GPU, and that the plugin passes it as a device-list mount. Set GPU_INDEX to a GPU the plugin shares on GPU_NODE. The pod asks for 1 GB of that GPU's memory, so it stays Pending while the GPU has none free:

GPU_NODE="<NODE_NAME>"
GPU_INDEX=<N>
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: vram-share-check
  namespace: default
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: ${GPU_NODE}
  containers:
  - name: smi
    image: <YOUR_REGISTRY>/library/ubuntu:24.04
    command: ["sh", "-c", "ls /var/run/nvidia-container-devices 2>/dev/null | sed 's/^/device-list mount: /' | grep . || echo 'no device-list mount'; nvidia-smi -L 2>/dev/null || echo 'no GPU'"]
    env:
    - {name: NVIDIA_DRIVER_CAPABILITIES, value: utility}
    resources:
      limits:
        kamiwaza.ai/vram-gb-gpu-${GPU_INDEX}: 1
EOF
kubectl -n default wait --for=jsonpath='{.status.phase}'=Succeeded pod/vram-share-check --timeout=5m
kubectl -n default logs vram-share-check
kubectl -n default delete pod vram-share-check

device-list mount: <N>
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-...)

Expect exactly one device-list mount line, naming GPU_INDEX, and exactly one GPU 0: ... line: the container numbers its one GPU from 0. no device-list mount means the plugin passes the GPU by environment only: upgrade it before the node refuses the environment. On a node that already refuses it, that plugin's share also prints no GPU.

To remove the plugin, run helm uninstall vram-plugin -n kamiwaza-vram-plugin. It leaves its kamiwaza.ai/vram-gb-gpu-<N> node labels behind. Remove them with kubectl label node <NODE_NAME> kamiwaza.ai/vram-gb-gpu-<N>-, once for each GPU it shared.

Move a plugin installed in the kamiwaza namespace

The Kamiwaza 1.3.2 and 1.3.3 instructions installed the plugin in kamiwaza, where Kamiwaza's own identities can act as its ServiceAccount. Move it to kamiwaza-vram-plugin. Until you do, upgrading the old release with the 1.3.4 chart fails with vram-plugin must not be released into tenant namespace "kamiwaza".

Helm cannot move a release, so uninstall it and install it again. Running deployments keep their GPU shares. New deployments that need a share wait until the new pod registers, usually a few seconds.

1. Create and label kamiwaza-vram-plugin, as above.

2. Save the release's values, so the new release gets the same settings:

   helm get values vram-plugin -n kamiwaza -o yaml > vram-plugin-values.yaml

3. If the values name a pull Secret (imagePullSecrets), create it in kamiwaza-vram-plugin now, as Install Without Internet Access shows. Otherwise the new pod cannot pull its image.

4. Get the chart onto this machine before you remove the old release, so nothing in the next two steps needs the network:

   helm pull oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/vram-plugin --version 1.3.4

   Without internet access, use the vram-plugin-1.3.4.tgz chart from your mirror instead, as Install Without Internet Access describes.

5. Uninstall it, and wait until its pod is gone. The old pod removes the plugin's kubelet sockets as it stops, so the new one must not start first:

   helm uninstall vram-plugin -n kamiwaza
   kubectl -n kamiwaza wait --for=delete pod -l app.kubernetes.io/name=vram-plugin --timeout=2m

6. Install it in its own namespace from the chart you saved, then run the check above:

   helm upgrade --install vram-plugin ./vram-plugin-1.3.4.tgz \
     -n kamiwaza-vram-plugin -f vram-plugin-values.yaml

Next step

Give the cluster owner the node facts the signed profile attests. They run the detector from the Kamiwaza core image on each GPU node; see Tenant-mode inference.

Troubleshooting

The VRAM plugin advertises a GPU's memory wrongly, or its node has no kamiwaza.ai/vram-gb-gpu-<N> label. The plugin reads and labels its node through the Kubernetes API only when it starts. If it could not reach the API server then, it falls back to sizing the GPU from host memory and publishes no label. Check that no NetworkPolicy in kamiwaza-vram-plugin blocks its egress to the API server, then restart it with kubectl -n kamiwaza-vram-plugin rollout restart daemonset/kamiwaza-vram-plugin.

GPU Operator pods fail with RunContainerError: context deadline exceeded, and the operator never reports ready. Persistence mode is off; see step 2. Time nvidia-smi -L on the node: several seconds means persistence mode is off.

GPU pods stay Pending with Insufficient nvidia.com/gpu. Compare the request with the node's allocatable resources, from step 5. No NVIDIA resources at all means the device plugin is not running on the node, as expected on a node you kept out; otherwise check kubectl -n gpu-operator get pods -o wide. On k0s, a device plugin that runs but registers nothing means hostPaths.kubeletRootDir does not match the kubelet root: /var/lib/k0s/kubelet on a default k0s, and the GPU Operator's default, /var/lib/kubelet, on a k0s started with --kubelet-root-dir=/var/lib/kubelet.

MIG resources read 0 after a reboot, and nvidia.com/mig.config.state is failed. mig-manager could not apply the layout when the node started. Read its log:

kubectl get node <NODE_NAME> -o jsonpath='{.metadata.labels.nvidia\.com/mig\.config\.state}'; echo
kubectl get node <NODE_NAME> -o jsonpath='{.status.allocatable}' | tr ',' '\n' | grep mig
kubectl -n gpu-operator logs -l app=nvidia-mig-manager --tail=50 | grep -i 'fail\|error'

failed to stop service nvidia-persistenced.service: timeout stopping service means stopping the persistence daemon, which mig-manager does before a MIG change, took longer than mig-manager's 30-second limit. On the verification host it took 32 seconds while the node was still starting. Once the node is up, restart mig-manager, which applies the layout again:

kubectl -n gpu-operator delete pod -l app=nvidia-mig-manager
for i in $(seq 30); do   # up to a minute for the new pod to appear
  kubectl -n gpu-operator get pod -l app=nvidia-mig-manager -o name | grep -q . && break
  sleep 2
done
kubectl -n gpu-operator wait --for=condition=Ready pod -l app=nvidia-mig-manager --timeout=5m

Then wait for success with the step 6 block that selects the layout, including its sleep 30: the label reads failed until the new pod updates it. selected mig-config not present means the label names a layout mig-manager has not loaded; see step 6.

GPU pods stay Pending with had untolerated taint. A GPU node is tainted; see step 7.

After the node refuses GPU requests by environment, a GPU deployment starts without a GPU. The engine log reports no CUDA device, or libcuda.so.1: cannot open shared object file. Either the pod was given its GPU before the change, so stop the deployment and deploy it again; or the device plugin still passes GPUs by environment (check it as in step 5); or the VRAM plugin is from Kamiwaza 1.3.3 or earlier (its share check prints no device-list mount). See Refuse GPU requests by environment.

Installing the VRAM plugin fails with an error such as vram-plugin: image.tag "v0.2.6" is older than v0.2.7. Your values pin the plugin's image.tag to a plugin version that passes each share's GPU by environment only. Remove image.tag from the values, so the chart runs its own release's image, and install again.

GPU pods fail with no runtime for "nvidia" is configured. The node's containerd has no nvidia runtime; check runtimeHandlers as in step 3. On k0s, check the drop-in's version = 3 line and restart k0s.

nvidia-smi prints Driver/library version mismatch. The libraries were upgraded while the old kernel module is loaded. Reboot the node, then pin the stack as in step 8.

An engine exits at start with CUDA driver version is insufficient for CUDA runtime version. The host driver is older than the engine image's CUDA major. Upgrade the driver to the floor in Supported GPUs and driver floors, or have the owner attest the driver's CUDA major, so the platform picks an image the driver can run; see Attest the NVIDIA driver's CUDA major.

nvidia-smi finds no GPU with Secure Boot on. The kernel refused an unsigned module. Check dmesg | grep -i 'module verification failed', then use the prebuilt, signed modules from step 1 or enroll a MOK as in Secure Boot.