Tenant-mode inference

Support scope

This page documents tenant-mode inference for Kamiwaza 1.3.4. Support covers the configurations marked Supported in Compatibility scope, qualified on k0s Kubernetes 1.36.

Tenant-mode inference is how a namespaced Kamiwaza install serves local models on GPUs and CPUs. Kamiwaza installs into one namespace and cannot read the cluster's Nodes, so it cannot discover GPUs itself. The cluster owner describes the hardware instead, in a profile that the owner signs, and Kamiwaza places each deployment on hardware the signed profile vouches for.

A default install has tenant-mode inference off and cannot use GPUs: a GPU deployment is refused with HTTP 422. To serve models on GPUs:

1. Prepare the GPU nodes; see GPU Nodes.
2. Create the owner key, detect each GPU host's facts, and sign a profile, with the owner tool in the Kamiwaza core image. See Turn on tenant-mode inference.
3. Point the install at the signed profile, in four Helm values.

Compatibility scope

Live qualification for this release targets k0s Kubernetes 1.36 only. Chart, API, and client contract checks cover Kubernetes 1.28 through 1.36, but passing a contract check is not a live support claim for that minor version: only 1.36 has been exercised end to end. The allocation classes are:

Class	Engines	Kubernetes resource	Isolation statement	Release status
CPU	llama.cpp, whisper.cpp	cpu and memory requests	Kubernetes scheduler allocation	Supported on k0s 1.36
Whole NVIDIA GPU	llama.cpp, vLLM, diffusion (CUDA)	nvidia.com/gpu	Whole-device allocation	Supported on k0s 1.36, one device per replica only. A deployment that spans more than one device is implemented but not qualified live
Whole AMD GPU	None in this release; no ROCm engine image ships	amd.com/gpu	Whole-device allocation	Not qualified in this release. Do not deploy against a support claim
NVIDIA MIG	llama.cpp, vLLM, diffusion (CUDA)	Qualified nvidia.com/mig-*	Hardware partition, exact shape only	Supported on k0s 1.36, recorded slice shape only
VRAM-plugin sharing	llama.cpp, vLLM, diffusion (CUDA)	Owner-advertised kamiwaza.ai/vram-gb-*	Accounted sharing, not hard isolation	Supported on k0s 1.36, recorded plugin and profile only. Optional: the cluster owner installs the VRAM plugin

Engines

Tenant mode serves only the engines in the platform engine catalog that ships inside the release's core image:

Engine	Serves	Placement	Images
llama.cpp	Text generation; embeddings when the model configuration sets embedding and states its pooling	GPU or CPU	NVIDIA CUDA 13 and CUDA 12, amd64 and arm64; CPU amd64 and arm64
vLLM	Text generation	GPU only	NVIDIA CUDA 13, amd64 and arm64 (GB10 only)
Diffusion	Image generation	GPU only, one device per replica	NVIDIA CUDA 12 (built with 12.8), amd64 only
whisper.cpp	Transcription	CPU only	CPU amd64 and arm64

The engine catalog has no arm64 image for diffusion in this release. A binding on an arm64 node therefore cannot serve diffusion, and the platform refuses the deployment with no diffusion image is published for nvidia/arm64.

vLLM's arm64 image serves one card, the GB10 (DGX Spark, compute capability 12.1). The platform refuses vLLM on any other arm64 NVIDIA card, such as GH200, GB200, Jetson Thor, or a discrete Blackwell card in an arm64 server, with a reason such as vllm requires compute capability 12.1; this binding declares 10.0.

A GPU binding serves an engine only when the engine publishes an image for the binding's vendor and architecture, and the hardware its accelerator block attests meets the engine's requirements. Each NVIDIA image is built against one CUDA major and carries kernels down to a lowest compute capability, so the card and the host driver both limit which image can run:

Engine	NVIDIA image	Architectures	Lowest compute capability	Host driver
llama.cpp	CUDA 13	amd64, arm64	7.5 (Turing)	580.65.06 (R580) or later
llama.cpp	CUDA 12	amd64, arm64	7.0 (Volta)	525.60.13 or later; R570 or later for Blackwell GPUs
vLLM	CUDA 13	amd64	7.5, and the engine requires bf16 (8.0 or later)	580.65.06 (R580) or later
vLLM	CUDA 13	arm64	12.1 (GB10 only)	580.65.06 (R580) or later
Diffusion	CUDA 12 (built with 12.8)	amd64	8.0, and the engine requires bf16	R570 or later recommended, not enforced

llama.cpp therefore requires compute capability 7.0 or later: a Volta card runs the CUDA 12 image, and no image serves Pascal or older. vLLM requires a CUDA 13 driver. For how the platform picks between the llama.cpp images, see Attest the NVIDIA driver's CUDA major.

The platform compares only the driver's CUDA major, so it does not refuse the diffusion image on an older CUDA 12 driver. The image is built with PyTorch for CUDA 12.8. Unlike the llama.cpp CUDA 12 image, which runs on older CUDA 12 drivers under NVIDIA's minor-version compatibility, the diffusion image is not qualified below R570, so run diffusion on R570 or later.

A diffusion deployment serves one diffusers pipeline on one device through /v1/images/generations. The model must be a diffusers repository with model_index.json at its top; a single-file checkpoint is refused. A pipeline has no context length, so the platform sizes the device from the pipeline's weights. Tenant mode refuses a diffusion deployment that configures LoRA or Lightning adapters, a stream profile, trust_remote_code, DFloat11 settings, or more than one device (tensor_parallel_size or gpu_count above 1). When a configuration key causes the refusal, the reason names that key. A DFloat11 model that the platform recognizes by its name is also refused; that reason says DFloat11 models need more than one device and names no key. A stream profile that the platform detects from the model's name is kept, not refused.

When the release sets core.inferenceResources.enabled: true, every local deployment takes the tenant path, including a request without an inferenceResources block, which the platform derives. There is no fallback to the standard deployment path. An engine that the engine catalog does not list is refused. MLX is not in the catalog, so MLX is not supported in tenant mode in this release. A deployment without a block that selects such an engine receives HTTP 422, and the detail ends with the engine and the reason:

cannot derive inference resources for this deployment: engine 'mlx' is not qualified in this release

A request that carries its own block is accepted and then fails at dispatch with the same reason.

External endpoints are not refused. An external endpoint proxies a remote service and allocates no local resources, so on a tenant cluster the platform does not size it and creates no workload in the tenant namespace. On a tenant cluster, two external-endpoint requests receive HTTP 422:

• A request that supplies an inferenceResources block:

  external endpoints allocate no local resources; omit inferenceResources

• A request that sets a nonzero gpu_allocation or vram_allocation. A cluster without tenant resources still honors these overrides:

  external endpoints allocate no local resources; omit gpu_allocation and vram_allocation

One replica across several devices

A deployment can span up to eight whole devices in a single Pod through tensor parallelism. This is implemented but not qualified live in this release; see Compatibility scope. The device count comes from tensor_parallel_size in the model configuration, or from a signed recipe when one pins it. An absent value is one device — never the node's device count, so a first deployment does not take a whole multi-GPU node.

Tensor parallelism needs a peer-to-peer path between the devices, so it is admitted only on a whole-device binding whose accelerator block attests the p2p feature. The owner tool's detector claims p2p only when every pair of whole-mode GPUs on the host reports a peer path, because the device plugin can give a multi-GPU request any whole GPUs. MIG-mode GPUs are left out: a whole-device binding is never given one. On a host whose GPUs are bridged in pairs, partition the GPUs outside one bridged pair with MIG, so its whole GPUs are exactly that pair; see MIG and tensor parallelism.

A MIG partition does not have a peer path, and an accounted VRAM share is a slice of a device the workload does not exclusively hold, so both refuse a count above one rather than quietly serving a single device to an engine told to shard across several. A count that is not a whole number, is below one, or exceeds eight is refused for the same reason: the request and the Pod must agree.

Pipeline and data parallelism are not part of this release.

Ownership boundary

The cluster owner supplies drivers, device plugins, optional VRAM sharing, RuntimeClasses, admission policy, the signed profile, and the kamiwaza-owner-accelerators quota that caps the accelerators any pod in the namespace can request (see Cap the namespace's accelerators). Every GPU binding in a profile attests its hardware in an accelerator block, which the owner tool detects on the host; a CPU binding needs only its architecture. Recipe catalogs are optional, GPU and CPU alike: without one, the platform derives each deployment, and a published recipe pins a specific model and configuration. The tenant supplies only namespaced Helm and serving inputs.

Kubernetes namespace-admin permissions do not grant the Kamiwaza admin role. Model deployment through the API, SDK, or UI remains Kamiwaza-admin-only.

A deployment request needs no inferenceResources block: the platform derives one. For a GPU deployment it sizes memory from the model's own metadata and the context the model configuration sets (max_model_len), or else the context the model declares, and places the request on the smallest binding that holds it. A deployment goes to CPU instead when it sets force_cpu, when its engine ships only a CPU build (whisper.cpp), or when the owner declared no GPU hardware; it is sized from the platform's CPU floor and the model's weights, and runs the platform's CPU build for the binding's architecture. A request that cannot be served as asked -- no context declared anywhere, or an embedding configuration that does not state its pooling -- is refused with HTTP 422 and the reason; one the platform cannot evaluate, such as an unreadable profile bundle, with HTTP 503.

The tenant chart creates no cluster-scoped GPU resource and does not require nodes/get, nodes/list, or nodes/watch. A profile or allocator failure is reported as a failure; the platform does not silently fall back to CPU or a weaker isolation class.

Turn on tenant-mode inference

Before you start

You need	Notes
Prepared GPU nodes	Every requirement on GPU Nodes passes. Skip for CPU-only serving.
The release's core image, by digest	The owner tool ships in it. Sign with the core image of the release you install, so the release that verifies the profile is the one that signed it.
Docker or Podman	On the machine where you sign, not on the GPU nodes: step 3 runs the detector as a pod. The owner tool runs in the core image; it contacts no cluster and no network.
kubectl and jq	On the machine where you publish the profile.

Do not install Docker on a Kubernetes node

Docker sets the host's iptables FORWARD policy to DROP, which breaks pod networking on the node. Run the owner tool on a separate machine, and the detector as a pod.

Read the core image from a running install:

kubectl -n kamiwaza get deploy core-scheduler \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="core")].image}'; echo

<YOUR_REGISTRY>/kamiwaza-ai/releases/kamiwaza/images/core@sha256:<DIGEST>

Before the install, take the same reference from the release's image list; see Install Without Internet Access.

If your registry requires credentials, log the container engine in to it first: docker login <REGISTRY_HOST>, or podman login <REGISTRY_HOST>. Podman also reads an existing credentials file named with --authfile <FILE> or the REGISTRY_AUTH_FILE environment variable, in the same format as the install's pull Secret.

Set the image, and define a shell function that runs the owner tool with the current directory mounted. With Docker:

export CORE_IMAGE="<YOUR_REGISTRY>/kamiwaza-ai/releases/kamiwaza/images/core@sha256:<DIGEST>"
owner_tool() {
  docker run --rm -i --user "$(id -u):$(id -g)" -v "$PWD:/work" -w /work \
    "$CORE_IMAGE" python -m kamiwaza.serving.inference_resources.owner_tool "$@"
}

With rootless Podman, which needs --userns=keep-id to write files you own, and :Z on the mount to use the directory on a host with SELinux, such as RHEL or Fedora:

export CORE_IMAGE="<YOUR_REGISTRY>/kamiwaza-ai/releases/kamiwaza/images/core@sha256:<DIGEST>"
owner_tool() {
  podman run --rm -i --userns=keep-id --user "$(id -u):$(id -g)" -v "$PWD:/work:Z" -w /work \
    "$CORE_IMAGE" python -m kamiwaza.serving.inference_resources.owner_tool "$@"
}

:Z relabels the current directory for the container, and Podman ignores it on a host without SELinux. Podman refuses to relabel your home directory, so run the function from a directory of its own.

The steps below use the function; the GPU detector in step 3 runs on the GPU node instead, because it must see the GPUs.

Step 1: Create the owner signing key

owner_tool keygen --private-key-out owner-ed25519-private.pem

{
  "keyId": "sha256:6e1f...",
  "publicKey": "8Ljv..."
}

The tool writes an unencrypted Ed25519 private key readable only by you, and refuses to replace an existing file. It prints the public key, which goes into the Helm values as ownerPublicKey, and the keyId your profile names. Keep the private key out of source control, out of the tenant namespace, and out of any evidence you record. To print the public half again:

owner_tool pubkey --private-key owner-ed25519-private.pem

Step 2: Choose a cluster binding ID

Pick a clusterBindingId for this cluster: 1 to 128 characters, starting with a letter or digit, then letters, digits, ., _, :, or -. The profile and the Helm values must carry the same value; Kamiwaza refuses a profile signed for another binding.

Step 3: Detect each GPU host's facts

Run the detector on every GPU node, with all of the node's GPUs visible to it. Run it as a one-shot privileged pod on the nvidia RuntimeClass: the node then needs no container engine of its own. It must be privileged because GPU nodes give an unprivileged pod only the GPUs a device plugin allocated to it (see GPU Nodes). So it runs in a namespace of its own, never the Kamiwaza namespace, where no pod may be privileged.

Create that namespace once, and copy the install's pull Secret into it so the pod can pull the core image. If your install has no pull Secret, skip the copy and delete the two imagePullSecrets lines below:

kubectl create namespace kamiwaza-owner-detect
kubectl label namespace kamiwaza-owner-detect --overwrite \
  pod-security.kubernetes.io/enforce=privileged
kubectl -n kamiwaza get secret kamiwaza-pull -o json \
  | jq '{apiVersion, kind, type, data, metadata: {name: .metadata.name}}' \
  | kubectl -n kamiwaza-owner-detect apply -f -

Then, for each GPU node, set NODE_NAME to the node and run:

NODE_NAME="<NODE_NAME>"
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: owner-tool-detect
  namespace: kamiwaza-owner-detect
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  runtimeClassName: nvidia
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: $NODE_NAME
  imagePullSecrets:
  - name: kamiwaza-pull
  containers:
  - name: detect
    image: $CORE_IMAGE
    command: ["python", "-m", "kamiwaza.serving.inference_resources.owner_tool", "detect"]
    securityContext:
      privileged: true
    env:
    - {name: NVIDIA_VISIBLE_DEVICES, value: all}
    - {name: NVIDIA_DRIVER_CAPABILITIES, value: "compute,utility"}
EOF
kubectl -n kamiwaza-owner-detect wait --for=jsonpath='{.status.phase}'=Succeeded pod/owner-tool-detect --timeout=5m
kubectl -n kamiwaza-owner-detect logs owner-tool-detect
kubectl -n kamiwaza-owner-detect delete pod owner-tool-detect

When every node is done, delete the namespace, which also deletes its copy of the pull Secret:

kubectl delete namespace kamiwaza-owner-detect

The detector prints the accelerator block for the node's GPUs:

{
  "accelerator": {
    "computeCapability": "8.0",
    "cudaDriverMajor": 13,
    "features": [
      "bf16",
      "mig",
      "nvlink",
      "p2p"
    ],
    "vendor": "nvidia"
  }
}

A GPU node that already has a container engine can run the detector directly instead, privileged for the same reason as the pod. With Podman, as root, through the NVIDIA runtime; add --authfile <FILE> if your registry requires credentials:

sudo podman run --rm --privileged --runtime /usr/bin/nvidia-container-runtime \
  -e NVIDIA_VISIBLE_DEVICES=all "$CORE_IMAGE" \
  python -m kamiwaza.serving.inference_resources.owner_tool detect

podman run --device nvidia.com/gpu=all does not work with Podman 4.9, the version Ubuntu 24.04 ships: it cannot read the device description that current NVIDIA container toolkits write, and fails with unresolvable CDI devices nvidia.com/gpu=all. On a node that already runs Docker with the NVIDIA runtime, docker run --rm --privileged --gpus all "$CORE_IMAGE" python -m kamiwaza.serving.inference_resources.owner_tool detect also works. Do not install Docker on a Kubernetes node to run it.

The detector reads the facts from nvidia-smi instead of expecting you to know them:

• computeCapability is the GPUs' compute capability. The detector refuses a node with two kinds of GPU: a binding cannot single out one kind on such a node, so keep it out of tenant-mode GPU serving; see GPU Nodes.
• cudaDriverMajor is the major of the CUDA Version in nvidia-smi's header. The detector exits with an error rather than guess when it cannot read it.
• bf16 and fp8 follow from the compute capability, and nvlink from the links the GPUs report.
• p2p appears only when every pair of whole-mode GPUs has a peer path; see One replica across several devices.
• mig appears when the GPUs can be partitioned, whether or not a GPU is in MIG mode now. It is informational: no engine requires it, and a MIG binding does not need it.

Copy the block into the binding for those nodes. A binding covers every node of its architecture that advertises its resource (see nodeSelector in step 4), so when those nodes differ, attest only what holds on all of them: the lowest compute capability and cudaDriverMajor, and only the features every node has. Drop p2p and nvlink from a MIG or VRAM-share binding: a slice has neither.

Step 4: Write the profile envelope

The envelope lists the profiles the cluster offers. Each profile has one guarantee and one or more bindings; each binding names the Kubernetes resource a deployment requests and the nodes it runs on. This envelope offers CPU serving on amd64 nodes and whole NVIDIA GPUs. Save it as owner-profile-envelope.json:

{
  "schemaVersion": 1,
  "clusterBindingId": "<CLUSTER_BINDING_ID>",
  "namespace": "kamiwaza",
  "version": 1,
  "keyId": "<KEY_ID>",
  "issuedAt": "<ISSUED_AT>",
  "expiresAt": "<EXPIRES_AT>",
  "evidenceId": "gpu-nodes-checked-2026-10-01",
  "profiles": [
    {
      "name": "cpu-standard",
      "capability": "cpu",
      "guarantees": [],
      "compatibleVariantIds": ["cpu-default"],
      "bindings": [{
        "id": "cpu-amd64",
        "allocator": "cpu",
        "architecture": "amd64",
        "runtime": {"kind": "cpu", "variantId": "cpu-default", "image": "<LLAMACPP_CPU_IMAGE>"},
        "capacity": {"kind": "scheduler"},
        "nodeSelector": {"kubernetes.io/arch": "amd64"}
      }]
    },
    {
      "name": "nvidia-whole",
      "capability": "gpu",
      "guarantees": ["whole-device"],
      "compatibleVariantIds": ["llamacpp-nvidia-amd64", "vllm-nvidia-amd64", "diffusion-nvidia-amd64"],
      "bindings": [{
        "id": "nvidia-whole-amd64",
        "allocator": "extended-resource",
        "architecture": "amd64",
        "accelerator": {
          "vendor": "nvidia",
          "computeCapability": "8.0",
          "cudaDriverMajor": 13,
          "features": ["bf16", "mig", "nvlink", "p2p"]
        },
        "capacity": {"kind": "scheduler", "usableMemory": "79Gi"},
        "resource": {"name": "nvidia.com/gpu", "quantity": 1},
        "runtime": {"kind": "cuda", "variantId": "llamacpp-nvidia-amd64", "image": "<LLAMACPP_CUDA_IMAGE>"},
        "runtimeClassName": "nvidia",
        "nodeSelector": {"kubernetes.io/arch": "amd64"}
      }]
    }
  ]
}

Fill in the placeholders:

Field	Value
clusterBindingId	The ID from step 2.
namespace	The namespace Kamiwaza is installed in.
version	An integer you raise with every new revision. Kamiwaza refuses a revision below core.inferenceResources.minimumBundleVersion (default 1).
keyId	The keyId from step 1.
issuedAt, expiresAt	UTC timestamps with a Z suffix, such as 2026-10-01T00:00:00Z. Kamiwaza refuses the profile outside that window, so publish a new revision before expiresAt.
evidenceId	A label for the checks behind this revision, for your records.
accelerator	The block from step 3.
nodeSelector	kubernetes.io/arch only; this release accepts no other key. A binding therefore covers every node of that architecture that advertises its resource, so its accelerator facts and usableMemory must hold on all of them.
usableMemory	The GPU memory a deployment may plan on, less what the driver and CUDA context hold back. Write a whole number of Gi, Mi or Ki; the profile schema refuses a fraction such as 121.7Gi. The right value depends on the binding; see the list below this table.
runtime.image	A digest-pinned engine image. Derived deployments take their image from the engine catalog; this value is used only for a deployment that a recipe pins, which runs it as written. In a mirrored or air-gapped install, name your mirror's copy. Print the catalog's images with the command below.

Set usableMemory for each kind of binding as follows:

• Whole device or MIG partition on a discrete card: the device's or the slice's memory less the driver's and CUDA context's share, 79Gi on an 80 GB card. Understating it is the safe direction.
• VRAM plugin: state it accurately. vLLM reserves a fraction of the whole card: 90%, the engine catalog default, times the request divided by usableMemory. Understating usableMemory therefore raises what vLLM reserves, and the reservation passes the grant once usableMemory is about a tenth below the card's memory.
• Whole-device GB10: its GPU memory is the host's, so set usableMemory to the device total rounded down, 121Gi on a DGX Spark, whose total is about 121.69 GiB. vLLM's budget is the request divided by usableMemory, rounded up to a hundredth, applied to that total. At 121Gi vLLM already reserves slightly more than the request: 121.69/121 of it, plus up to 0.01 of the total from the rounding. A smaller value reserves further above the request. The host's own processes use the same memory, so a request must also fit in what the host leaves free. A VRAM-plugin binding on a GB10 follows the VRAM-plugin rule instead.

A GPU profile's compatibleVariantIds must list <engine>-nvidia-<architecture> for each engine it may serve, and each binding's runtime.variantId must be in its profile's list. To print the engine images the core image carries:

docker run --rm "$CORE_IMAGE" python -c \
  "from importlib.resources import files; print(files('kamiwaza.serving.inference_resources').joinpath('engine-catalog.json').read_text())" \
  | jq -r '.engines[] | .engine as $e | .images[] | [$e, .vendor, .architecture, (.cuda // "-" | tostring), .image] | @tsv'

llamacpp	nvidia	amd64	13	ghcr.io/kamiwaza-internal/containers/images/llamacpp-cuda13@sha256:a482...
llamacpp	cpu	amd64	-	ghcr.io/kamiwaza-internal/containers/images/llamacpp-cpu@sha256:ccbd...

For MIG, add a profile whose guarantee is mig and whose binding requests the partition the node advertises. This profile offers the 3g.40gb slices from GPU Nodes; add it to the profiles list:

{
  "name": "nvidia-mig-3g40gb",
  "capability": "gpu",
  "guarantees": ["mig"],
  "compatibleVariantIds": ["llamacpp-nvidia-amd64", "vllm-nvidia-amd64", "diffusion-nvidia-amd64"],
  "bindings": [{
    "id": "nvidia-mig-3g40gb-amd64",
    "allocator": "extended-resource",
    "architecture": "amd64",
    "accelerator": {
      "vendor": "nvidia",
      "computeCapability": "8.0",
      "cudaDriverMajor": 13,
      "features": ["bf16", "mig"]
    },
    "capacity": {"kind": "scheduler", "usableMemory": "38Gi"},
    "resource": {"name": "nvidia.com/mig-3g.40gb", "quantity": 1},
    "runtime": {"kind": "cuda", "variantId": "llamacpp-nvidia-amd64", "image": "<LLAMACPP_CUDA_IMAGE>"},
    "runtimeClassName": "nvidia",
    "nodeSelector": {"kubernetes.io/arch": "amd64"}
  }]
}

The accelerator block is the node's, from step 3, without p2p and nvlink: a slice has neither. Set usableMemory to the slice's memory less headroom for the CUDA context. On the verification host that GPU Nodes describes, a 3g.40gb slice of an A100 80GB reports 40192 MiB in nvidia-smi, so the example uses 38Gi. Binding id values must be unique across the whole envelope.

An accounted-share profile, for the VRAM plugin, uses the vram-plugin allocator and {"kind": "accounted", ...} capacity, and reserves a bookkeeping bucket; it does not provide hard VRAM or compute isolation.

Step 5: Sign and publish the profile

owner_tool sign --envelope owner-profile-envelope.json \
  --private-key owner-ed25519-private.pem > owner-profile-configmap.json

The tool runs Kamiwaza's own verifier over what it signs, so it refuses a profile this release would reject, and names the check that failed:

error: signed bundle would be refused: invalid inference profile authorization: bundle is expired or its window is empty

It emits an immutable ConfigMap named kamiwaza-inference-profiles. Give each revision its own name, then apply it:

export PROFILE_CONFIGMAP=kamiwaza-inference-profiles-r1
jq --arg name "$PROFILE_CONFIGMAP" '.metadata.name = $name' owner-profile-configmap.json \
  > "$PROFILE_CONFIGMAP.json"
kubectl apply --server-side --field-manager=owner-inference-publisher -f "$PROFILE_CONFIGMAP.json"

Check:

kubectl -n kamiwaza get configmap "$PROFILE_CONFIGMAP" -o jsonpath='{.immutable} {.data.bundle\.sha256}'; echo

true 9c1d...

Anchor the profile on the namespace

As the cluster owner, anchor the profile you published with one annotation on the kamiwaza namespace. Kamiwaza then trusts only the key, cluster binding, and profile ConfigMap the annotation names, and refuses a profile whose version is lower than the anchored one, whatever the Helm values say. The namespace administrator can read the annotation but cannot change or remove it, so a superseded profile cannot be put back.

The owner tool writes the annotation as a patch file from the published ConfigMap. It sets the minimum version to this profile's version, and checks that Kamiwaza would accept the profile under it. It needs no private key. Run the block as one command, so the patch is applied only when the tool succeeds:

owner_tool anchor --configmap "$PROFILE_CONFIGMAP.json" \
  --public-key <OWNER_PUBLIC_KEY> > owner-anchor-patch.json \
  && kubectl patch namespace kamiwaza --type merge --patch-file owner-anchor-patch.json

If the tool refuses the profile, nothing is applied and the namespace keeps its current anchor. kubectl patch also refuses an empty or missing patch file. Do not set the annotation from the patch file any other way, such as with kubectl annotate: if the tool failed, that writes an empty annotation, which is malformed and refuses every deployment.

Check:

kubectl get namespace kamiwaza \
  -o jsonpath='{.metadata.annotations.inference\.kamiwaza\.ai/owner-anchor}'; echo

{"bundleConfigMap":"kamiwaza-inference-profiles-r1","clusterBindingId":"...","minimumBundleVersion":1,"publicKeys":["8Ljv..."],"schemaVersion":1}

The anchor is optional in this release and is planned to be required from 1.4. Without it, Kamiwaza takes its trust from the Helm values in step 6 alone and logs UNANCHORED OWNER TRUST, and anyone who can change those values can point Kamiwaza at an older profile or another key. A malformed anchor refuses every deployment rather than falling back to the Helm values.

Cap the namespace's accelerators

Any identity that can create pods in the kamiwaza namespace, including the namespace administrator and Kamiwaza's own service accounts, can request any accelerator a node advertises, outside the profile: a whole GPU the profile shares only through VRAM accounting, a MIG slice it never granted, or all of a VRAM bucket. As the cluster owner, cap that with one ResourceQuota, kamiwaza-owner-accelerators. Kubernetes enforces it and Kamiwaza never reads it. The namespace administrator can read it but cannot change or remove it.

The quota is optional in this release, like the anchor, and is planned to be required from 1.4.

The owner tool renders the quota from the published profile and a snapshot of your nodes. It checks the profile as owner_tool anchor does, and contacts no cluster:

• Each accelerator a GPU binding names is set to what the nodes the binding selects advertise. A VRAM bucket counts at most its totalUnits from each node that advertises it, and the quota is the sum: it caps the namespace's total across those nodes, not each node. With a single node, the quota is totalUnits.
• Every other NVIDIA, AMD, Intel or VRAM-bucket resource is set to 0.
• Every key is requests.<resource>, the only form a quota counts for these resources. Kubernetes accepts limits. and bare keys but counts nothing against them, so do not write the quota by hand.

As the cluster owner, with cluster-admin credentials, take the node snapshot, then render and apply the quota as one command, so nothing is applied when the tool refuses the profile:

kubectl get nodes -o json > nodes.json
owner_tool quota --configmap "$PROFILE_CONFIGMAP.json" \
  --public-key <OWNER_PUBLIC_KEY> --allocatable nodes.json > owner-quota.json \
  && kubectl apply --server-side --field-manager=owner-inference-publisher -f owner-quota.json

--limit <RESOURCE>=<COUNT> lowers the count of a resource the profile grants. It cannot grant a resource the profile does not, or raise a VRAM bucket above its totalUnits.

If the kamiwaza-extensions namespace exists and no extension needs an accelerator, cap it at zero. The quota cannot grant an accelerator the profile does not, and that namespace has no profile, so a zero quota refuses every accelerator an extension asks for. First check that no pod there requests one; the command prints nothing when none does:

kubectl -n kamiwaza-extensions get pods -o json | jq -r '.items[]
  | select([.spec.containers[], (.spec.initContainers // [])[]
      | .resources.requests // {} | keys[]
      | select(test("^(nvidia\\.com|amd\\.com|gpu\\.intel\\.com)/|^kamiwaza\\.ai/vram-"))]
    | length > 0)
  | .metadata.name'

Then render and apply it. Without --configmap, every accelerator is 0:

owner_tool quota --allocatable nodes.json --namespace kamiwaza-extensions \
  > extensions-quota.json \
  && kubectl apply --server-side --field-manager=owner-inference-publisher -f extensions-quota.json

Check: the quota lists its limits and current use, and the namespace administrator is refused a pod that asks for an accelerator the profile does not grant:

kubectl -n kamiwaza get resourcequota kamiwaza-owner-accelerators \
  -o jsonpath='{.status.hard}{"\n"}{.status.used}{"\n"}'
kubectl -n kamiwaza run quota-probe --image=busybox:1.37 --restart=Never --dry-run=server \
  --annotations=sidecar.istio.io/inject=false \
  --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"quota-probe","image":"busybox:1.37","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}},"resources":{"requests":{"amd.com/gpu":"1"},"limits":{"amd.com/gpu":"1"}}}]}}' \
  --as=system:serviceaccount:kamiwaza:kamiwaza-installer \
  --as-group=system:serviceaccounts --as-group=system:serviceaccounts:kamiwaza \
  --as-group=system:authenticated

{"requests.amd.com/gpu":"0","requests.nvidia.com/gpu":"2",...}
{"requests.amd.com/gpu":"0","requests.nvidia.com/gpu":"0",...}
Error from server (Forbidden): pods "quota-probe" is forbidden: exceeded quota: kamiwaza-owner-accelerators, requested: requests.amd.com/gpu=1, used: requests.amd.com/gpu=0, limited: requests.amd.com/gpu=0

The probe pod meets the restricted Pod Security standard, so it reaches the quota check even in a namespace that enforces that standard. Use above a limit means pods admitted before the quota still hold those accelerators. They keep running; find out why.

Once the quota is in place:

• A deployment over the quota is refused when its pod is created, not left pending, and stays pending with the reason exceeded quota: kamiwaza-owner-accelerators. Kubernetes retries it with a backoff that doubles, about 3, 5.5, 11 and 22 minutes after the first refusal, and nothing retries it when capacity frees. So it can start some minutes after capacity frees, not at once: about 5.5 minutes after the first refusal in our test. If capacity frees after about 22 minutes, the held deployment gets no further attempt before the 30-minute scheduling deadline and fails; deploy it again. On full hardware, stop one deployment before you start the next.
• Kamiwaza checks each VRAM share on its own, while the quota caps the bucket's total, so a share that fits alone can wait with the quota's reason.
• Running pods are never evicted, and a pod that requests no accelerator is never counted. Any quota your platform sets still applies.

Render and apply the quota again whenever you publish a profile, change GPU hardware, MIG layout, the device plugin or the VRAM plugin, or recreate the namespace. A resource that appears after the snapshot, such as a new MIG profile, is not limited until you do. If the signed file is no longer at hand, read the published profile back and take a fresh snapshot, then run the render and apply command above again:

kubectl -n kamiwaza get configmap "$PROFILE_CONFIGMAP" -o json > "$PROFILE_CONFIGMAP.json"
kubectl get nodes -o json > nodes.json

The quota does not count a pod that selects GPUs through NVIDIA_VISIBLE_DEVICES without requesting them; GPU nodes must refuse such requests from unprivileged containers (see GPU Nodes). It does not count Dynamic Resource Allocation claims either. When more than one node advertises the same VRAM bucket, one pod can take more than its totalUnits of one node's bucket, as long as the namespace's total stays under the quota; hold a per-node limit with admission policy.

To remove the quota, delete it as the cluster owner, and the kamiwaza-extensions quota too if you applied one:

kubectl -n kamiwaza delete resourcequota kamiwaza-owner-accelerators
kubectl -n kamiwaza-extensions delete resourcequota kamiwaza-owner-accelerators

Step 6: Set the Helm values

Add these values to the install's values file and upgrade the release with it:

core:
  inferenceResources:
    enabled: true
    bundleConfigMap: kamiwaza-inference-profiles-r1   # the name from step 5
    ownerPublicKey: <OWNER_PUBLIC_KEY>                 # publicKey from step 1
    clusterBindingId: <CLUSTER_BINDING_ID>             # from step 2

Helm value	Notes
bundleConfigMap	Kamiwaza reads the profile only from the ConfigMap with this name. Left at the default, kamiwaza-inference-profiles, it cannot read a renamed revision. On a namespace without an anchor, if an older ConfigMap with the default name still exists, Kamiwaza uses that older profile, with no error, for as long as it verifies.
ownerPublicKey	The canonical base64 of the raw 32-byte public key, as keygen and pubkey print it.
clusterBindingId	Must equal the profile's clusterBindingId.

On an anchored namespace, the anchor's values replace these three and minimumBundleVersion. Keep them equal to the anchor anyway: a Kamiwaza release from before the anchor ignores it, so rolling Kamiwaza back returns to these values.

On an install whose global.imagePullSecrets names the pull Secret, these four values are enough; an install that pulls every image without credentials needs no pull Secret. The staging values default as follows, and an explicit value always wins over its default:

Value	Default with tenant-mode inference on
gpuRegistry, cpuRegistry	The in-cluster model registry, which the model downloader already pushes to. It serves plain HTTP, so gpuRegistryInsecure and cpuRegistryInsecure default to true with it.
gpuStagingImage, cpuStagingImage	The chart's own model-fetch image, relocated like every chart image.
gpuRegistryCredentialsSecret, cpuRegistryCredentialsSecret	The first entry of global.imagePullSecrets, the install's pull Secret. Not defaulted when global.openshift.serviceAccountImagePullSecretsOnly is true.

Engine pods carry only this staging credential, as their image pull Secret: the engine and model-fetch images pull with it. Set gpuRegistryCredentialsSecret and cpuRegistryCredentialsSecret yourself only when your mirror needs different credentials from the install's pull Secret.

Deployments created before you turn tenant-mode inference on carry no tenant resources, so they cannot start again once it is on. After the upgrade, stop each one and deploy it again (see Deploy through the API); the new deployment takes its resources from the signed profile and its image from the engine catalog. While core.context.embedding.autoProvision is on (the default, except on global.platform: osgm), the platform's embedding model needs only the stop: the platform deploys it again on the next embedding request. With it off, deploy the embedding model again yourself.

While core.context.embedding.autoProvision is on, turning tenant-mode inference on for an install with no earlier embedding deployment also deploys the platform's bundled embedding model, all-MiniLM-L6-v2, on CPU, so expect one CPU engine pod shortly after the upgrade. On an install that already served the embedding model before you turned tenant-mode inference on, the platform finds that earlier deployment, still DEPLOYED, and deploys nothing until you stop it and an embedding request arrives. With it off, the platform does not deploy the embedding model on its own.

Step 7: Verify

Deploy a model that fits a GPU, from the Models page, then check the engine pods. The columns are the pod, its node, its RuntimeClass, the NVIDIA resources it was granted, and whether the grant was verified:

kubectl -n kamiwaza get pods -l kamiwaza.ai/deployment-id -o json | jq -r '.items[]
  | [.metadata.name, .spec.nodeName, (.spec.runtimeClassName // "-"),
     ([.spec.containers[].resources.limits // {} | to_entries[]
       | select(.key | startswith("nvidia.com/")) | "\(.key)=\(.value)"]
      | join(",") | if . == "" then "-" else . end),
     ([.status.conditions[]? | select(.type == "kamiwaza.ai/AllocationVerified") | .status]
      | first // "-")]
  | @tsv'

tmi-3f2a9c...   gpu-node-1   nvidia   nvidia.com/gpu=1           True
tmi-8b1e04...   gpu-node-1   nvidia   nvidia.com/mig-3g.40gb=1   True
tmi-51c7aa...   gpu-node-1   -        -                          True

Expect the nvidia RuntimeClass and the GPU or MIG resource the deployment asked for on a GPU pod, and no RuntimeClass or NVIDIA resource on a CPU pod. Every pod should read True in the last column. Readiness alone is not proof of the requested grant: the kamiwaza.ai/AllocationVerified condition is.

A CPU pod requests the engine's CPU floor from the engine catalog, 100m for llama.cpp and 500m for whisper.cpp, and memory of at least 1.25 times the model's weights, rounded up to a whole GiB and never below the floor of 1Gi for llama.cpp or 2Gi for whisper.cpp. It sets requests only, no limits.

Recipe catalogs (optional)

A recipe catalog pins a specific model and configuration to exact settings. Without one, every deployment is derived. Render each catalog with the owner tool, which validates it with the parser Kamiwaza runs when it mounts it and emits an immutable ConfigMap whose name carries the catalog's digest:

owner_tool render-catalog --catalog gpu-recipes.json \
  --namespace kamiwaza --capability gpu > gpu-catalog.json
kubectl apply --dry-run=server -f gpu-catalog.json
kubectl apply -f gpu-catalog.json
export GPU_CATALOG_CONFIGMAP="$(jq -r '.metadata.name' gpu-catalog.json)"

Use --capability cpu for a CPU catalog. Point the release at the exact name with core.inferenceResources.gpuRecipeCatalogConfigMap or cpuRecipeCatalogConfigMap. Publish a new revision for every change and never mutate a published ConfigMap. Keep the previous revision until every consumer has rolled out; roll back by selecting the previous name.

A pinned deployment runs the image its recipe variant names, and the profile binding's runtime.image must be the same reference. Kamiwaza does not relocate it: the bundle's image map, global.imageRegistryPrefix, and global.airgap.registry.url apply only to the engine catalog's images. In a mirrored or air-gapped install, name your mirror's copy, by digest, in both the recipe variant and the binding.

Engine images

The engine catalog in the core image pins every engine image by digest. Kamiwaza relocates those references once, when it loads the catalog, using the same values the chart relocates its own images with:

1. core.inferenceResources.engineImageMap, when set. The tenant OCI bundle writes it for you; see Bundle installs.
2. Otherwise global.imageRegistryPrefix: <prefix>/images/<name>@<digest>, where <name> is the last segment of the catalog's repository.
3. Otherwise global.airgap.registry.url: <registry>/<catalog repository>@<digest>.
4. Otherwise the reference as the catalog pins it.

An engine image must therefore be in your registry by digest, at the location the rule produces. Most catalog pins are untagged manifests, so a copy by tag does not bring them. Print the pins with the catalog command in step 4, and see Install Without Internet Access for the copy.

Engine pods pull with the staging registry Secret, core.inferenceResources.gpuRegistryCredentialsSecret and cpuRegistryCredentialsSecret, which default to the install's pull Secret; see step 6. There is no separate engine pull-secret setting. Set those two values only when your mirror needs different credentials.

A deployment that a recipe pins is the exception to the relocation above: it runs the image the owner signed, exactly as written. See Recipe catalogs.

After you change any of these values, redeploy the affected inference deployments: stop each one and deploy it again (see Deploy through the API). A running deployment keeps the engine reference it was planned with, and so does the platform's embedding model. Stop the embedding model's deployment too. While core.context.embedding.autoProvision is on, leave the redeploy to the platform: it deploys the model again on the next embedding request, such as POST /api/embedding/generate, and not before. That request returns HTTP 503 with the code embedding_deploying and a retry_after_seconds value, and embedding requests return it until the new deployment is ready, about 75 seconds on an air-gapped A100 install; retry them. With it off, the platform does not redeploy the model, and embedding requests return HTTP 503 with the code embedding_unavailable until you deploy it again yourself. On an upgrade, keep the previous release's engine images in your registry until every deployment has been redeployed; removing them breaks rescheduling.

Bundle installs

A tenant OCI bundle carries the engine images for one architecture, and writes values/99-engine-images.yaml with the map that tells Kamiwaza where it imported each one. Keep that file last among the install's values files. A cluster with amd64 and arm64 nodes needs a bundle for each architecture.

A bundle can leave engines out. An engine the bundle does not carry is refused at admission, before the deployment record exists:

cannot derive inference resources for this deployment: engine 'vllm' is not qualified in this release

Attest the NVIDIA driver's CUDA major

For each NVIDIA binding, the platform selects the newest image of the engine that both the card and the host driver can run. The card's limit is the computeCapability the binding attests. The driver's limit is cudaDriverMajor, an optional integer from 11 to 99 in the same accelerator block: the major of the CUDA Version that nvidia-smi prints in its header on the host (e.g., 12 on an R570 driver, which reports CUDA 12.8, and 13 on an R580 driver, which reports CUDA 13.0). An image is selected only when its CUDA major is at or below the attested value.

The detector in step 3 attests cudaDriverMajor by default. Without it, the platform does not judge the driver: a Turing or newer card on an R570 host is given the CUDA 13 llama.cpp image, which exits at startup with CUDA driver version is insufficient for CUDA runtime version. Re-detect and publish a new signed profile revision after any driver change, including a driver rollback.

A binding without cudaDriverMajor keeps verifying and selects the newest image its card can run:

• llama.cpp resolves its CUDA 13 image on compute capability 7.5 or later, and its CUDA 12 image on Volta (7.0).
• vLLM resolves its CUDA 13 image on amd64 on compute capability 8.0 or later, because the engine also requires bf16, and on arm64 only on a GB10 (12.1).
• Diffusion resolves its one image, CUDA 12, on compute capability 8.0 or later, on amd64 only.
• A Pascal (6.x) or older card is refused.

vLLM ships only a CUDA 13 image. On a binding that attests cudaDriverMajor: 12 the platform refuses it:

host driver supports CUDA 12 (attested cudaDriverMajor); engine vllm ships nvidia/amd64 images for CUDA 13+

On a binding that does not attest the fact, the platform cannot tell, and the vLLM image fails at startup on a CUDA 12 driver. When the card is older than every image, the reason names the compute capability instead:

llamacpp requires compute capability 7.0; this binding declares 6.1

A deployment without an inferenceResources block is refused with HTTP 422 before the deployment record exists, with the reason after cannot derive inference resources for this deployment:. This happens only when the driver or the card rules out every image on every GPU binding in the profile. A request that carries its own block is refused at binding selection: the deployment fails with NoFit and the reason in its error message. To resolve a driver refusal, upgrade the host driver, re-detect, and publish a new signed profile revision. No owner action resolves a card refusal, because the release ships no image for that card.

Before you roll Kamiwaza back to a release earlier than 1.3.0, publish a new signed profile revision without cudaDriverMajor and point the release at it. Earlier releases refuse the unknown key and reject the whole profile, so every tenant deployment, CPU deployments included, fails authorization.

Requests that name their own resources

A deployment from the Models page needs no inferenceResources block: the platform derives one. To choose the profile, device count or memory yourself, deploy through the API, POST /api/serving/deploy_model, with the block in the request body. The block names the logical profile and capability rather than a Node or a provider-specific inventory:

{
  "inferenceResources": {
    "schemaVersion": 1,
    "accelerator": {
      "capability": "gpu",
      "count": 1,
      "memory": {"minimum": "1Gi"},
      "isolation": "any-qualified",
      "profile": "nvidia-whole"
    },
    "runtime": {"selection": "automatic"}
  }
}

For cpu-standard, use the CPU request shape instead:

{
  "inferenceResources": {
    "schemaVersion": 1,
    "cpu": {
      "architecture": "amd64",
      "requests": {"cpu": "2", "memory": "4Gi"}
    },
    "runtime": {"selection": "automatic"}
  }
}

Do not add Node names, Node selectors owned by the tenant, or cluster-scoped resources to the tenant request.

Deploy through the API

Get a token as an administrator. Set KAMIWAZA_URL to your Kamiwaza domain; if the edge certificate is issued by a private authority, add --cacert <CA_FILE> to each curl command. The admin password is the one from your install page's verify step:

KAMIWAZA_URL="https://<KAMIWAZA_DOMAIN>"
printf 'admin password: '; read -rs ADMIN_PASSWORD; echo   # typed, not echoed
# curl reads the password from stdin, so it never appears on a command line.
TOKEN=$(printf '%s' "$ADMIN_PASSWORD" | curl -sS -X POST "$KAMIWAZA_URL/api/auth/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "username=admin" --data-urlencode "password@-" \
  | jq -r '.access_token')
unset ADMIN_PASSWORD

Find the model, its configuration and its weights file. A model is in the list once it has been downloaded; see Downloading Models:

curl -sS -H "Authorization: Bearer $TOKEN" "$KAMIWAZA_URL/api/models/" \
  | jq -r '.[] | [.id, .name] | @tsv'
MODEL_ID="<MODEL_ID>"
curl -sS -H "Authorization: Bearer $TOKEN" "$KAMIWAZA_URL/api/model_configs/?model_id=$MODEL_ID" \
  | jq -r '.[] | [.id, .name, .default] | @tsv'
curl -sS -H "Authorization: Bearer $TOKEN" "$KAMIWAZA_URL/api/models/$MODEL_ID" \
  | jq -r '.m_files[] | [.id, .name] | @tsv'

Write the request, with the configuration marked true in the second list and the weights file to serve, then deploy:

cat > deploy.json <<'JSON'
{
  "m_id": "<MODEL_ID>",
  "m_config_id": "<CONFIG_ID>",
  "m_file_id": "<FILE_ID>",
  "inferenceResources": {
    "schemaVersion": 1,
    "accelerator": {
      "capability": "gpu",
      "count": 1,
      "memory": {"minimum": "8Gi"},
      "isolation": "any-qualified",
      "profile": "nvidia-whole"
    },
    "runtime": {"selection": "automatic"}
  }
}
JSON
curl -sS -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  --data @deploy.json "$KAMIWAZA_URL/api/serving/deploy_model"

"631e7142-3417-487c-be65-d7715ba6cbf2"

The response is the deployment's ID, returned before the launch finishes. Follow its status until DEPLOYED; see Model Deployment for the statuses:

curl -sS -H "Authorization: Bearer $TOKEN" \
  "$KAMIWAZA_URL/api/serving/deployment/<DEPLOYMENT_ID>/status"; echo

To stop a deployment, e.g. before deploying it again after a registry change:

curl -sS -X DELETE -H "Authorization: Bearer $TOKEN" \
  "$KAMIWAZA_URL/api/serving/deployment/<DEPLOYMENT_ID>"; echo

The response is true, and the deployment's status becomes STOPPED.

Upgrade and recover

To change the profile, publish a new revision under a new name with a higher version. If the namespace has an accelerator quota, render and apply it from the new revision with owner_tool quota before you anchor it, so the old quota never refuses a deployment the new profile allows. Then anchor it. The anchor takes effect at the next deployment, with no upgrade or restart. Point bundleConfigMap at the new revision too, upgrade the release, and repeat the checks in step 7. On a namespace without an anchor, pointing bundleConfigMap at the new revision and upgrading is what changes the profile.

To roll back on an anchored namespace, the cluster owner re-signs the previous profile at a higher version and anchors that, or anchors the previous revision itself, which lowers the minimum version to its version. Pointing bundleConfigMap at an older revision has no effect there. On a namespace without an anchor, point bundleConfigMap at the previous revision.

If a create, readiness, authorization, or deadline step fails, the platform marks the allocation failed and removes only resources owned by that attempt; keep the prior known-good workload until the replacement passes its checks.

Upgrading Kamiwaza, including its core image, does not require a new signed profile. Sign a new revision only when your profiles change, or when the release notes say the profile schema changed.

To rotate the owner key, create a new key, sign a new revision with it, and anchor that revision with owner_tool anchor --public-key <NEW_PUBLIC_KEY>; the annotation changes the key, the profile name, and the minimum version together. Then set ownerPublicKey and bundleConfigMap together in one upgrade. On a namespace without an anchor, the upgrade is what rotates the key.

The anchor stops a superseded or foreign profile from being accepted. It does not stop a namespace administrator who can change Kamiwaza's own workloads, or turn tenant-mode inference off in the Helm values, from running GPU workloads without a profile. Hold that boundary as the cluster owner, with the accelerator quota and with admission policy. After any change to GPU hardware, MIG layout or device plugins, render and apply the quota again with owner_tool quota. A quota limits GPUs only on nodes that refuse GPU requests by environment; see GPU Nodes.

Turning that setting on for a node that already serves GPUs changes how pods get their GPU. Follow the order in Turn the setting on for a node that serves GPUs, which includes redeploying every GPU deployment before the node refuses the environment: a deployment whose pod holds its GPU by environment only starts without one when its container next restarts after the change.

Troubleshooting

A GPU deployment returns HTTP 422 with Core cannot see a GPU on this install. Tenant-mode inference is off. Turn it on as described in Turn on tenant-mode inference.

A deployment returns HTTP 503 with owner profile bundle: invalid inference profile authorization: and a reason. Kamiwaza found the profile but refused it. The reason names the check:

Reason	Fix
bundle did not arrive from the configured ConfigMap	bundleConfigMap does not name the published revision, or it is in another namespace.
bundle targets another cluster binding or namespace	Compare the profile's clusterBindingId and namespace with the anchor, or on a namespace without one, with the Helm values.
bundle version is below the minimum bundle version	The profile is older than the anchored minimum, or minimumBundleVersion on a namespace without an anchor. Publish and anchor a newer revision.
bundle is signed by a key this cluster does not trust	The anchor's keys, or ownerPublicKey without an anchor, do not include the key that signed the profile. Print it with owner_tool pubkey.
the namespace's owner anchor is malformed	The inference.kamiwaza.ai/owner-anchor annotation is not what owner_tool anchor writes. The cluster owner re-applies it.
the configured namespace is not this pod's namespace	KAMIWAZA_K8S_NAMESPACE names another namespace. Reinstall with the release's own values.
bundle is expired or its window is empty	Publish a new revision with a later expiresAt.

A deployment returns HTTP 422 with engine '<name>' is not qualified in this release. The engine is not in the engine catalog on this install: MLX is never in it, and a bundle install may have left the engine out.

A deployment returns HTTP 422 naming the driver's CUDA major or the card's compute capability. See Attest the NVIDIA driver's CUDA major.

A deployment returns HTTP 503 with owner profile bundle: the namespace's owner anchor could not be read. Kamiwaza cannot read its own namespace, so it refuses rather than trusting the Helm values. Upgrade the release so the chart restores the read, and check that the installer still holds the admin role in the namespace.

A deployment stays pending with exceeded quota: kamiwaza-owner-accelerators and a requests.<resource>. The namespace's accelerator quota is full for that resource, usually because another deployment holds it. Stop that deployment. This one starts at Kubernetes' next retry after that pod is gone, which can be up to about 11 minutes later. If it has been held for more than about 22 minutes, it gets no further retry before the scheduling deadline and fails; deploy it again. If the quota for the resource reads 0, either a --limit set it, or the profile does not grant it: render the quota without that limit, or publish a profile that grants the resource, then apply the quota again.

A request with its own block fails with NoFit. No binding in the profile holds the request: compare its memory, count, and isolation with the profile's bindings.

A deployment stays REQUESTED with no error, and its engine pod is Pending. The pod cannot be scheduled, and the deployment reports no error until the per-attempt scheduling deadline, schedulingTimeoutSeconds, passes. A tensor-parallel deployment that needs two whole GPUs while another deployment holds one of them is a common cause. Find the pod and read its events:

kubectl -n kamiwaza get pods -l kamiwaza.ai/deployment-id --field-selector=status.phase=Pending
kubectl -n kamiwaza describe pod <POD_NAME> | sed -n '/Events:/,$p'

Insufficient nvidia.com/gpu means no node has enough free GPUs of the requested kind; stop a deployment that holds one, or add GPUs. had untolerated taint means a GPU node is tainted; see GPU Nodes.

An engine pod fails with ImagePullBackOff. Read the image it pulls:

kubectl -n kamiwaza get pod <POD_NAME> -o jsonpath='{.spec.containers[*].image}'; echo

An image that names a registry you do not mirror means the relocation values are not set; see Engine images. An image in your registry means that digest was not copied there, or the staging registry Secret does not hold its credentials.

A deployment returns HTTP 503 with engine image map carries no image from this release's catalog. The bundle's map was built for another release. Build the bundle for the core image you install.

Support status

Support covers the configurations marked Supported in Compatibility scope, qualified on k0s Kubernetes 1.36, within the limits each row states. Anything the table does not mark Supported, such as AMD GPUs or a deployment that spans more than one device, is not supported in this release.