Skip to main content
Version: 1.3.3

Fractional GPU Serving

Not in tenant mode

This page describes placement on installs where Kamiwaza can read the cluster's Nodes. A namespaced install serves local models through tenant-mode inference instead. Placement there follows the bindings in the cluster owner's signed profile, not the node labels, VRAM budgets, and notices this page describes.

Fractional GPU serving lets multiple models share one GPU, each with its own reserved memory budget. Instead of claiming a whole card, a deployment reserves the number of gigabytes it is estimated to need; the remaining capacity stays available for other models. This page explains when fractional serving applies, how the budgeting works, and how to read the error you get when a model does not fit.

When fractional serving applies​

Fractional serving is governed by the hardware class of the GPU:

HardwareFractional?How sharing works
Discrete GPU without partitioning (T4, L4, A100/H100 with MIG disabled)YesPer-GPU memory budgets in GB
Unified memory (Apple Silicon, Strix Halo, DGX Spark)YesBudget against the shared system memory pool
MIG / hardware partitionsNoOne model per partition; the partition itself is the unit of sharing
Any GPU with fractional placement disabledNoWhole-GPU exclusive — one model per card

Fractional placement is enabled by default. Installations with stricter compliance requirements can disable it at install time with the Helm value placement.vramPluginV2.enabled=false on the placement-operator chart, which falls back to whole-GPU exclusive placement: the first model claims the card, and a second model on the same card fails with reason insufficient_capacity.

How the VRAM budget works​

When you deploy a model, Kamiwaza estimates its memory footprint — weights, context/KV cache, and per-deployment overhead. (The SDK exposes this same estimator as client.serving.estimate_model_vram() if you want to preview a deployment request's footprint.)

The estimate becomes a reservation against a specific GPU. Each GPU on a node exposes its capacity as a per-device resource named kamiwaza.ai/vram-gb-gpu-<i> (where <i> is the GPU index), measured in GB. Kubernetes enforces these budgets when the model is scheduled, so an over-budget model is refused before anything starts — it is never left half-running or silently pending.

Details worth knowing:

  • One budget per physical GPU. A node with four cards exposes four independent budgets (...-gpu-0 through ...-gpu-3); each model lands on exactly one card's budget.
  • Unified-memory machines expose a single budget (kamiwaza.ai/vram-gb-gpu-0) sized to the shared memory pool, since the CPU and GPU draw from the same pool.
  • Unified-memory machines hold back an 8 GiB operating-system reserve from the shared pool before the budget is advertised, so deployments cannot starve the host. Discrete-GPU budgets are sized to the card's detected VRAM.

On a standalone cluster you can see the advertised budgets directly:

kubectl get node <node-name> -o jsonpath='{.status.allocatable}' | tr ',' '\n' | grep vram-gb-gpu

Example: packing models on one GPU​

A 16 GB NVIDIA T4 with two 6 GB models:

  1. Deploy model A (estimated 6 GB). It reserves 6 GB on gpu-0. Remaining budget ≈ 10 GB.
  2. Deploy model B (estimated 6 GB). It fits the remaining budget and lands on the same card. Both deployments reach DEPLOYED and serve traffic concurrently.
  3. Deploy model C (estimated 6 GB). The remaining budget (≈ 4 GB) is too small. The deployment fails fast with the structured no-fit reason insufficient_capacity — no pending pod, no partial deployment.

On a node with mixed cards (say, an 8 GB and a 24 GB GPU), placement picks a card whose remaining budget fits the request, so a 10 GB model goes to the 24 GB card even if the 8 GB card is idle.

Models larger than one GPU​

A model whose footprint exceeds any single GPU can be deployed with tensor parallelism across multiple GPUs on one node: the request reserves a per-shard budget on each of N cards. If no node has N GPUs with enough free capacity, the request is rejected with insufficient_gpu_count. Multi-node parallelism is not supported in this release.

What a NoFit error means​

The deploy API is asynchronous: it accepts the request and returns the deployment ID immediately. When no eligible GPU (or set of GPUs) can satisfy the request, the deployment then fails fast — before any pod is created — instead of sitting pending. The deployment shows status FAILED with last_error_code set to NoFitError and the no-fit reason recorded in last_error_message. (When deploying through the SDK with the default wait=True, this surfaces as a DeploymentFailedError; see the Placement Deployment Guide.)

The reason tells you why the model did not fit:

ReasonMeaningCommon causes and what to do
insufficient_capacityNo single GPU (or partition) has enough free budget for the requestThe model is too large for the hardware, or other deployments hold the budget. Choose a smaller or more quantized variant, reduce context length, or stop an unused deployment.
insufficient_system_memoryUnified-memory budget exhaustedThe shared memory pool is fully reserved. Stop another deployment or pick a smaller model.
insufficient_gpu_countA tensor-parallel request needs N GPUs but no node has N freeLower the tensor-parallel size or free GPUs on a multi-GPU node.
vendor_mismatchThe selected engine requires a GPU vendor the cluster does not haveFor example, an engine that requires CUDA on an AMD-only cluster. Pick a model/engine variant that matches your hardware.
accel_version_unmetNo node satisfies the engine's minimum accelerator version (for example, a required CUDA version)Upgrade GPU drivers, or choose an engine/model variant built for your driver generation.
sharing_not_configuredA managed cluster has no GPU sharing configured and the model cannot fit as whole-GPUAsk your cluster admin to enable a sharing strategy, or free a GPU.

Limits to know​

Fractional budgets are reservations, not real-time meters:

  • Budgets track what Kamiwaza deployments have reserved. Memory used by non-Kamiwaza processes on the same GPU or host is not tracked, so a model that fit at scheduling time can still hit an out-of-memory error at load time if something else consumed the memory — surfaced on the deployment as error code OOM (see troubleshooting).
  • The serving engines self-limit to their allocation (for example, vLLM's GPU memory utilization setting), but enforcement is cooperative on shared cards.
  • Models sharing a GPU share compute and fault domains: there is no performance or crash isolation between them. If you need hard isolation, use hardware partitioning (hardware-isolated class).

See also​