Import a model without internet access

An air-gapped install cannot download models from the Hugging Face Hub. To run a model in the platform there, you pack it into a container image on a connected machine, move that image into your site registry with the rest of your images, and add one Helm value. The platform then imports the model, and you deploy it from the UI like any downloaded model.

Use this page when:

• your cluster has no internet access, and
• you want the model to run on the platform's own engines (llama.cpp, vLLM).

If a model download fails with HTTP 503 and a message that starts with Model hub (huggingface.co) unreachable (or your Hugging Face mirror's host), this page is the fix: the platform cannot reach the Hugging Face Hub. The whole message reads:

Model hub (<host>) unreachable: <ErrorClass>. On an air-gapped install, import the model with the offline model import (kz-model-pack + core.offlineModelImports); see the field guide.

<host> is the hub the platform is set to use, and <ErrorClass> names the kind of network error.

If you already serve the model from an HTTPS endpoint inside your site, you do not need an import. Register that endpoint as an external model instead; see Alternative: an external HTTPS endpoint.

If you install with the air-gapped field kit, follow section 8 of the kit's README.md, "Optional: bring a model in offline". It runs the same steps with the kit's own helm block and site.env. This page gives them for any air-gapped install.

How it works

1. Connected side. kz-model-pack.sh, in the field kit, downloads one model at one revision from Hugging Face. It writes a manifest, kamiwaza-model.json, that lists every file with its size and sha256, and builds an ordinary image that holds the manifest and the files. It prints the image reference and the manifest sha256.
2. Transfer. The image travels to your site registry like any other image.
3. Site side. You add the image and the manifest sha256 to core.offlineModelImports and run helm upgrade. A Job copies the files out of the image, checks the manifest sha256 and every file's sha256, and pushes the files to the platform's model registry. It then registers the model, so it shows as downloaded.

The manifest sha256 is the trust anchor. You copy it by hand from the pack output into your values file, so a changed manifest, or a changed file, stops the import.

The import needs namespace administrator rights only. For each entry the chart creates, in the platform's namespace, the Job, a ServiceAccount, a Role that allows get and list on pods and get and create on pods/exec, and a RoleBinding. It creates nothing cluster-scoped.

Before you start

You need	Notes
A connected machine with the field kit, python3, and Docker with buildx that builds multi-platform images (Docker Desktop does)	Runs kz-model-pack.sh. It needs to reach huggingface.co and a registry you can push to; sign in to that registry with docker login first. A gated or private model also needs a Hugging Face token: export HF_TOKEN="<your token>". To download through a Hugging Face mirror, export HF_ENDPOINT="https://<your mirror>"; without export the pack does not see it and downloads from huggingface.co.
The model's Hugging Face repository ID	For example Qwen/Qwen3-0.6B-GGUF. The pack resolves the revision you give it to a commit and records that commit.
On the site side, the field kit, python3, kubectl, helm and curl; skopeo if your registry needs credentials and the kit has no site.env for it	kz-model-pack.sh verify checks the copy in your registry. kubectl needs namespace administrator access to the platform's namespace.
Space in your site registry	About the size of the model, for the weights image.
Space in the platform's model registry	At least 1.1 times the size of the files it does not already hold. The import checks this before it pushes anything.
A running Kamiwaza 1.4.1 install, its values file and its helm command	Add the import once the platform is installed and working, in an upgrade.

Step 1: Pack the model (connected side)

On the connected machine, from the field kit directory, pack the model and push the image to a registry you can copy from. Replace registry.example.com/models with that registry and path, and the tag with one of your own that you never reuse for a different pack:

bash kz-model-pack.sh pack Qwen/Qwen3-0.6B-GGUF --revision main \
  --include 'Qwen3-0.6B-Q8_0.gguf' --include LICENSE \
  --primary-file Qwen3-0.6B-Q8_0.gguf \
  --image registry.example.com/models/qwen3-0.6b-gguf:2026-11-12

The pack resolves --revision (a branch, tag or commit) to a commit and downloads the files of that commit. Without --include it downloads every file; for a repository that holds several quantizations, name the files you need with --include. --exclude drops files. Both take a glob and can repeat. A vLLM model needs every file: weights, config.json and the tokenizer files. When you keep several weights files, --primary-file names the one the model's default configuration uses; otherwise Kamiwaza picks it as it does for a download. bash kz-model-pack.sh pack --help lists every option.

File names must also be names the platform's model registry accepts:

• Each part of a file's path may use only A-Z, a-z, 0-9, ., _ and -, and may not be made only of ., _ and -.
• Two paths may not differ only in case.
• Each part, once lowercased with _ read as - and its leading ., _ or - dropped, must end with a letter or digit, and every . or run of - must sit between letters or digits. So __init__.py, foo_.bin, a..b.bin and a name ending in . are refused.
• The model's repository ID follows the same rule, with / read as -: for example Acme/Tiny_ is refused.
• No two files may end up with the same registry name, with folders joined by -: a_b.bin and a-b.bin, sub/a.bin and sub-a.bin, or .config.json and config.json cannot be packed together.
• Each file's full registry name, models/<repository ID>/<file path> after these changes (models is the default project; a site that sets KAMIWAZA_MODEL_PROJECT uses its own prefix, and the error names it), must be at most 255 characters. A very long repository ID or a deeply nested path can pass the other rules and still be refused.

The platform refuses any other name, so the pack names each such file and stops; leave it out with --exclude. A refused repository ID cannot be changed at pack time: the import records the model under that ID. Pack the model from a copy published under an ID the rule accepts, for example on your Hugging Face mirror (HF_ENDPOINT), or contact Kamiwaza support.

When it finishes, the pack prints:

• the image it pushed
• the manifest sha256, 64 hex characters
• a core.offlineModelImports entry for your values file, with the name, the image, the manifest sha256 and a stagingSize filled in

Write down the manifest sha256 and the entry: you type them in at the site. The image is built from the release's model-fetch image, which is already in your site registry, so only the model layers are new to your scanners. Add --fips to build from the FIPS model-fetch image instead.

To check the image before it leaves, verify it against the manifest sha256 the pack printed. This reads the registry anonymously, so it works for a public registry; for a private one, add --output-oci model-oci to the pack command and verify the directory model-oci instead of the image:

bash kz-model-pack.sh verify registry.example.com/models/qwen3-0.6b-gguf:2026-11-12 \
  --manifest-sha256 "<manifest sha256 the pack printed>"

Step 2: Move the image to your site

Copy the image into your site registry with your normal transfer process. Both kinds of copy work:

• Digest-preserving (skopeo copy --all --preserve-digests, crane copy): the image keeps its digest.
• Re-pushing (docker save and load, a transfer that rebuilds the image): the digest changes. That is fine; the import does not trust the image digest. It checks the manifest sha256 you give it and the sha256 of every file. docker save and load can keep only your machine's platform; the copy must still hold your nodes' platform.

Push the image where the chart pulls it. With global.imageRegistryPrefix set, as on every air-gapped install, the chart pulls every image from <prefix>/images/<last segment of the repository>. For the example above and the prefix the air-gapped install uses, that is <your registry host>/kamiwaza-ai/releases/kamiwaza/images/qwen3-0.6b-gguf. Keep the tag.

Step 3: Check the image on the site side

Before you add it to your values, check that what arrived is what you packed. On the site side, open a shell in the field kit directory and set these five values. You use them in every step that follows; set them again in any new shell.

NS="kamiwaza"                     # replace: the namespace Kamiwaza is installed in
REGISTRY="<your registry host>"   # replace: bare host[:port], as on the air-gapped install page
MODEL="qwen3-0-6b-gguf"           # replace: the name: line of the entry the pack printed
IMAGE="$REGISTRY/kamiwaza-ai/releases/kamiwaza/images/qwen3-0.6b-gguf:2026-11-12"   # replace the last segment and the tag
SHA="<manifest sha256 the pack printed>"   # replace

IMAGE ends with the last segment of the repository you packed to and your tag. SHA is 64 characters, 0-9 and a-f.

If a private CA signs your registry's certificate, put that CA in the field kit directory as site-ca.pem, or name it for this shell with export REGISTRY_CA_FILE=/path/to/ca.pem (that file is then the only CA trusted). Without it, the check stops with a certificate error.

bash kz-model-pack.sh verify "$IMAGE" --manifest-sha256 "$SHA"

When the image's registry host is the HARBOR_URL in the field kit's site.env, the check signs in with that file's HARBOR_USERNAME and HARBOR_PAT; it reads any other registry anonymously. If your registry needs credentials and the kit has no site.env for it, copy the image to a directory and check that instead:

skopeo login "$REGISTRY"
skopeo copy --all "docker://$IMAGE" oci:model-oci
bash kz-model-pack.sh verify model-oci --manifest-sha256 "$SHA"

site-ca.pem and REGISTRY_CA_FILE reach only the kit. For a private CA, give skopeo a directory that holds the CA as ca.crt, with --cert-dir <dir> on login and --src-cert-dir <dir> on copy.

The check reads every platform of the image and checks the manifest sha256, the manifest itself, and every file's size and sha256. It prints an OK line for each platform, then must end with verified ...: every file matches manifest .... A FAIL line names each file that is missing, extra or changed; an ERROR line names a malformed image or a failed connection. Any outcome other than that final verified line means do not import. The import repeats these checks, so this step only finds a bad transfer sooner. verify also accepts a docker save tar from Docker 25 or later, unpacked with tar -xf; an older tar has no index.json, and verify refuses it.

Step 4: Add the import to your values

Type the entry the pack printed under core.offlineModelImports in your values file. Merge it into the file's existing core: key; do not add a second one.

core:
  offlineModelImports:
    - name: qwen3-0-6b-gguf
      image:
        registry: registry.example.com
        repository: models/qwen3-0.6b-gguf
        tag: "2026-11-12"
        digest: ""
      manifestSha256: "<manifest sha256 the pack printed>"
      stagingSize: 671Mi

Use the values the pack printed for name, stagingSize and the image. Keep manifestSha256 in quotes.

Value	What it means
name	A short name for this import: lowercase letters, digits and -, at most 36 characters. The pack derives it from the model's name. It names the import Job and is the value of the Job's kamiwaza.ai/offline-model-import label. Each entry needs its own name.
image	The weights image, in the same shape as other chart images. global.imageRegistryPrefix relocates it, so registry and the path before the last segment of repository do not matter on an air-gapped install. It needs a tag or a digest. Leave digest empty if your transfer changes digests; if it keeps them, you may fill in the digest the pack printed. Under global.fips.enabled, the same image is used unless you add an image.fips block; pack with --fips for a FIPS site.
manifestSha256	The manifest sha256 that kz-model-pack.sh printed: 64 lowercase hex characters, with no sha256: prefix. Required.
stagingSize	How much scratch space the Job may use to hold the files. Required. It must be at least the model's size; the pack fills it in at 1.1 times that, for headroom.
activeDeadlineSeconds	Optional, default 3600 (one hour), at least 300. How long the Job may take to pull the image, copy the files and import them. One hour suits a model of a few GB. For a larger one, allow about one hour per 25 GB, rounded up: for a 70 GB model, 10800.
resources	Optional. CPU and memory for the import container.
skipRegistrySpaceCheck	Optional, default false. Leave it false. true skips the model registry free-space check; the import logs that it did not check.

To import several models, add one entry for each.

The Core image may be pinned by tag or by digest; both work, so a field kit in tag mode or in digest mode can run the import. With only a tag on the weights image, a copy pushed again under the same tag looks unchanged to the chart, so nothing reruns; use a new tag for a new pack.

Treat the weights image like any other image you run. The stage-weights container runs it before the import checks its files, on the network that reaches the model registry, so a tampered image could change the registry even though the import then refuses it. Pin it by digest when your transfer keeps digests, and check it with Step 3 before you add its entry.

Check the entry before you upgrade. The render needs no cluster access. Run it in the same shell as Step 3, so REGISTRY and IMAGE are set, from the directory that holds your values files, with every values file you install with, in the same order:

helm template kamiwaza oci://$REGISTRY/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version 1.4.1 --namespace "$NS" \
  --values kamiwaza-values.yaml > rendered.yaml
grep -E '^[[:space:]]+(- )?image: ' rendered.yaml | grep -F "${IMAGE%:*}"

The second command must print one image: line, the weights image as the Job pulls it. ${IMAGE%:*} is IMAGE without its tag, so this works for both forms of the entry: with digest empty the line ends with your tag; with digest filled in it ends with @sha256: and that digest, and no tag. If it prints nothing, the chart pulls the weights image from somewhere else: check repository and global.imageRegistryPrefix. If the render fails, its message names the value to fix; see The chart refuses to render.

Step 5: Run the import

Run the same helm upgrade --install command you installed with, with every values file in the same order, and the file that holds the entry added. Run it from the same shell and directory as the render above. For example:

helm upgrade --install kamiwaza \
  oci://$REGISTRY/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version 1.4.1 \
  --namespace "$NS" \
  --values kamiwaza-values.yaml \
  --timeout 70m

The chart starts one Job for each entry, named core-model-import-<name>-<hash> and labeled kamiwaza.ai/offline-model-import=<name>. Each Job waits for the database and the model registry, then imports its model. helm upgrade returns as soon as the Job is created. It does not wait for the import, and a failed import does not fail it. Wait for the Job:

kubectl -n "$NS" wait --for=condition=complete --timeout=65m job -l kamiwaza.ai/offline-model-import="$MODEL"

It prints condition met when the import has finished. 65m is the Job's default deadline, one hour, plus 5 minutes; if you set activeDeadlineSeconds, wait that long plus 5 minutes. If it prints no matching resources found, check that MODEL is the entry's name and that the helm upgrade used the file that holds the entry.

A failed import does not end the wait early: the wait runs to its timeout even when the Job failed in its first minute. A few minutes after the upgrade, check the Job in a second shell, with the Step 3 values set there too:

kubectl -n "$NS" get jobs -l kamiwaza.ai/offline-model-import="$MODEL"

STATUS Running means wait on, even with a pod in Error: the Job is trying again. Complete means done; Failed means read the logs now. (On Kubernetes 1.28 and 1.29 there is no STATUS column; read the Job's conditions with kubectl -n "$NS" describe job -l kamiwaza.ai/offline-model-import="$MODEL".) For a failed Job, see When the import fails. Do the same if the wait ends with timed out.

Then read the import's receipt, one JSON line:

kubectl -n "$NS" logs -l kamiwaza.ai/offline-model-import="$MODEL" -c import --tail=-1 | grep '"model_id"'

The receipt names the model (repo_id, revision, model_id), its default configuration (model_config_id) and every file it pushed (files). "changed": true means the import created or updated the catalog entry (the model, its files or its default configuration); "changed": false means the catalog entry was already there with the same files. changed covers only the catalog entry: a rerun can still write a missing public-viewer grant, which public_viewer reports. registry_space_check says how the free space was checked. public_viewer and public_viewer_tenant say who can see the model; see Who can see the imported model.

Leave the entry in your values file. The Job name carries a hash of the whole Job: the entry, the Core image, and the Core settings the Job inherits. A later helm upgrade that changes none of them leaves the finished Job alone. One that changes any of them, such as an upgrade to a new Kamiwaza release or a change to other Core values, runs the import again: it pulls and copies the image again, checks the model and changes nothing. Leave node ephemeral storage free for that copy, as Sizing describes.

Keep the weights image in your site registry while its entry is in your values file, because a later helm upgrade pulls it again, and while you might roll back to a revision that lists it, because a rollback may run that revision's import again. When you clean up the registry, add it to your keep list by hand, as Decide what to keep describes for images you copied yourself. To retire it, remove the entry first. The imported model stays in the platform after you remove the entry.

Step 6: Deploy the model

Sign in, open Models, and find the model by its name. It shows as downloaded. Deploy it as you would any downloaded model. See Model Deployment.

Who can see the imported model

On an install with relationship-based access control (ReBAC), a model from the Hugging Face Hub gets the same public-viewer grant a download writes, so signed-in users can find it. The receipt's public_viewer says what the import did:

public_viewer	Meaning
granted	The grant is written. public_viewer_tenant names the one tenant it is written for.
rebac-disabled	ReBAC is off; no grant is needed. public_viewer_tenant is null.
auto-assign-disabled	Automatic grants are turned off on this install (AUTH_REBAC_AUTO_ASSIGN_OWNER is false for the Core API, and the import follows it), so the import wrote none. Write the grant yourself, in each tenant whose users need the model, with the call below.
not-required	The model's hub needs no public-viewer grant.

The grant is written for one tenant only: your install's default tenant, the same one the Core API uses. That is core.scheduler.rebac.defaultTenantId (default __default__), unless an AUTH_REBAC_DEFAULT_TENANT_ID entry in core.scheduler.extraEnv or core.scheduler.linuxNvidia.extraEnv overrides it (the last one wins). The receipt's public_viewer_tenant names it. A site with one tenant needs nothing more. On a ReBAC site with several tenants, users in the other tenants do not see the model from the import. Until you grant it in their tenant, only the users of public_viewer_tenant can find and deploy it.

To grant it, a Kamiwaza administrator calls POST /api/auth/tuples once for each other tenant. First find the public-viewer subject. It is AUTH_REBAC_PUBLIC_MODEL_VIEWER_URN when your install sets it (in core.scheduler.extraEnv or core.scheduler.linuxNvidia.extraEnv; the last one wins), and urn:li:corpuser:model-public otherwise. The import uses the Core API's settings for the subject, the default tenant and automatic grants. Read it from the Core API's settings:

kubectl -n "$NS" get deploy core-api -o yaml | grep -A1 AUTH_REBAC_PUBLIC_MODEL_VIEWER_URN

No output means the default, as long as kubectl printed no error. NotFound means NS is wrong; Forbidden means your account cannot read that namespace (a wrong NS, or ask the cluster owner). A value: line is the subject. A valueFrom: line means it comes from a ConfigMap or Secret: run the command again with grep -A4, and read the key it names, for example kubectl -n "$NS" get configmap <name> -o yaml. A subject urn:li:corpuser:<id> or urn:li:user:<id> is namespace user and id <id>; urn:li:group:<id> is namespace group. The default is namespace user, id model-public, as below.

Then get an administrator's token. Set KAMIWAZA_URL once, then use one of the two ways below. If a private CA signs the platform's certificate, add --cacert /path/to/ca.pem to every curl command.

KAMIWAZA_URL="https://<your Kamiwaza host>"   # replace

• Any install: an Admin-scope personal access token. Mint it from the console as an administrator, choosing Admin (full access) as its scope; a token with the default scope cannot write grants. This is the only way on an install whose sign-in redirects to your identity provider (core.auth.login.mode: redirect, which global.securityProfile: stig selects); see Use personal access tokens for automation. Paste it without echo:

  printf 'Admin-scope token: '; read -rs TOKEN; echo
  [ -n "$TOKEN" ] && echo "token set" || echo "no token: paste it again"

• Standard sign-in only: an administrator's password.

  ADMIN_USER="admin"                            # replace: an administrator's username
  printf 'password for %s: ' "$ADMIN_USER"; read -rs ADMIN_PASSWORD; echo
  OUT=$(printf '%s' "$ADMIN_PASSWORD" | curl -sS -w '\n%{http_code}' -X POST "$KAMIWAZA_URL/api/auth/token" \
    -d grant_type=password --data-urlencode "username=$ADMIN_USER" --data-urlencode "password@-")
  unset ADMIN_PASSWORD
  echo "sign-in: HTTP $(printf '%s\n' "$OUT" | tail -n 1)"
  TOKEN=$(printf '%s\n' "$OUT" | sed '$d' | python3 -c 'import json, sys; d = sys.stdin.read(); print(json.loads(d).get("access_token", "") if d.startswith("{") else "")')
  [ -n "$TOKEN" ] && echo "token ok" || echo "token failed: $(printf '%s\n' "$OUT" | sed '$d')"

  It must print HTTP 200 and token ok. 401 means a wrong username or password. 403 with password_login_disabled means the install uses redirect sign-in: use an Admin-scope token instead. It does not mean the user is not an administrator.

Then write the grant. Set MODEL_ID to the receipt's model_id and TENANT to the other tenant's ID, and run the curl once for each tenant:

MODEL_ID="<model_id from the receipt>"        # replace
TENANT="<another tenant's ID>"                # replace
curl -sS -w '\nHTTP %{http_code}\n' -X POST "$KAMIWAZA_URL/api/auth/tuples" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"subject":{"namespace":"user","id":"model-public"},"relation":"viewer","object":{"namespace":"model","id":"'"$MODEL_ID"'"},"tenant_id":"'"$TENANT"'"}'

It prints Kamiwaza's answer, then the HTTP status:

Status	Meaning
204	The grant is written. Writing it again is harmless.
401	The token is missing, wrong or expired: get a new one.
403	The token is not an administrator's (Admin role required), or is a personal access token without the Admin scope (Forbidden).
404 rebac_disabled	ReBAC is off on this install; no grant is needed.
422	The grant cannot be stored; the answer names why.

If you set a different subject, change namespace and id in the subject. For any other answer, or a 422, contact Kamiwaza support with the receipt and the answer.

Sizing

• stagingSize. The Job copies the whole model into a scratch volume limited to stagingSize, and asks for that much ephemeral storage on its node. A node without that much free stays unused; if none has it, the pod waits. If the model does not fit in stagingSize, Kubernetes stops the pod. See Errors.
• Model registry. The platform's model registry keeps its data on a persistent volume. Before it pushes anything, the import measures that volume's free space. It needs at least 1.1 times the size of the files the registry does not already hold. If your registry stores its data in an S3-compatible bucket instead, the import cannot measure it; it logs a warning and continues.
• Several entries. The Jobs for all entries start together, but only one import at a time pushes new file data. A first import of a model, or one with a file not yet in the catalog, holds the lock from its free-space check through its pushes, the read-back and the catalog update, so imports cannot together overfill the registry or claim the same registry names. A rerun whose files are all in the catalog and still in the registry does not wait; one whose file data went missing from the registry waits only for its free-space check and push. The others log Waiting for another offline model import to finish its space check and push. A Job's activeDeadlineSeconds bounds all its attempts, waits included, so size each entry's deadline for its own pull, copy, push and read-back plus the push and read-back of the other entries that may go first: in practice, the sum across the entries. Or add large models one helm upgrade at a time. A first attempt that waits half its deadline (30 minutes by default) stops with ImportLockTimeout and tries again; the retry starts with less time left and may end with the Job's deadline.
• Site registry. The weights image is about the size of the model.
• Deadline. activeDeadlineSeconds covers the image pull, the copy and the import. If you run helm with --wait --wait-for-jobs, keep its --timeout above the largest deadline; when core.scheduler.installerHelmTimeout is set, the chart refuses a deadline that is not below it.

When the import fails

Check the Job and its pods:

kubectl -n "$NS" get jobs -l kamiwaza.ai/offline-model-import="$MODEL"
kubectl -n "$NS" get pods -l kamiwaza.ai/offline-model-import="$MODEL"

The import has failed when the Job's STATUS is Failed (on Kubernetes 1.28 and 1.29, when kubectl describe job shows a Failed condition). A pod in Error while the Job is still Running is an attempt that the Job is retrying; do not delete the Job for it. Read the logs of its containers. stage-weights copies the files out of the image; wait-db-schema and wait-model-registry wait for the database and the model registry and log what they are waiting for; import checks and imports the files:

kubectl -n "$NS" logs -l kamiwaza.ai/offline-model-import="$MODEL" -c stage-weights --prefix --tail=-1
kubectl -n "$NS" logs -l kamiwaza.ai/offline-model-import="$MODEL" -c wait-db-schema --prefix --tail=-1
kubectl -n "$NS" logs -l kamiwaza.ai/offline-model-import="$MODEL" -c wait-model-registry --prefix --tail=-1
kubectl -n "$NS" logs -l kamiwaza.ai/offline-model-import="$MODEL" -c import --prefix --tail=-1

On Istio with classic sidecar injection, the two waits run inside the import container, so their lines are in its log, and the two middle commands answer that the container is not found.

A pod that never started (Pending, ImagePullBackOff) has no logs, and an eviction for stagingSize shows only in the events (Usage of EmptyDir volume ... exceeds the limit), not in the stage-weights log. Read the events:

kubectl -n "$NS" describe pods -l kamiwaza.ai/offline-model-import="$MODEL"

The import stops at the first problem and logs one line that names the exit code and the error, for example Offline model import refused (exit 2, permanent): FileDigestMismatch: ....

• Exit 2, permanent. Running the same image again cannot succeed. The Job fails at once. stage-weights exits 2 the same way when the image is not a weights image or holds no model files, and wait-db-schema or wait-model-registry exits 2 on a refusal that waiting cannot fix.
• Exit 3, transient, or exit 1, unexpected. The Job starts a new pod and tries again, up to three attempts in all, so the log can show the same error from several pods (--prefix names each pod). An eviction for stagingSize counts as an attempt; a preemption or node drain does not.

Fix the cause using Errors. If you changed the entry, run the helm upgrade from Step 5 again. If you did not, the failed Job keeps its name, so delete it first, then run the same helm upgrade, then wait again:

kubectl -n "$NS" delete job -l kamiwaza.ai/offline-model-import="$MODEL"

The Step 5 command never rolls back, so a failed Job stays for its logs. If your helm command uses --wait --wait-for-jobs with --rollback-on-failure (or --atomic), or your installer sets HELM_ROLLBACK_ON_FAILURE=true, a failed import fails the upgrade, and the rollback deletes the failed Job with its logs. The rollback may also rerun the previous revision's import Jobs (a Job whose name differs between the two revisions), so keep their weights images in your registry. For the upgrade that adds an entry, turn that off to keep the Job for kubectl logs: leave --rollback-on-failure and --atomic out of your own helm command, or set HELM_ROLLBACK_ON_FAILURE=false for an installer script that reads it.

Errors

Each named error below, except PublicViewerGrantFailed and PublicViewerGrantUnavailable, leaves the platform's catalog unchanged. Files the import already pushed stay in the model registry, where a later import reuses them.

Error in the log	What it means	What to do
ManifestDigestMismatch	Exit 2. The manifest in the image is not the one whose sha256 you set in manifestSha256.	Check that you copied the sha256 from the pack output for this image. If you did, the image is not the one you packed; transfer it again.
FileDigestMismatch	Exit 2. A file's size or sha256 does not match the manifest. The error names the file.	The image was damaged or changed in transfer. Transfer it again and check it with kz-model-pack.sh verify (Step 3). If you push it again under the same tag, a node may still hold the old copy: set image.pullPolicy: Always in the entry (in image.fips too, if the entry has one), or use a new tag.
ManifestInvalid	Exit 2. The manifest breaks a rule, for example an unknown schema or key, an unsafe file path, a file name or repository ID the model registry would not accept, two files with the same registry name, or a registry name over 255 characters (see the file name rules in Step 1). The error names the field.	Pack the model again with the kz-model-pack.sh from this release's field kit, and --exclude any file it names.
StageDirInvalid	Exit 2. The files copied out of the image are not exactly the files the manifest lists: one is missing, extra, a link or a special file.	Pack the model again, then check the image with kz-model-pack.sh verify.
RegistryInsufficientSpace	Exit 2. The model registry does not have room: the free-space check found too little, or the registry answered a push with HTTP 507. The error gives the bytes needed and the bytes free, or the 507.	Free registry space, or grow its volume, then run the import again.
RegistrySpaceCheckFailed	The import could not measure the registry's free space. The error names the cause. It is exit 3, and retried, when the Kubernetes API cannot be reached or answers with a server error, a throttle or a missing pod, when the measurement times out, or when there is not exactly one running registry pod: a second replica, a pod not yet Running, or a rollout in progress. It is exit 2 when no pod matches the registry selector at all (a stopped registry, or a wrong selector), when the registry pod has no container of the configured name, when the API refuses the Job's account (HTTP 401 or 403), when the registry image has no df, or when the registry pod's output cannot be read.	For exit 3, wait until exactly one registry pod is Running and Ready (instance= is your Helm release name): kubectl -n "$NS" get pods -l 'app.kubernetes.io/name=registry,app.kubernetes.io/instance=kamiwaza,app.kubernetes.io/component!=registry-gc'. For HTTP 401 or 403, check that the chart's Role and RoleBinding exist: kubectl -n "$NS" get role,rolebinding core-offline-model-import-registry-exec, and that no policy on your cluster blocks pods/exec. If the error says the API could not be reached, check that a NetworkPolicy allows egress to the Kubernetes API server for pods labeled app.kubernetes.io/name=core-offline-model-import: kubectl -n "$NS" get networkpolicy. On OpenShift, that rule comes from the kamiwaza-namespace chart (with openshift.platformApiEgress.enabled), not from the helm upgrade in Step 5; if your namespace was prepared with an older one, add the rule yourself, as Allow the import to reach the API server shows. For any other exit-2 cause, fix what the error names first. Then run the import again.
RegistryAccessDenied	Exit 2. The model registry refused a push, a check for a file it holds, or a read-back with HTTP 401 or 403.	Read the registry's log (instance= is your Helm release name, kamiwaza on this page): kubectl -n "$NS" logs -l 'app.kubernetes.io/name=registry,app.kubernetes.io/instance=kamiwaza,app.kubernetes.io/component!=registry-gc' --tail=200. Check whether your values set registry.auth.enabled: true (htpasswd), and, on Istio, whether an AuthorizationPolicy in the namespace denies the import pod: kubectl -n "$NS" get authorizationpolicy (a Forbidden answer means your account cannot list them, not that the import failed; ask the cluster owner). Then run the import again.
RegistryTransportError	Exit 3, retried. A push to, check of, or read-back from the model registry failed with a network error, a server error, a throttle, or a missing file just after the push, and its own retries ran out.	Check that the registry pod is Running and Ready (see RegistrySpaceCheckFailed), then run the import again.
RegistryPushMismatch or RegistryReadbackMismatch	Exit 2. The model registry stored, or returned, a file that does not match what was pushed.	Read the registry's log (the command in the RegistryAccessDenied row) and check its volume is Bound and not full: kubectl -n "$NS" get pvc. Then run the import again.
BindingConflict	Exit 2. The catalog already holds something that conflicts with this import and that the import will not change. The import never replaces files. The message names the cause and its remedy: another revision of the model; a file with different bytes; a non-default configuration that already uses the default configuration's name; a default configuration that points at a different primary file; a file already stored at another location, for example from a download before the site was air-gapped; a file bound to another manifest digest; a file whose registry name another catalog file already uses, either a file of this model (for example a_b.gguf imported earlier and a-b.gguf now) or a file of a model whose repository ID becomes the same name; the message names that file's ID, model and storage location; a hosted (not local) model with the same ID; or duplicate model rows.	Do what the message says. A configuration conflict needs no deletion: rename that configuration, or pack again without --primary-file. For a default configuration that uses another file than the pack's primary file, only clear its file and import again, as Clear the default configuration's file shows; nothing is deleted, and this also works when the message names the primary file as (None). For another revision, pack the revision the catalog records (the message names it), or delete the whole existing model as below. For a registry name already in use, different bytes, another storage location or another manifest digest, delete the file row the message names, as Delete a catalog file row shows. Or delete the whole existing model in Models, after you check that nothing you need uses it: deleting a model removes its configurations and deployments. For a registry name in use, renaming the file instead works only if the renamed file is published on the Hugging Face Hub or mirror the pack reads; if you do not need the file, leave it out with --exclude and pack again. Then run the import again. For a hosted model or duplicate rows, contact Kamiwaza support with the message.
ActiveDownload	Exit 3, retried. A download of the same model is running.	Wait for it to end or cancel it, then run the import again.
ImportLockTimeout	Exit 3, retried. Another import held the import lock for longer than this Job may wait: half its activeDeadlineSeconds. The error names the holder.	Let the other import finish (kubectl -n "$NS" get jobs -l kamiwaza.ai/offline-model-import), then run this one again. If it keeps happening, raise activeDeadlineSeconds or add the models one helm upgrade at a time.
CatalogDatabaseUnavailable	Exit 3, retried. The platform's database could not be reached, dropped the connection, or timed out on the catalog lock. The error names the cause.	Check that the database pod is Running and Ready: kubectl -n "$NS" get pods. Then run the import again.
ImportUsageError	Exit 2. The import was started with a missing or malformed argument or setting, such as a model registry address that is not exactly host[:port], a malformed model registry project name, the lock wait, or an AUTH_REBAC_PUBLIC_MODEL_VIEWER_URN that is not a valid subject. The error names it, and nothing was pushed.	Check the entry against Step 4. For the public-viewer subject, fix or remove the AUTH_REBAC_PUBLIC_MODEL_VIEWER_URN entry in core.scheduler.extraEnv or core.scheduler.linuxNvidia.extraEnv (see Who can see the imported model). Otherwise contact Kamiwaza support with the Job log.
PublicViewerGrantFailed	Exit 1, retried. The model is imported, but the import could not write its public-viewer grant, so other users may not see it. It comes after the catalog changes.	If the Job still fails after its attempts, fix the cause the error names, delete the Job and run the same helm upgrade again; the import changes no rows and retries the grant.
PublicViewerGrantUnavailable	Exit 3, retried. The model is imported, but the store that holds access grants could not be reached for its public-viewer grant. It comes after the catalog changes.	The same as PublicViewerGrantFailed: if the Job still fails after its attempts, check that the platform's database and authorization services are Running (kubectl -n "$NS" get pods), delete the Job and run the same helm upgrade again.
RegistryRequestFailed	Exit 1, retried. The model registry answered with an HTTP status the import does not expect, for example 400 or 422. The error names the status.	Read the registry's log (the command in the RegistryAccessDenied row). If the cause is not clear, contact Kamiwaza support with the Job log and the registry log.
Usage of EmptyDir volume ... exceeds the limit in the pod's events (the pod is evicted), or stage-weights fails with No space left on device; up to three attempts, then the Job fails	The files do not fit in stagingSize.	Raise stagingSize, then run helm upgrade again.
The pod stays Pending with Insufficient ephemeral-storage	No node has stagingSize of free ephemeral storage.	Free space on a node, or lower stagingSize, but never below the model's size.
wait-db-schema exits 2, logging that the Core database refused this client	Exit 2: the Job fails at once, after the database refused the import's login or a permission for about a minute: a wrong password (password authentication failed), the database's own client rules (no pg_hba.conf entry), or a missing privilege (permission denied).	The import logs in as user core, with the password in key password of Secret core-postgres; that password must still match the database's password for core, for example after a restore or a rotation. If the rest of the platform works, the Secret matches. Check the settings the Job uses: kubectl -n "$NS" get job -l kamiwaza.ai/offline-model-import="$MODEL" -o yaml | grep -A4 'name: POSTGRES_'. Check that the Secret holds a password: kubectl -n "$NS" get secret core-postgres -o jsonpath='{.data.password}' | wc -c prints a number above 0 (the length of its encoded value, not the password); a Forbidden line means your account cannot read the Secret, not that it is empty. Then delete the Job and run the helm upgrade again.
wait-db-schema logs Waiting for the Core database: ... (for example server closed the connection unexpectedly) and ends with Could not query the Core database within 900s, or with Core model schema did not appear within 900s	Exit 1, retried: up to three attempts of 15 minutes each. The import pod cannot reach the database at all, or the database has no Core schema yet. Under Istio, a core-postgres AuthorizationPolicy that does not admit the import closes the connection this way. This is not a deadline problem; raising activeDeadlineSeconds does not help.	Check that core-postgres is Running and Ready: kubectl -n "$NS" get pods. Check that the namespace's NetworkPolicies let pods labeled app.kubernetes.io/name=core-offline-model-import reach it: kubectl -n "$NS" get networkpolicy. On Istio, check that the core-postgres AuthorizationPolicy admits the import: kubectl -n "$NS" get authorizationpolicy core-postgres -o yaml (a Forbidden answer means your account cannot read it; ask the cluster owner). For a missing schema, check the core-db-init Job. Then delete the Job and run the helm upgrade again.
wait-model-registry exits 2, logging that the registry kept refusing this client	Exit 2: the Job fails at once, after the registry answered 403 or 404 for about a minute. 403: a proxy or policy in front of the registry refuses the import pod; the log quotes the start of its answer, which names it. 404: no registry API at that address. A registry that is only not ready yet is waited for, with its last status logged.	Check KAMIWAZA_MODEL_REGISTRY in the core-config ConfigMap: kubectl -n "$NS" get configmap core-config -o yaml | grep KAMIWAZA_MODEL_REGISTRY, and that the registry pod is Running. For 403, check the namespace's NetworkPolicies and, on Istio, its AuthorizationPolicy objects. Then delete the Job and run the helm upgrade again.
wait-model-registry logs Waiting for the internal model registry at ... and ends with did not become reachable in 120 seconds	Exit 1, retried. The import pod cannot reach the model registry, or it answers with a server error.	Check that the registry pod is Running and Ready (the command in the RegistrySpaceCheckFailed row), and that the namespace's NetworkPolicies, and on Istio its AuthorizationPolicy objects, let the import pod reach it. Then delete the Job and run the helm upgrade again.
stage-weights: ... has no /models/kamiwaza-model.json or ... /models/weights is missing or empty in the stage-weights log	Exit 2: the Job fails at once. The image is not a weights image built by kz-model-pack.sh, or holds no model files.	Check the entry names the right image; pack the model again if needed.
ImagePullBackOff on the stage-weights container	The node cannot pull the weights image.	Check the image is at <prefix>/images/<last segment of repository>:<tag> in your site registry, the $IMAGE of Step 3, and that it holds your nodes' platform. With digest filled in, the node pulls <prefix>/images/<last segment of repository>@<digest>, so the copy must also keep that digest; if your transfer changed it, set digest: "". Run the Step 3 check again.
The Job reaches its deadline (DeadlineExceeded in kubectl describe job)	The pull, copy and import took longer than activeDeadlineSeconds.	First read the logs: if wait-db-schema or wait-model-registry was still waiting, or timed out, use its row above; a longer deadline does not fix that. Otherwise raise activeDeadlineSeconds in the entry, then run helm upgrade again.

Clear the default configuration's file

Use this for a BindingConflict that says default config <ID> uses model_file <file ID>, not the manifest's primary_file .... Clearing the default configuration's file is all it needs: the next import points the default configuration at the pack's primary file, or its first weights file. No configuration or file is deleted.

You need a token for a user who can manage that model, as in Who can see the imported model (set KAMIWAZA_URL and TOKEN there). Set CONFIG_ID to the configuration ID the message names:

CONFIG_ID="<default config ID from the message>"   # replace
OUT=$(curl -sS -w '\n%{http_code}' "$KAMIWAZA_URL/api/model_configs/$CONFIG_ID" -H "Authorization: Bearer $TOKEN")
echo "HTTP $(printf '%s\n' "$OUT" | tail -n 1)"
BODY=$(printf '%s\n' "$OUT" | sed '$d' | python3 -c 'import json, sys
t = sys.stdin.read()
try:
    d = json.loads(t)
except ValueError:
    sys.exit("not JSON: " + t)
c = d if isinstance(d, dict) and d.get("id") else sys.exit("no configuration in: " + t)
c["m_file_id"] = None
print(json.dumps({k: c.get(k) for k in ("m_id", "m_file_id", "name", "default", "description", "config", "system_config")}))')
[ -n "$BODY" ] && printf '%s' "$BODY" | curl -sS -w '\nHTTP %{http_code}\n' -X PUT "$KAMIWAZA_URL/api/model_configs/$CONFIG_ID" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' --data-binary @-

It must print HTTP 200 twice. The same commands clear the file of any configuration, default or not.

• If you came here from Delete a catalog file row, go back and finish it: delete the file row before you import again.
• Otherwise delete the failed Job and run the helm upgrade again, as When the import fails shows.

A deployment of this model that was running keeps the file it started with; redeploy it after the import to use the new primary file. For any other answer, see the status list at the end of the next section.

Delete a catalog file row

The Models page cannot delete a single file of a model; use the API, with KAMIWAZA_URL and TOKEN set as in the previous section. Set FILE_ID to the model_file ID the BindingConflict message names, then find the model it belongs to and that model's repository ID:

FILE_ID="<model_file ID from the message>"   # replace
OUT=$(curl -sS -w '\n%{http_code}' "$KAMIWAZA_URL/api/model_files/$FILE_ID" -H "Authorization: Bearer $TOKEN")
echo "HTTP $(printf '%s\n' "$OUT" | tail -n 1)"
OWNER_MODEL_ID=$(printf '%s\n' "$OUT" | sed '$d' | python3 -c 'import json, sys
t = sys.stdin.read()
try:
    d = json.loads(t)
except ValueError:
    sys.exit("not JSON: " + t)
print(d["model_id"]) if isinstance(d, dict) and d.get("model_id") else sys.exit("no model ID in: " + t)')
echo "model: $OWNER_MODEL_ID"
curl -sS "$KAMIWAZA_URL/api/models/$OWNER_MODEL_ID" -H "Authorization: Bearer $TOKEN" | python3 -c 'import json, sys
t = sys.stdin.read()
try:
    d = json.loads(t)
except ValueError:
    sys.exit("not JSON: " + t)
print(d["repo_modelId"]) if isinstance(d, dict) and "repo_modelId" in d else sys.exit("no model in: " + t)'

It must print HTTP 200, a model ID and a repository ID. Compare the repository ID with the one you are importing: it decides what you may change below. Then list the model's configurations that use the file, since a file a configuration still uses cannot be deleted:

OUT=$(curl -sS -w '\n%{http_code}' "$KAMIWAZA_URL/api/model_configs/?model_id=$OWNER_MODEL_ID" -H "Authorization: Bearer $TOKEN")
echo "HTTP $(printf '%s\n' "$OUT" | tail -n 1)"
printf '%s\n' "$OUT" | sed '$d' | python3 -c 'import json, sys
t = sys.stdin.read()
try:
    d = json.loads(t)
except ValueError:
    sys.exit("not JSON: " + t)
isinstance(d, list) or sys.exit("not a list of configurations: " + t)
[print(c["id"], c.get("name"), "default" if c.get("default") else "") for c in d if str(c.get("m_file_id")) == sys.argv[1]]' "$FILE_ID"

HTTP 200 with no other line means nothing uses the file. Each line printed is a configuration that uses it, marked default for the model's default configuration.

If it is the repository ID you are importing, free the file in this model without deleting anything else: for each configuration printed, default or not, clear its file with the commands in Clear the default configuration's file, with CONFIG_ID set to its ID. Do not delete the default configuration; deleting it is refused (today with HTTP 500). After the import, the default configuration points at the pack's file again; point a configuration that is not the default at the file you want, as Point a configuration at another file shows, or delete it if you no longer need it (see the end of this section). Redeploy any running deployment of this model.

If it is another repository ID, the file belongs to a different model. Do not delete that model's configurations. Either point each configuration printed, default or not, at another file of that model, as Point a configuration at another file shows, or delete that whole model in Models, after you check that nothing you need uses it: deleting a model removes its configurations, its deployments and its file rows, this one included. After a whole-model delete, skip the file delete below and run the import again. Either way, check that model's deployments first.

Point a configuration at another file

GET /api/models/$OWNER_MODEL_ID lists the model's files under m_files. This works for a configuration of any model, the one you are importing too:

CONFIG_ID="<configuration ID printed above>"   # replace
NEW_FILE_ID="<ID of another file of that model>"   # replace
OUT=$(curl -sS -w '\n%{http_code}' "$KAMIWAZA_URL/api/model_configs/$CONFIG_ID" -H "Authorization: Bearer $TOKEN")
echo "HTTP $(printf '%s\n' "$OUT" | tail -n 1)"
BODY=$(printf '%s\n' "$OUT" | sed '$d' | python3 -c 'import json, sys
t = sys.stdin.read()
try:
    d = json.loads(t)
except ValueError:
    sys.exit("not JSON: " + t)
c = d if isinstance(d, dict) and d.get("id") else sys.exit("no configuration in: " + t)
c["m_file_id"] = sys.argv[1]
print(json.dumps({k: c.get(k) for k in ("m_id", "m_file_id", "name", "default", "description", "config", "system_config")}))' "$NEW_FILE_ID")
[ -n "$BODY" ] && printf '%s' "$BODY" | curl -sS -w '\nHTTP %{http_code}\n' -X PUT "$KAMIWAZA_URL/api/model_configs/$CONFIG_ID" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' --data-binary @-

Delete the file row

Go on only after each PUT above prints HTTP 200. Then delete the file row:

curl -sS -w '\nHTTP %{http_code}\n' -X DELETE "$KAMIWAZA_URL/api/model_files/$FILE_ID" -H "Authorization: Bearer $TOKEN"

HTTP 200 with Model file deleted successfully means it is gone; run the import again.

After the import, delete a configuration of this model that is not the default and that you no longer need:

CONFIG_ID="<configuration ID>"   # replace
curl -sS -w '\nHTTP %{http_code}\n' -X DELETE "$KAMIWAZA_URL/api/model_configs/$CONFIG_ID" -H "Authorization: Bearer $TOKEN"

For the calls in these two sections:

• 400: the file you pointed a configuration at is not a file of its model, or the configuration you sent is malformed.
• 401: the token is missing or expired.
• 403: your user cannot manage that model (an install without ReBAC).
• 404 with Model file ... not found: no such file; check FILE_ID. After a whole-model delete this is expected: the file row is already gone.
• 404 with Model configuration ... not found: no such configuration, or your user cannot see it; check CONFIG_ID.
• 404 with Model not found: on a ReBAC install, your user cannot manage that model, or it does not exist.
• 500 on the file delete almost always means a configuration still uses the file. If the listing above printed nothing and the delete still answers 500, contact Kamiwaza support with the file ID.

Allow the import to reach the API server

The import's free-space check reaches the Kubernetes API server. On a namespace whose NetworkPolicies block that, and whose kamiwaza-namespace chart does not add the rule, a namespace administrator can add it. Copy the API server addresses and ports from the policy the namespace chart wrote for the scheduler:

kubectl -n "$NS" get networkpolicy kamiwaza-core-scheduler-apiserver-egress -o yaml

Its egress: block, under spec:, lists every API server address range with its port: usually one rule for the service address on port 443 and one with several endpoint addresses, one per control-plane node, on port 6443. If the policy does not exist, ask the cluster owner for the API server's service and endpoint addresses. Write this file as import-egress.yaml, and replace the example egress: block with the one from the policy, copied whole and unchanged: every rule, and every ipBlock in each rule. A policy missing one of the endpoint addresses lets the check through only some of the time.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: offline-model-import-apiserver-egress
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: core-offline-model-import
  policyTypes:
    - Egress
  egress:            # replace from here with the policy's egress: block
    - to:
        - ipBlock:
            cidr: <service address range>
      ports:
        - protocol: TCP
          port: 443
    - to:
        - ipBlock:
            cidr: <endpoint address range 1>
        - ipBlock:
            cidr: <endpoint address range 2>
        - ipBlock:
            cidr: <endpoint address range 3>
      ports:
        - protocol: TCP
          port: 6443

The example only shows the shape; its placeholders are replaced by the block you copy.

Apply it, then delete the failed Job and run the helm upgrade again:

kubectl -n "$NS" apply -f import-egress.yaml

The chart refuses to render

The chart refuses to render, with a message that names the value, when:

• manifestSha256 or stagingSize is missing or malformed, or an entry has an unknown key
• a name is used twice, is longer than 36 characters, or is not a valid Kubernetes name
• the image has no repository, or neither a tag nor a digest
• activeDeadlineSeconds is below 300, or, when core.scheduler.installerHelmTimeout is set, not below it
• global.modelRegistry or core.scheduler.registryExternalHost is set, or registry.enabled is false. The import needs the platform's built-in model registry.
• core.scheduler.enabled is false.
• an AUTH_REBAC_DEFAULT_TENANT_ID, AUTH_REBAC_PUBLIC_MODEL_VIEWER_URN or AUTH_REBAC_AUTO_ASSIGN_OWNER entry in core.scheduler.extraEnv or core.scheduler.linuxNvidia.extraEnv has a value that uses $(...). Set the literal value, or use valueFrom.
• authentication is off (core.auth.enabled: false), or the Core image has neither a tag nor a digest. Release values and field kits pin one.

Alternative: an external HTTPS endpoint

If your site already runs an inference server, such as vLLM or another OpenAI-compatible service, at an HTTPS address inside the site, you can register it as an external model instead of importing the weights. The model then runs outside the platform. See OpenAI-Compatible Chat. An endpoint on a private address needs core.scheduler.allowInsecureModelEndpoints: true; see Plain HTTP and private endpoints. If a private CA signs its certificate, add that CA as TLS Trust for Private Certificate Authorities describes.