Skip to main content

Help & Fixes

This page provides resources for getting help with Kamiwaza and solutions to common issues you might encounter.

Getting Help​

If you have questions or run into issues, use any of these channels:

Kamiwaza is committed to making your experience as smooth as possible.

Reporting Issues​

When reporting issues to the support team or community, please include:

  • Environment Details: OS version, Kubernetes distribution and version, hardware specs, and the Kamiwaza release version
  • Error Messages: Complete error text and stack traces
  • Steps to Reproduce: Detailed steps that led to the issue
  • Logs: Relevant Kubernetes pod logs, platform logs, and UI error details
  • Configuration: Any custom configuration or settings

This information enables faster and more accurate solutions.

Common Issues and Fixes​

Installation Issues​

Docker GPU Error: Could Not Select Device Driver​

Problem: NVIDIA Container Runtime not found or misconfigured.

Solution:

  • Ensure NVIDIA drivers are properly installed
  • Install NVIDIA Container Toolkit
  • Verify Docker can access GPU devices

Port Already in Use​

Problem: Kamiwaza fails to start because required ports are occupied.

Solution:

  • Check what's running on ports 3000, 8000, 5432, 19530, 9090
  • Stop conflicting services or change Kamiwaza's port configuration
  • Use lsof -i :PORT_NUMBER to identify processes using specific ports

Insufficient System Resources​

Problem: Installation fails due to low disk space, RAM, or CPU cores.

Solution:

  • Ensure at least 16GB RAM available
  • Verify CPU supports required virtualization features

Model Deployment Issues​

Gated Model Downloads and Rate Limits​

Problem: Downloads fail for gated models (Llama, Mistral, etc.) or you encounter Hugging Face rate limit errors.

Symptoms:

  • "Access denied" or "401 Unauthorized" when downloading certain models
  • "Rate limit exceeded" errors during model downloads
  • Gated models appear in search but fail to download

Solution: Add your Hugging Face token to your Kamiwaza deployment configuration:

  1. Get a Hugging Face token from huggingface.co/settings/tokens (a read token is sufficient)
  2. For gated models, accept the license terms on the model's Hugging Face page
  3. Add the token to the Secret or deployment values used for your Kamiwaza environment: 
    HF_TOKEN="hf_your_token_here"
  4. Apply the updated configuration using your standard cluster release workflow and confirm the model-serving pods restart successfully.
  5. If you need setup details for secrets or deployment values, refer to the Configuration Reference and the Administrator Guide.

Model Deployment Failures​

Problem: Models fail to deploy or become unavailable.

Solutions:

  • Model not found: Ensure the model exists in your catalog or use Novice Mode
  • Checkpoint too large for VRAM: Choose a smaller/quantized variant (AWQ, MLX, GGUF) or reduce batch size
  • Service unavailable/port errors: Stop/Remove and redeploy the model
  • Outdated catalog: Refresh the Models page or restart the server

Performance Problems​

Problem: Slow responses or high resource usage.

Solutions:

  • Slow responses: Use faster models or quantized variants; reduce max tokens and context length
  • High memory/OOM: Lower batch size, context length, and KV cache; use lower-VRAM variants
  • Cold starts: First request may be slower; send a short warm-up prompt after deploy

SDK and API Issues​

Module Import Error​

Problem: ModuleNotFoundError: No module named 'kamiwaza_sdk' when using notebooks and Kamiwaza SDK.

Solution:

!pip uninstall -y kamiwaza kamiwaza-sdk
!pip install kamiwaza-sdk

Then restart the notebook kernel or Python session before importing kamiwaza_sdk again.

App Garden Issues​

App Not Showing Latest Version​

Problem: An app in App Garden is not displaying the latest content or updates due to cache TTL.

Solution: Force a cache refresh by syncing remote templates:

# Step 1: Get an auth token
TOKEN=$(curl -sk -X POST "https://your-kamiwaza-instance/api/auth/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=admin&password=your-password" \
  | jq -r '.access_token')

# Step 2: Force a cache refresh
curl -sk -X POST "https://your-kamiwaza-instance/api/v1/apps/{app_id}/refresh" \
  -H "Authorization: Bearer ${TOKEN}"

Extension Platform Gateway Issues​

Gateway problems appear on the extension deployment's status (platform_transport_warnings in the API). See Platform Gateway for Extensions.

No GreyMatter Edge Service Found​

Problem: The deployment status shows:

PlatformGatewayUndiscovered: no GreyMatter edge Service (edge-ingress-*, annotation greymatter.io/managed-by=greymatter, port 10908) found in <namespace>; found <n>: <names>. Set KAMIWAZA_TRAEFIK_GATEWAY_HOST on Core or fix the edge Service.

The message can add a cause in parentheses after the namespace.

Solution: Fix the edge Service so that it matches the name, annotation and port in the message, or set KAMIWAZA_TRAEFIK_GATEWAY_HOST on Core to pick the edge.

Istio Ingress Gateway Service Not Found​

Problem: The deployment status shows:

PlatformGatewayUndiscovered: Istio ingress gateway Service <namespace>/<service> was not found, so extensions get no KAMIWAZA_PLATFORM_GATEWAY_URL and call the public origin. Set KAMIWAZA_INGRESS_GATEWAY_SERVICE_HOST on Core to the gateway Service host (global.ingress.gateway.serviceHostname).

Solution: Set global.ingress.gateway.serviceHostname to your gateway Service host and upgrade, or, on a non-chart install, set KAMIWAZA_INGRESS_GATEWAY_SERVICE_HOST on Core.

Extension Operator Cannot Confirm the Istio Gateway​

Problem: An extension's PlatformGateway condition is False with reason PlatformGatewayUnconfirmed. The operator sets no KAMIWAZA_PLATFORM_GATEWAY_URL, so unless a service sets its own, the extension uses the public origin. With a networkPolicy, it reaches that origin only when the policy allows external access.

Solution: Read the condition's message. If the operator is not allowed to read the gateway Service, the message names the Service and the permission to add. On a namespace-admin install or with serviceReadGrant: false under rbacMode: namespaced, the chart adds no Role, so a cluster administrator adds it; see Extension operator on Istio. If the host is not an in-cluster Service name, set global.ingress.gateway.serviceHostname to the form <service>.<namespace>.svc[.<cluster-domain>], or set extensionOperator.istioGatewayEgress.enabled: false. If the message names an API error, or says the Service has not been confirmed yet, the operator retries by itself.

Edge Service Has No Ready Endpoints​

Problem: The deployment status shows:

<GreyMatter|Istio> edge Service <namespace>/<service> has no ready endpoints on <port> (checked when this deployment was last applied); extensions cannot reach the platform until the edge pods are ready.

Solution: Get the edge or gateway pods Ready, then redeploy the extension so the check runs again.

core-api CrashLoopBackOff: KAMIWAZA_ORIGIN Is Unset​

Problem: After an upgrade to 1.4, core-api is in CrashLoopBackOff and logs:

FATAL: KAMIWAZA_ORIGIN is unset; extensions would be told the platform lives at https://localhost. Set global.domain (or core KAMIWAZA_ORIGIN) to the public https origin.

Solution: Set global.domain, or Core's KAMIWAZA_ORIGIN, to the public HTTPS origin. The chart sets it from global.domain, so this happens only when Core's configuration was edited by hand.

General Troubleshooting Steps​

When encountering issues, follow these diagnostic steps:

  1. Check Service Status: Verify all Kamiwaza services are running
  2. Review Logs: Check container logs for specific error messages
  3. Verify Resources: Ensure sufficient CPU, RAM, and disk space
  4. Test Connectivity: Verify network connectivity between components
  5. Restart Services: Try stopping and restarting affected services
  6. Check Configuration: Verify configuration files and environment variables