Skip to main content
Version: 1.2.1 (Latest)

Help & Fixes

This page provides resources for getting help with Kamiwaza and solutions to common issues you might encounter.

Getting Help​

If you have questions or run into issues, we're here to help:

We're committed to making your experience with Kamiwaza as smooth as possible.

Reporting Issues​

When reporting issues to our support team or community, please include:

  • Environment Details: OS version, Kubernetes distribution and version, hardware specs, and the Kamiwaza release version
  • Error Messages: Complete error text and stack traces
  • Steps to Reproduce: Detailed steps that led to the issue
  • Logs: Relevant Kubernetes pod logs, platform logs, and UI error details
  • Configuration: Any custom configuration or settings

This information helps us provide faster and more accurate solutions to your problems.

Common Issues and Fixes​

Installation Issues​

Docker GPU Error: Could Not Select Device Driver​

Problem: NVIDIA Container Runtime not found or misconfigured.

Solution:

  • Ensure NVIDIA drivers are properly installed
  • Install NVIDIA Container Toolkit
  • Verify Docker can access GPU devices

Port Already in Use​

Problem: Kamiwaza fails to start because required ports are occupied.

Solution:

  • Check what's running on ports 3000, 8000, 5432, 19530, 9090
  • Stop conflicting services or change Kamiwaza's port configuration
  • Use lsof -i :PORT_NUMBER to identify processes using specific ports

Insufficient System Resources​

Problem: Installation fails due to low disk space, RAM, or CPU cores.

Solution:

  • Ensure at least 16GB RAM available
  • Verify CPU supports required virtualization features

Model Deployment Issues​

Gated Model Downloads and Rate Limits​

Problem: Downloads fail for gated models (Llama, Mistral, etc.) or you encounter Hugging Face rate limit errors.

Symptoms:

  • "Access denied" or "401 Unauthorized" when downloading certain models
  • "Rate limit exceeded" errors during model downloads
  • Gated models appear in search but fail to download

Solution: Add your Hugging Face token to your Kamiwaza deployment configuration:

  1. Get a Hugging Face token from huggingface.co/settings/tokens (a read token is sufficient)
  2. For gated models, accept the license terms on the model's Hugging Face page
  3. Add the token to the Secret or deployment values used for your Kamiwaza environment: 
    HF_TOKEN="hf_your_token_here"
  4. Apply the updated configuration using your standard cluster release workflow and confirm the model-serving pods restart successfully.
  5. If you need setup details for secrets or deployment values, refer to the Configuration Reference and the Administrator Guide.

Model Deployment Failures​

Problem: Models fail to deploy or become unavailable.

Solutions:

  • Model not found: Ensure the model exists in your catalog or use Novice Mode
  • Checkpoint too large for VRAM: Choose a smaller/quantized variant (AWQ, MLX, GGUF) or reduce batch size
  • Service unavailable/port errors: Stop/Remove and redeploy the model
  • Outdated catalog: Refresh the Models page or restart the server

Performance Problems​

Problem: Slow responses or high resource usage.

Solutions:

  • Slow responses: Use faster models or quantized variants; reduce max tokens and context length
  • High memory/OOM: Lower batch size, context length, and KV cache; use lower-VRAM variants
  • Cold starts: First request may be slower; send a short warm-up prompt after deploy

SDK and API Issues​

Module Import Error​

Problem: ModuleNotFoundError: No module named 'kamiwaza_sdk' when using notebooks and Kamiwaza SDK.

Solution:

!pip uninstall -y kamiwaza kamiwaza-sdk
!pip install kamiwaza-sdk

Then restart the notebook kernel or Python session before importing kamiwaza_sdk again.

App Garden Issues​

App Not Showing Latest Version​

Problem: An app in App Garden isn't displaying the latest content or updates due to cache TTL.

Solution: Force a cache refresh by syncing remote templates:

# Step 1: Get an auth token
TOKEN=$(curl -sk -X POST "https://your-kamiwaza-instance/api/auth/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=admin&password=your-password" \
  | jq -r '.access_token')

# Step 2: Force a cache refresh
curl -sk -X POST "https://your-kamiwaza-instance/api/v1/apps/{app_id}/refresh" \
  -H "Authorization: Bearer ${TOKEN}"

General Troubleshooting Steps​

When encountering issues, follow these diagnostic steps:

  1. Check Service Status: Verify all Kamiwaza services are running
  2. Review Logs: Check container logs for specific error messages
  3. Verify Resources: Ensure sufficient CPU, RAM, and disk space
  4. Test Connectivity: Verify network connectivity between components
  5. Restart Services: Try stopping and restarting affected services
  6. Check Configuration: Verify configuration files and environment variables