Developing Kamiwaza Extensions

This document describes how to build, test, and ship a Kamiwaza extension.

What Is An Extension?

A Kamiwaza extension is a containerized application, service, or tool that runs inside the Kamiwaza platform.

Type	What It Is	Example
App	A multi-service application with a frontend and backend, deployed to the App Garden	AI chatbot, document analyzer, recruitment tool
Service	A backend utility that other extensions can depend on	Vector database, shared cache
Tool	A server that exposes MCP functions and/or a REST API facade via the Tool Shed	Web scraper, code executor, API connector, document processor

Extensions are packaged as Docker images, described by metadata (kamiwaza.json), and deployed to Kubernetes via the Kamiwaza platform. Each extension lives in its own repository.

For Core version requirements and selecting compatible releases, see Version Compatibility.

Extension Requirements

Required artifacts:

kamiwaza.json — extension metadata (name, version, description, env defaults)
docker-compose.yml — service definitions
Dockerfile per service — container build instructions
README.md — documentation

Deployment requirements (enforced when deploying to the platform, not during local development):

No hardcoded host port bindings (ports: ["8000"] not "8000:8000")
No bind mounts (named volumes only)
Resource limits defined for all services (deploy.resources.limits)
Health endpoint at GET /health (apps)
MCP over streamable HTTP at path /mcp on port 8000 (tools). Kamiwaza and Kaizen call tools only this way. A server that uses the SSE transport (/sse) can deploy and show Running in the Tool Shed, but Kaizen cannot call it. See Building an MCP tool.

A health probe that your server answers is recommended for tools and services but not enforced; kz-ext validate in kamiwaza-sdk releases after 1.3.x warns a tool without one. See Per-service healthCheck.

Your local docker-compose.yml can and should use host port mappings, bind mounts, and whatever else makes local development convenient. When you run kz-ext dev, the CLI transforms your compose file to meet deployment requirements automatically — stripping host ports, replacing bind mounts with named volumes, and adding resource limits.

Conventions:

Image tags follow {version} format (no v prefix)
Tool names start with tool- or mcp-
Service names start with service-

Per-service `healthCheck`

When a service has a health probe, it is marked ready, and its pod kept running, only while the probe passes. Core uses the one healthCheck as the container's startup, liveness and readiness probe, so a probe that keeps failing restarts the container. Declare the probe for a service in kamiwaza.json under services.<service-name>.healthCheck. The service name is the one in docker-compose.yml:

{
  "services": {
    "tool": {
      "healthCheck": {
        "httpGet": { "path": "/health", "port": 8000 },
        "periodSeconds": 10,
        "timeoutSeconds": 5,
        "failureThreshold": 3
      }
    }
  }
}

You can also put the same block under x-kamiwaza.healthCheck on the service in docker-compose.yml. If both are present, kz-ext dev uses the one in kamiwaza.json.

Where you declare it decides which deployments get it. kz-ext dev reads both places. A deployment from a catalog card (the App Garden or Tool Shed card that kz-ext dev or kz-ext publish writes, for example a deploy into a workroom) carries only the compose file, so it gets only the x-kamiwaza.healthCheck probe, or, when that is absent, a probe Core derives from the service's Compose healthcheck: (a CMD list test that runs curl against a URL becomes httpGet; anything else, including CMD-SHELL, becomes exec in the container). kz-ext dev ignores Compose healthcheck:, so an exec healthcheck that needs a program the image lacks, such as curl, fails only in card deployments. A probe that must apply everywhere belongs in docker-compose.yml. A deployment with no probe is marked ready as soon as its container starts.

The block uses Kubernetes probe shapes. It declares exactly one of:

Shape	Fields	Probe passes when
`httpGet`	`path`, `port`, optional `scheme`, `host`, `httpHeaders`	`GET path` returns a 2xx or 3xx status
`tcpSocket`	`port`, optional `host`	the port accepts a TCP connection
`exec`	`command` (a non-empty list)	the command exits 0 inside the container

Optional timing fields, all in seconds or counts: initialDelaySeconds, periodSeconds, timeoutSeconds, failureThreshold, and startPeriod. Core refuses any other key, including successThreshold. startPeriod must be 0 or at least periodSeconds; Core fills an omitted startPeriod with 60, so a periodSeconds above 60 needs an explicit startPeriod. kz-ext validate checks the shape and the periodSeconds/startPeriod rule before you deploy, but it does not range-check the other timings (Core requires timeoutSeconds of at least 1, for example), and it accepts a few keys Core refuses (successThreshold, terminationGracePeriodSeconds and a grpc probe), so leave those out.

Using services.<name>.healthCheck, or x-kamiwaza.healthCheck in docker-compose.yml, needs kz_ext_version with a lower bound of at least 0.2.0, the CLI release that added them. A new scaffold already declares a higher floor.

When you declare no healthCheck, kz-ext dev adds a default probe to each service that exposes a port, using the first row that matches. Core adds no probe of its own beyond translating a Compose healthcheck: (above), so declare a probe, in docker-compose.yml, for anything you publish.

Service	Default probe from `kz-ext dev`
A service named `frontend` with Node.js environment hints (the app scaffold's Next.js frontend)	`exec`: a Node script that calls the app's `/health` route
Primary service of a tool	kamiwaza-sdk 1.3.x: `httpGet /sse`, which fails on every streamable-HTTP server, so declare a probe. Later releases: `tcpSocket` on the service's port.
Other `frontend` services	`httpGet /`
Primary service of a service extension	`httpGet /`
Any other service	`httpGet /health`

A probe that keeps failing restarts the pod, and the extension stays at Provisioning with no error in the Tool Shed or App Garden. See Troubleshooting MCP tools.

Getting Started

Install the CLI

pip install kamiwaza-sdk

This installs the kz-ext command as part of the Kamiwaza SDK package. Build and deployment logic lives in this package rather than in copied template files in your repo.

Because CLI updates can change how your extension is built and deployed, kamiwaza.json declares the CLI version range your extension is compatible with:

{
  "kz_ext_version": ">=0.2.2,<1.0.0"
}

kz_ext_version is the version of the kz-ext CLI, not of the kamiwaza-sdk package that installs it. kamiwaza-sdk 1.3.x installs kz-ext 0.2.2; run kz-ext --version to see yours. Where this guide says a behavior differs in kamiwaza-sdk releases after 1.3.x, check the package with pip show kamiwaza-sdk: the kz-ext version alone does not tell them apart. kz-ext create writes a range that starts at the CLI that created the extension.

The CLI checks this on every command and warns (or errors) if there is a mismatch. You can also run a compatibility check explicitly:

kz-ext doctor

This verifies CLI version compatibility, checks that required tools are installed (Docker, etc.), validates your Kamiwaza connection, and reports any issues.

Optional extras for publishing and conversion:

pip install kamiwaza-sdk[publish]   # Adds boto3 for kz-ext publish
pip install kamiwaza-sdk[convert]   # Adds anthropic + openai for kz-ext convert
pip install kamiwaza-sdk[all]       # Both

Connect to a Kamiwaza instance

Most extensions need Kamiwaza platform features (model access, auth, workrooms). Set up your connection first:

kz-ext login https://your-cluster.kamiwaza.test/api
# kz-ext login https://dev-host.example.com/api --no-verify-ssl
#   turns off TLS verification: use it only for a development instance with
#   a self-signed certificate, never a shared or production one

The URL must end in /api. Without it, the CLI fails; with a username and password, for example, it says it received an HTML page and that base_url must end with /api.

This authenticates via the Kamiwaza SDK, stores a PAT locally, and configures the CLI to target that instance for both local development and deployment. You only do this once per instance.

Create an extension

mkdir my-app
cd my-app
kz-ext create --type app --name my-app

This creates a production-ready app with a styled frontend and fully-wired backend:

my-app/
├── kamiwaza.json
├── docker-compose.yml
├── frontend/
│   ├── Dockerfile
│   ├── start.mjs
│   ├── package.json
│   ├── tailwind.config.ts
│   └── src/
│       ├── app/
│       │   ├── layout.tsx          # SessionProvider + Google Fonts
│       │   ├── page.tsx            # AuthGuard + dashboard with session/models
│       │   ├── globals.css         # Tailwind + Kamiwaza dark theme
│       │   ├── providers.tsx       # SessionProvider wrapper
│       │   ├── logged-out/         # Logout redirect page
│       │   ├── session/            # Session proxy route
│       │   ├── auth/               # Auth proxy routes
│       │   └── api/[...path]/      # Catch-all backend proxy
│       └── components/
│           └── NavBar.tsx
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── app/
│       └── main.py                 # Session router, auth, models, chat
└── README.md

No template infrastructure, no scripts/ directory. The kz-ext CLI provides all tooling externally.

kz-ext create --type tool --name tool-my-tool creates a one-file MCP server instead, including src/server.py, requirements.txt, a Dockerfile, docker-compose.yml, kamiwaza.json and a README.md. See Building an MCP tool.

The generated code comes pre-wired with the Kamiwaza extension libraries:

Backend (requirements.txt): kamiwaza-extensions-lib — auth middleware (require_auth), model client (get_model_client), model discovery (list_available_models), session endpoints (create_session_router)
Frontend (package.json): @kamiwaza-ai/extensions-lib — SessionProvider, AuthGuard, useSession(), useModels(), proxy utilities (createProxyHandlers)
Frontend styling: Tailwind CSS with a Kamiwaza dark theme (custom kw-* color tokens, CRT scanline effects, component classes)

These are published packages (pip install / npm install), not copied source files. The health endpoint, session router, auth dependencies, and model endpoints are already configured in the generated main.py. The frontend includes proxy routes for session, auth, and all backend API calls.

The Development Loop

Local development

Your first feedback loop runs containers locally via docker-compose, with the Kamiwaza SDK pointing at your connected instance for platform features:

kz-ext dev local

This runs docker-compose up --build with environment variables configured so that:

Your backend can call Kamiwaza-managed models via the OpenAI-compatible endpoint
KAMIWAZA_USE_AUTH is set to false so the session router returns an anonymous identity
The SDK can make platform API calls (model discovery, etc.) via your connected instance

You get a working app at http://localhost:3000 and iterate with normal development tools.

Port auto-detection

If ports are already in use, the CLI automatically remaps to the next available port:

$ kz-ext dev local
Port 3000 in use — remapping frontend to 3001
Port 8000 in use — remapping backend to 8001
frontend: http://localhost:3001
backend:  http://localhost:8001

This lets you run multiple extensions simultaneously without port conflicts. The internal Docker networking (frontend → backend) is unaffected — only the host-mapped ports change.

Hot reload

Your local docker-compose.yml can use host port mappings, bind mounts, and dev server entrypoints — none of the deployment restrictions apply locally. To get hot reload, use the standard mechanisms for your stack:

Frontend (Next.js/Vite): Bind-mount your src/ directory and use the dev server entrypoint. The generated Dockerfile includes a dev stage for this.
Backend (FastAPI): Bind-mount your app/ directory and run with uvicorn --reload.

These bind mounts and dev entrypoints exist only in your local docker-compose.yml. When you run kz-ext dev for deployed development, the CLI strips them and uses immutable container images instead.

For extensions that don't need Kamiwaza integration (e.g., a standalone tool that will be accessed by apps but doesn't call Kamiwaza APIs itself), kz-ext dev local still works — it just runs docker-compose without the SDK configuration.

Local auth bridge (`--auth`)

By default, kz-ext dev local runs with KAMIWAZA_USE_AUTH=false so the session router returns an anonymous identity. To exercise the real ForwardAuth flow against your connected instance — useful when iterating on auth-gated routes, role checks, or workroom scoping — pass --auth:

kz-ext dev local --auth

This stands up a local sidecar that injects the same identity headers (X-User-Id, X-User-Email, X-Workroom-Id, etc.) the platform would inject in a deployed extension, sourced from your kz-ext login session. X-Workroom-Id defaults to the global-workroom sentinel (ffffffff-ffff-ffff-ffff-ffffffffffff); set --workroom <uuid> to scope to a specific workroom. The bridge also configures Docker host-gateway routing so the local backend can reach a host-loopback Kamiwaza ingress (e.g. https://kamiwaza.test) without manual host.docker.internal plumbing.

--auth is most useful for app scaffolds that exercise authenticated browser flows. Tool and service scaffolds typically don't need it.

Deployed development

When you need to test in the full platform environment — real ingress, real auth flows, real workroom scoping:

kz-ext dev

This:

Builds Docker images with a unique dev tag (e.g., 1.0.0-dev+abc1234.1711900800). No version bump needed.
Pushes images to the container registry.
Creates or updates the extension deployment through the platform's extensions API. The deployment is named <name>-dev-<id>, and runs in No Workroom.
Waits for the rollout to complete.
Prints the URL.
Writes a development card to your instance's catalog, so the extension also appears in the App Garden or Tool Shed for deploying into a workroom. The output confirms it with ✓ Catalog overlay: <name> .... It is skipped, with a note saying why, with --no-push, --service or remote secret references, or when the platform has no overlay support; in kamiwaza-sdk releases after 1.3.x, also when the platform's managed catalog already selects that name. When the write fails it prints a warning and the deployment still completes. Remove the card with kz-ext dev --unload.

On the first deploy, a Request failed: {"detail":"Extension '...' not found"} line is expected; the CLI logs it before it creates the extension.

Building my-app (1.0.0-dev+abc1234.1711900800)...
  ✓ backend image built
  ✓ frontend image built
Pushing to your-cluster.kamiwaza.test...
  ✓ extension updated
  ✓ rollout complete (18s)

my-app is running at:
https://your-cluster.kamiwaza.test/runtime/apps/my-app-dev-a1b2c3

Image registry and CPU architecture

kz-ext dev pushes images to a registry and the cluster pulls them from there:

Set KAMIWAZA_REGISTRY to the registry the cluster can pull from, as host, host:port or either with a path (for example ghcr.io/acme or registry.example.com:5000/team). Without KAMIWAZA_REGISTRY, for a remote instance the CLI pushes to registry.<your-kamiwaza-host>, which usually doesn't exist. kz-ext doctor shows the registry the CLI would use. docker login to the registry first: kz-ext dev pushes with your engine's own login.
If this machine reaches the registry at another address than the cluster's nodes, set that address, with the same path, in KAMIWAZA_PUSH_REGISTRY. For a registry on plain HTTP, also set KAMIWAZA_REGISTRY_INSECURE=1 and allow the registry in your engine (Docker's insecure-registries, or the Podman machine's registry settings); kz-ext dev checks this before it builds.
Images are named <KAMIWAZA_REGISTRY>/<name>-<service>:<dev tag>, so a path in KAMIWAZA_REGISTRY is the namespace, such as an organization on ghcr.io. You can instead keep KAMIWAZA_REGISTRY=ghcr.io and set "image_basename": "acme/tool-hello" in kamiwaza.json (giving ghcr.io/acme/tool-hello-tool:...), or give the service a registry-qualified image: in docker-compose.yml (for example ghcr.io/acme/tool-hello), whose repository path kz-ext dev keeps. An unqualified image: such as acme/tool-hello gets the default name.
If the registry is private, the cluster needs a pull secret. A Kamiwaza administrator adds it to the Helm value global.imagePullSecrets; extension pods use that list.
Build for the cluster's CPU architecture. From an Apple Silicon Mac to an x86-64 cluster, set DOCKER_DEFAULT_PLATFORM=linux/amd64. A pod that logs exec format error was built for the wrong architecture.

export KAMIWAZA_REGISTRY=ghcr.io/acme   # registry and namespace the cluster pulls from
docker login ghcr.io
export DOCKER_DEFAULT_PLATFORM=linux/amd64   # Apple Silicon to an x86-64 cluster only
kz-ext dev

Iterating

The deployed dev loop is:

Change code.
Run kz-ext dev.
Test at the printed URL.
Repeat.

Each run builds only what changed (Docker layer caching), pushes a uniquely-tagged image, and updates the running extension. The Kubernetes operator handles the rolling update. No version bumps. No release builds. No manual redeploy.

Rollout characteristics

The PATCH-update path uses the standard Kubernetes rolling-update strategy on the underlying Deployment. In-flight requests at the moment a pod is replaced may be dropped — under default container settings this typically registers as a small spike of connection reset / 502 errors during the rollout window (low-loss, but not strictly zero-downtime). For most dev iteration this is invisible; for live user traffic, design idempotent retries on the client or pin pods with a lifecycle.preStop hook in your service spec.

Options:

kz-ext dev --no-build         # Push only (image already built)
kz-ext dev --service backend  # Rebuild and update one service only
kz-ext dev --revision custom  # Custom image tag

Local SDK development (`--sdk-repo`)

When iterating on the runtime libraries themselves (kamiwaza-extensions-lib or @kamiwaza-ai/extensions-lib), use --sdk-repo to override the published packages with your local SDK source:

# Via CLI flag
kz-ext dev local --sdk-repo ~/repos/kamiwaza-sdk

This mounts the SDK repo into containers and overrides the published package installs:

Python backend: pip install -e /sdk/kamiwaza_extensions_lib --no-deps replaces the PyPI package
TypeScript frontend: npm pack + npm install from the local kamiwaza-ai-extensions-lib/ replaces the npm package

The overrides are ephemeral — your extension repo's docker-compose.yml and Dockerfiles are never modified.

Repo-local config

Instead of passing --sdk-repo every time, create a gitignored config file:

# .kz-ext/local.yaml
sdk_repo: /Users/you/repos/kamiwaza-sdk
runtime_libs:
  python: local       # override Python lib (default when sdk_repo is set)
  typescript: local   # override TypeScript lib
build_typescript: true  # auto-build TS package if dist/ is missing or stale

Then just run kz-ext dev local — it reads the config automatically. CLI flags override the config file.

You can override only one library:

sdk_repo: /Users/you/repos/kamiwaza-sdk
runtime_libs:
  python: local
  typescript: published   # keep using npm version

Remote deploy with local SDK

The --sdk-repo flag also works with kz-ext dev for remote deployment:

kz-ext dev --sdk-repo ~/repos/kamiwaza-sdk

Instead of volume mounts, this bakes the local runtime libraries into the Docker images at build time using Docker BuildKit additional build contexts. The resulting images contain your local lib code and are pushed to the cluster normally.

This requires Docker BuildKit (Docker 20.10+). The CLI checks and fails with a clear message if BuildKit is not available.

Validating the override

kz-ext doctor validates the SDK override configuration:

$ kz-ext doctor

  ✓ Python 3.11.2
  ✓ Docker 24.0.7
  ✓ Docker Compose v2.23.0
  ✓ Connection: dev-cluster (healthy)

  SDK Override (.kz-ext/local.yaml):
  ✓ SDK repo exists: /Users/you/repos/kamiwaza-sdk
  ✓ Python runtime lib: kamiwaza_extensions_lib/ found
  ✓ TypeScript runtime lib: kamiwaza-ai-extensions-lib/ found
  ✓ TypeScript dist/: built, up to date
  ✓ Template contract: requirements.txt ✓  package.json ✓

Authentication

How it works

You do not implement auth protocols. No login pages, no token validation, no OAuth flows, no session storage. The platform handles all of that at the ingress layer before requests reach your code.

What you do is wire up the provided libraries to read user identity from platform-injected headers and, on the frontend, wrap your app in the session/auth components. If you use the generated scaffolding from kz-ext create, this wiring is already done for you.

The platform flow:

Browser → supported ingress → platform authorization → Your Extension
                                                          ↓
                                            Request includes identity headers
                                            set by the platform auth service

Your extension receives identity headers on every authenticated request. The exact headers are part of the platform runtime contract (see below).

When an unauthenticated user tries to access your extension, the platform handles the redirect automatically:

Direct URL access (user navigates to your extension in the browser):

The platform detects the unauthenticated browser request and redirects to the login page with a return URL.
After login, the user is sent back to the original extension URL.
Your extension is not involved — this happens entirely at the ingress layer.

SPA session expiry (user's session expires while using your app):

API calls from your frontend will receive 401 responses.
The AuthGuard component handles this automatically — it detects session errors, redirects to login, and restores the session when the user returns.

The scaffolded frontend wraps your app in SessionProvider and AuthGuard from the start:

// This is generated for you by kz-ext create
import { SessionProvider, AuthGuard } from '@kamiwaza-ai/extensions-lib/client';

export default function RootLayout({ children }) {
  return (
    <SessionProvider>
      <AuthGuard>
        {children}
      </AuthGuard>
    </SessionProvider>
  );
}

This gives you:

Automatic redirect to login when the session expires or is missing
Session state available via useSession() hook
Session expiry countdown tracking
Configurable public routes that skip auth (e.g., /logged-out)
Loading and redirecting states while auth is being resolved

import { useSession } from '@kamiwaza-ai/extensions-lib/client';

function Dashboard() {
  const { session, loading, logout } = useSession();

  if (loading) return <Loading />;

  return (
    <div>
      <h1>Welcome, {session.name}</h1>
      <button onClick={() => logout()}>Logout</button>
    </div>
  );
}

You do not need to build a login page, handle OAuth callbacks, manage tokens, or implement session storage. If you use the generated scaffolding, SessionProvider + AuthGuard + create_session_router() are already wired up. If you're integrating into an existing app, you add these yourself — but you still never implement auth protocols.

Reading user identity

Python (FastAPI)

The kamiwaza-extensions-lib package extracts identity from the platform's forwarded auth headers:

from kamiwaza_extensions_lib import Identity, get_identity, require_auth, create_session_router
from fastapi import Depends, Request

# Add session endpoints to your app (/session, /auth/login-url, /auth/logout)
app.include_router(create_session_router())

# Require authentication on a route
@app.get("/api/documents")
async def list_documents(identity: Identity = Depends(require_auth)):
    # identity.user_id, identity.email, identity.name, identity.roles, identity.workroom_id
    return await get_documents_for_user(identity.user_id)

# Or get identity without requiring auth (returns unauthenticated identity if no session)
@app.get("/api/public")
async def public_endpoint(request: Request):
    identity = await get_identity(request)
    if identity.is_authenticated:
        return {"greeting": f"Hello, {identity.name}"}
    return {"greeting": "Hello, guest"}

# Require a specific role
from kamiwaza_extensions_lib import require_role

@app.get("/api/admin")
async def admin_only(identity: Identity = Depends(require_role("admin"))):
    return {"admin": identity.email}

create_session_router() adds three endpoints that the frontend's SessionProvider relies on:

GET /session — returns the current user's session info
GET /auth/login-url — builds a login redirect URL
POST /auth/logout — logs out and returns redirect URLs

TypeScript (Next.js)

On the server side, extract identity from request headers:

import { extractIdentity } from '@kamiwaza-ai/extensions-lib/server';
import { headers } from 'next/headers';

export default async function Page() {
  const headersList = await headers();
  const identity = extractIdentity(headersList);
  // identity.userId, identity.email, identity.name, identity.resolvedUuid
  return <h1>Welcome, {identity?.name}</h1>;
}

In API route handlers, use the proxy utilities to forward requests to your backend with auth headers preserved:

import { createProxyHandlers } from '@kamiwaza-ai/extensions-lib/server';

const { GET, POST, PUT, DELETE } = createProxyHandlers({
  target: process.env.BACKEND_URL || 'http://backend:8000',
});

export { GET, POST, PUT, DELETE };

Calling Kamiwaza APIs from your backend

When your extension's backend needs to call Kamiwaza platform APIs (e.g., model discovery, data catalog), forward the user's auth headers:

from kamiwaza_extensions_lib import KamiwazaExtClient, forward_auth_headers

@app.get("/api/available-models")
async def list_models(request: Request, identity: Identity = Depends(require_auth)):
    client = KamiwazaExtClient.from_env()
    models = await client.get_models(request.headers)  # Forwards auth context
    return models

For background tasks that outlive the original request (e.g., long-running document processing), use a service account key:

import httpx
import os

# Uses KAMIWAZA_API_KEY (injected by platform) — no user context needed
async def background_task():
    headers = {"Authorization": f"Bearer {os.getenv('KAMIWAZA_API_KEY')}"}
    async with httpx.AsyncClient(verify=False) as client:
        resp = await client.get(f"{os.getenv('KAMIWAZA_API_URL')}/models/", headers=headers)
        return resp.json()

Local development

During kz-ext dev local, the CLI injects environment variables (KAMIWAZA_API_URL, KAMIWAZA_USE_AUTH, etc.) so that the same library code works without a real auth flow. When KAMIWAZA_USE_AUTH is false, create_session_router() returns an anonymous identity and the AuthGuard allows access without authentication. Your code doesn't need to change between local and deployed modes.

Workroom context

Extensions are workroom-scoped. The workroom_id is available on the Identity dataclass:

@app.get("/api/projects")
async def list_projects(request: Request, identity: Identity = Depends(require_auth)):
    return await get_projects(workroom_id=identity.workroom_id)

You do not need to validate workroom membership — the platform enforces this at the ingress layer before requests reach your extension.

Model Integration

How models work with extensions

Kamiwaza manages model deployments. Your extension declares what kind of model it needs, and the platform provides access. Extensions don't deploy, manage, or health-check models — the platform does that.

Declaring model requirements

In kamiwaza.json, declare what your extension needs:

{
  "preferred_model_type": "reasoning",
  "env_defaults": {
    "MODEL_CAPABILITIES": "chat,function_calling"
  }
}

Using models in your code

The extensions library provides a get_model_client() helper that returns an OpenAI-compatible async client:

from kamiwaza_extensions_lib import get_model_client, require_auth, Identity
from fastapi import Depends, Request

@app.post("/api/chat")
async def chat(request: Request, messages: list[dict], identity: Identity = Depends(require_auth)):
    client = await get_model_client(request)
    response = await client.chat.completions.create(
        model="auto",
        messages=messages,
    )
    return response

Letting users select a model

For apps that want to offer model choice (e.g., a chatbot where users pick which model to talk to):

from kamiwaza_extensions_lib import list_available_models

@app.get("/api/models")
async def available_models(request: Request, identity: Identity = Depends(require_auth)):
    models = await list_available_models(request)
    return [{"id": m.id, "name": m.name, "type": m.model_type} for m in models]

@app.post("/api/chat")
async def chat(request: Request, model_id: str, messages: list[dict], identity: Identity = Depends(require_auth)):
    client = await get_model_client(request)
    response = await client.chat.completions.create(
        model=model_id,
        messages=messages,
    )
    return response

Streaming

For streaming responses (the common case for chat apps):

from starlette.responses import StreamingResponse

@app.post("/api/chat/stream")
async def chat_stream(request: Request, messages: list[dict], identity: Identity = Depends(require_auth)):
    client = await get_model_client(request)
    stream = await client.chat.completions.create(
        model="auto",
        messages=messages,
        stream=True,
    )

    async def generate():
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                yield f"data: {chunk.model_dump_json()}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

Local development

During kz-ext dev local, the KamiwazaExtClient.from_env() reads KAMIWAZA_ENDPOINT (model access) and KAMIWAZA_API_URL (API access) from the environment. The CLI configures these to point at your connected Kamiwaza instance. You use real models during local development — no mocks or stubs.

Runtime Environment

When your extension is deployed, the platform injects environment variables and HTTP headers that your code can use.

Platform runtime contract (stable)

These are injected by the extension operator and can be relied on by extensions:

Environment variables:

Variable	Description	Example
`KAMIWAZA_DEPLOYMENT_ID`	Unique identifier for this deployment	`my-app-a1b2c3d4`
`KAMIWAZA_APP_NAME`	Extension name from kamiwaza.json	`my-app`
`KAMIWAZA_APP_PORT`	Port the primary service is exposed on	`8000`
`KAMIWAZA_APP_PATH`	Ingress path prefix	`/runtime/apps/my-app-a1b2c3d4`
`KAMIWAZA_APP_URL`	Full public URL for the extension	`https://cluster.test/runtime/apps/my-app-a1b2c3d4`
`KAMIWAZA_API_URL`	Internal Kamiwaza API endpoint	`http://core-api.kamiwaza.svc.cluster.local:7777/api`
`KAMIWAZA_PUBLIC_API_URL`	Public Kamiwaza API endpoint	`https://cluster.test/api`
`KAMIWAZA_ORIGIN`	Platform origin for CORS	`https://cluster.test`
`KAMIWAZA_USE_AUTH`	Whether auth is enabled	`true`

Platform gateway variables

Starting in Kamiwaza 1.4, the platform sets one in-cluster address for itself in each extension's runtime environment. Call the platform through it, not through the public origin.

Variable	Set by	Description
`KAMIWAZA_PLATFORM_GATEWAY_URL`	Platform	The in-cluster platform gateway. An origin only (`scheme://host[:port]`, no path; one trailing slash is tolerated). Append `/api/...` yourself.
`KAMIWAZA_GATEWAY_CA_FILE`	Platform	Path to the CA bundle that verifies the gateway, when the platform publishes one. Otherwise, verify with the image's system roots.
`KAMIWAZA_GATEWAY_TLS_SERVER_NAME`	Platform	TLS server name for the gateway. GreyMatter only.
`KAMIWAZA_PLATFORM_GATEWAY_CONTRACT`	Extension	Set to `1` in any service's env to declare that every service in the extension reads `KAMIWAZA_PLATFORM_GATEWAY_URL` with its two companions. Needed only on GreyMatter, where an extension that does not declare it gets the deprecated `KAMIWAZA_TRAEFIK_GATEWAY_HOST` instead of the URL.

Always read the URL together with KAMIWAZA_GATEWAY_CA_FILE and KAMIWAZA_GATEWAY_TLS_SERVER_NAME, each when set. kamiwaza_extensions_lib.platform_gateway (kamiwaza_extensions_lib 0.6.3) has the variable names and a resolver.

These variables are deprecated, still set through 1.4.x, and removed after it:

Deprecated	Use instead
`KAMIWAZA_INTERNAL_GATEWAY_URL`	`KAMIWAZA_PLATFORM_GATEWAY_URL`
`KAMIWAZA_TRAEFIK_GATEWAY_HOST`	`KAMIWAZA_PLATFORM_GATEWAY_URL`, `KAMIWAZA_GATEWAY_CA_FILE` and `KAMIWAZA_GATEWAY_TLS_SERVER_NAME`
`KAMIWAZA_INTERNAL_ORIGIN`	`KAMIWAZA_PLATFORM_GATEWAY_URL`. It is still set, but it is not a gateway; do not use it as one.

How each routing provider sets the URL is described in Platform Gateway for Extensions.

Identity headers (set by ForwardAuth on authenticated requests):

Header	Description
`X-User-Id`	User's unique identifier
`X-User-Email`	User's email address
`X-User-Name`	User's display name
`X-User-Roles`	Comma-separated list of roles
`X-Workroom-Id`	Workroom the user is operating in
`X-Request-Id`	Unique request identifier

Headers with limited documentation (present in the platform but not yet formally part of the extension contract — use via the libraries rather than reading directly):

Header / Variable	Notes
`X-Auth-Token`	Raw JWT token; prefer forwarding via `forward_auth_headers()`
`KAMIWAZA_API_KEY`	Service account key; injection mechanism TBD

Model variables

Variable	Description	Example
`KAMIWAZA_MODEL_PORT`	Port for the paired model endpoint	`8080`
`KAMIWAZA_MODEL_URL`	Full URL for the paired model	(cluster-internal URL)
`KAMIWAZA_ENDPOINT`	Alias for model endpoint (legacy, may be deprecated)	(cluster-internal URL)

Template variables in kamiwaza.json

In your kamiwaza.json env_defaults, you can use template variables that are resolved at deployment time:

Variable	Resolves To
`{app_port}`	The assigned port for your extension
`{model_port}`	The assigned port for the paired model
`{deployment_id}`	The deployment identifier
`{app_name}`	The extension name

Example:

{
  "env_defaults": {
    "PUBLIC_URL": "https://localhost:{app_port}",
    "CALLBACK_URL": "https://localhost:{app_port}/api/auth/callback"
  }
}

Deployment inspection

For deployment inspection and day-to-day operations, prefer the supported SDK and CLI surfaces instead of hardcoding raw platform endpoints in your extension code. In practice that means leaning on commands such as kz-ext status and the shared runtime libraries rather than treating the platform's internal API paths as part of your extension contract.

Workroom-scoped runtime sessions

If your extension launches a runtime inside a workroom, use the generated auth and session scaffolding plus the shared SDK helpers instead of wiring token handling by hand.

Keep SessionProvider, AuthGuard, and create_session_router() in place for browser and session lifecycle behavior.
Use KamiwazaExtClient.from_env() and forward_auth_headers() when your backend calls Kamiwaza APIs on behalf of the active user.
Treat any runtime launch token as opaque and short-lived. Do not parse it, log it, persist it outside the active runtime session, or pass it in a URL query string.
If a downstream platform call returns 401, fetch a fresh runtime launch token through the supported platform flow and retry once rather than looping retries.

For the public platform contract around workroom-scoped runtimes, membership enforcement, and collaboration streams, see Workroom Runtime Contract.

Writing portable code

Use the library's config and client classes rather than reading environment variables directly. This ensures your code works in both local and deployed modes:

from kamiwaza_extensions_lib import AuthConfig, KamiwazaExtClient

# AuthConfig reads from environment automatically
config = AuthConfig.from_env()
# config.api_url      — KAMIWAZA_API_URL
# config.use_auth     — KAMIWAZA_USE_AUTH (bool)

# KamiwazaExtClient reads endpoints from environment
client = KamiwazaExtClient.from_env()
# client.api_base     — KAMIWAZA_API_URL
# client.openai_base  — KAMIWAZA_ENDPOINT

The CLI configures these environment variables for both kz-ext dev local and kz-ext dev, so the same code works in both modes.

Debugging

Logs

Stream logs from your deployed extension:

kz-ext logs                    # All services
kz-ext logs --service backend  # One service only
kz-ext logs --follow           # Stream continuously

Shell access

Exec into a running container:

kz-ext shell                    # Primary service
kz-ext shell --service backend  # Specific service

Status

Check the deployment state:

kz-ext status

Shows:

Extension name, version, and dev revision
Per-service status (image tag, ready replicas, restart count)
Ingress URL

It does not show pod events such as probe failures; see Troubleshooting MCP tools for reading them.

Port forwarding

Access a service directly without going through the platform ingress:

kz-ext port-forward --service backend --port 8000
# Backend is now available at localhost:8000

Useful for testing backend APIs directly, connecting a debugger, or inspecting a database. Auto-detects the service and port if not specified.

Bringing an Existing App to Kamiwaza

If you have an existing containerized application and want to run it as a Kamiwaza extension:

kz-ext convert /path/to/existing-app

The conversion tool uses an AI agent to analyze and modify your existing code:

Analyze the existing Dockerfiles, compose files, and application code.
Check requirements — verify the app meets extension requirements (no host ports, no bind mounts, has health endpoint, etc.) and report what needs to change.
Generate kamiwaza.json — infer metadata from existing configuration.
Modify files — add kamiwaza-extensions-lib to Python backends, @kamiwaza-ai/extensions-lib to Node.js frontends, wire in health endpoints, auth middleware, session router, and resource limits.
Report — show what was created, what was modified, and what needs manual attention.

All file modifications are tracked by git — review with git diff and revert anything you disagree with.

The AI agent supports multiple LLM providers, picked in this order:

Claude CLI / Codex CLI (subscription auth): If claude or codex is on your PATH and already authenticated (e.g. you've run claude login), kz-ext convert reuses that session — no API key needed. This is the most convenient path if you already use either CLI day-to-day.
OpenAI-compatible (API key): Set OPENAI_API_KEY. Set OPENAI_BASE_URL for Kamiwaza, vLLM, Ollama, or any OpenAI-compatible endpoint.
Anthropic (API key): Set ANTHROPIC_API_KEY.
No credentials: Falls back to basic kamiwaza.json generation with a compatibility report — useful for a quick read on what would need to change without mutating files.

The agent also handles monorepo layouts (it discovers per-app Dockerfile / package.json / requirements.txt under a configurable subdirectory), and can vendor template files via copy rather than symlink when needed for distroless dev-local builds.

Preview without modifying files:

kz-ext convert /path/to/existing-app --dry-run

After conversion:

cd /path/to/existing-app
git diff                # Review changes
kz-ext validate         # Verify everything passes
kz-ext dev local        # Test locally
kz-ext dev              # Deploy to Kamiwaza

Updating Scaffolds When the Template Evolves

When the kz-ext CLI ships a newer template (new generated files, refreshed proxy routes, updated AGENTS.md, etc.), existing scaffolded extensions can pick up the changes without a full rescaffold:

kz-ext update

update reconciles only template-owned files — the wiring scaffolded by kz-ext create. Author-owned files (your src/, your routes, anything you've added or substantially edited) are never touched. Each template-owned file gets one of three strategies:

overwrite — replaced unconditionally; a .orig backup is written if your on-disk copy differs from the new template.
preserve_if_modified — replaced only if the on-disk copy is identical to the previous template render. If you've edited it, the file is skipped (or, in interactive mode, you see a unified diff and choose apply / keep).
merge — reserved for future smart-merge; v1 behaves as preserve_if_modified.

Modes:

kz-ext update                     # interactive — prompts on every conflict
kz-ext update --dry-run           # print planned changes only; no writes
kz-ext update --force             # apply all template-owned updates, write .orig on conflicts
kz-ext update --non-interactive   # fail with non-zero exit on the first conflict (for CI)
kz-ext update --bootstrap         # adopt current state as baseline for older scaffolds

The set of template-owned files is enumerated explicitly in code (not glob-based) — there are no surprises about what update can or can't touch. Use --dry-run first when in doubt; review the proposed diff with git diff after running.

Validation

Before opening a PR:

kz-ext validate

This checks:

kamiwaza.json is well-formed and complete
Docker images build successfully
Compose file is App Garden-compatible
No host port bindings, no bind mounts
Resource limits defined
Health endpoint exists (apps)
services.<name>.healthCheck blocks have one valid probe shape and a valid periodSeconds/startPeriod
For tools (kamiwaza-sdk releases after 1.3.x), these errors:
- a server file whose only transport is transport="sse";
- code that imports mcp.server.fastmcp while requirements.txt lets pip install mcp 2.x: an mcp line with no cap below 2.0, or no mcp line and a fastmcp line that allows 4.x.
And these warnings:
- a server file that runs SSE beside streamable HTTP, or a FastMCP run() with no transport (stdio, which never listens);
- code that imports the standalone fastmcp package while requirements.txt names neither fastmcp nor fastmcp-slim;
- for code that imports mcp.server.fastmcp: a fastmcp line below 4 with no mcp cap, no mcp or fastmcp line at all, an mcp line under an environment marker, or a requirements.txt it cannot read (-r, -c, pip options);
- a tool that declares no healthCheck, probes /sse, or probes a path no Python source mentions.
Preview image path is valid (if specified)
With the unreleased compatibility-aware SDK changes, kamiwaza_version uses the supported numeric Core requirement grammar (if specified)

Testing beyond structural validation is the developer's responsibility. Use whatever test framework fits your stack.

Isolated catalog testing (low-level publisher)

Only the shared extension workflow writes the dev-info bucket, and only from successful develop builds. Do not use the low-level SDK publisher to write that bucket or a production catalog. For isolated compatibility testing, use a dedicated non-production bucket and image namespace:

kz-ext config publish-profile sandbox \
  --registry ghcr.io/my-org/sandbox \
  --catalog-endpoint https://my-account.r2.cloudflarestorage.com \
  --catalog-bucket extensions-dev-sandbox \
  --catalog-credentials aws-profile:my-sandbox-profile

kz-ext publish --stage sandbox --catalog-schema compat-v1

This low-level path exercises SDK publication behavior only; it does not validate a Kajiya-owned manifest or authenticated promotion receipt. On a real publish, SDK-built images are resolved to their pushed image digests and written as digest-qualified catalog references. A dry run does not perform that registry resolution or prove the final digest. External or prebuilt images must already carry digest pins. --revision supplies the revision tag and catalog revision field; it does not pin an image digest. Canonical release publication requires owner and promotion evidence, and the receipt-backed release route is not yet available.

Replace the example namespace and profile with sandbox-only values. The S3 endpoint may be shared, but the bucket and credentials must be scoped to the sandbox bucket. The sandbox credentials must not be able to write dev-info or production buckets. Registry credentials must be limited to the sandbox image namespace and unable to push release images. Prefer the explicit sandbox AWS profile shown here over env; when using env, every ambient AWS_* value and AWS_PROFILE must be sandbox-scoped. A catalog_prefix is only a key prefix, not an access-control boundary.

Each saved publish profile specifies:

registry — where Docker images are pushed (GHCR, ECR, private registry, etc.)
catalog-endpoint — S3-compatible endpoint for the extension catalog
catalog-bucket — bucket name for the catalog
catalog-credentials — how to authenticate (aws-profile:<name>, env, etc.)

For isolated CI experiments, the low-level publisher also accepts environment overrides. Each of these five KZ_PUBLISH_* values overrides that field from the named profile, including an empty value. A stray value can redirect a publish away from the sandbox. Set all five explicitly and keep them scoped to the sandbox; never set the bucket to dev-info or a production catalog:

export KZ_PUBLISH_REGISTRY=ghcr.io/my-org/sandbox
export KZ_PUBLISH_CATALOG_ENDPOINT=https://my-account.r2.cloudflarestorage.com
export KZ_PUBLISH_CATALOG_BUCKET=extensions-dev-sandbox
export KZ_PUBLISH_CATALOG_PREFIX=eng-12421
export KZ_PUBLISH_CATALOG_CREDENTIALS=aws-profile:my-sandbox-profile

Profiles can be stored at the user level (~/.kamiwaza/profiles/) or at the repo level (.kz-ext/profiles/ — useful for team-shared configs). Repo-level profiles override user-level profiles by name.

Do not commit repo-level profiles that target the shared dev-info bucket or production storage. Keep committed examples and shared configs scoped to an isolated sandbox. Registry login credentials used by the publisher must also be unable to push to production/release image namespaces.

You do not need publish profiles for the dev loop — kz-ext dev only needs the Kamiwaza instance connection from kz-ext login.

Profile JSON shape

Behind the CLI, each profile is just a JSON file at ~/.kamiwaza/profiles/<name>.json (file mode 0600). You can edit it directly if you'd rather not re-run the CLI:

{
  "name": "sandbox",
  "registry": "ghcr.io/my-org/sandbox",
  "catalog_endpoint": "https://my-account.r2.cloudflarestorage.com",
  "catalog_bucket": "extensions-dev-sandbox",
  "catalog_credentials": "aws-profile:my-sandbox-profile",
  "catalog_prefix": "",
  "created_at": "2026-05-06T00:00:00.000000+00:00"
}

The catalog_credentials field accepts:

aws-profile:<name> — kz-ext runs boto3.Session(profile_name="<name>"), which reads ~/.aws/credentials. This is the most reliable option for Cloudflare R2 (where you've already configured an r2.cloudflarestorage.com profile in ~/.aws/config with region = auto).
env — kz-ext runs boto3.Session() with no args. boto3 then resolves credentials from the standard chain: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY exported in the shell, or AWS_PROFILE if set. Note: kz-ext publish does not load .env files — these env vars must be exported in the shell that runs the command (or set in your CI environment). If you have a repo-level .env with custom variable names like AWS_PROFILE_DEV=…, those are not picked up; either export them with eval $(grep '^AWS_' .env | xargs) first, or switch the profile to aws-profile:<name> (recommended).
sso — reserved for Cloudflare SSO; not yet implemented for kz-ext publish. Use aws-profile:<name> with an SSO-configured AWS profile instead.

Endpoint and credentials must match

The catalog_endpoint and the credentials you point at have to be for the same provider. R2 keys talk to R2 endpoints; AWS S3 keys talk to AWS S3 endpoints. Common signs of a mismatch:

Unable to locate credentials — boto3 can't find any credentials at all. Either ~/.aws/credentials has no [<profile>] section by that name, or catalog_credentials is "env" and no AWS env vars are exported. Fix: confirm aws --profile <name> sts get-caller-identity works from your shell, then make sure your kz-ext profile points at that AWS profile.
An error occurred (InvalidAccessKeyId) when calling the GetObject operation — boto3 found credentials, but the provider rejected them. This usually means the endpoint and the keys are for different providers (e.g., R2 keys hitting s3.amazonaws.com, or AWS keys hitting r2.cloudflarestorage.com). Fix: align catalog_endpoint with whichever provider holds the keys.

A quick way to check whether an AWS profile is configured for R2: grep -A2 '\[profile <name>\]' ~/.aws/config — R2 profiles typically have region = auto.

Preparing for a future release

The 1.3.0 canonical release catalog route is not available yet. The shared extension workflow publishes development discovery from successful develop builds but does not publish canonical release catalogs. The SDK's direct kz-ext publish command does not validate Kajiya-owned manifests or authenticated promotion receipts, so it is not a release command. Publication must wait for the Kajiya receipt-backed route.

When your extension is ready for a wider audience, you can prepare its manifest and validate it. Do not use a sandbox publication as the canonical release or reserve its immutable version for production. Until the receipt-backed release route exists, a version bump is optional:

Optional: bump the version in kamiwaza.json when preparing a sandbox-only compatibility experiment:

{
  "version": "1.1.0"
}

Validate the extension. This prepares it for a future release; it does not publish it:

kz-ext validate

Do not use kz-ext publish with production credentials. --force does not authorize or repair a release.

CI/CD

kz-ext works in automated pipelines. Use PAT-based authentication (no interactive login):

# In your CI environment
export KAMIWAZA_API_KEY=pat-...
export KAMIWAZA_URL=https://your-cluster.kamiwaza.test/api

# Validate
kz-ext validate

The shared workflow keeps development discovery separate from release publication. A successful development catalog update is not a promotion receipt and does not qualify a release.

Extension Anatomy

kamiwaza.json

Every extension has a kamiwaza.json at its root. This is the single source of truth for the extension's identity and version.

{
  "name": "my-app",
  "version": "1.0.0",
  "source_type": "kamiwaza",
  "description": "A brief description of what this extension does",
  "risk_tier": 1,
  "verified": false,
  "env_defaults": {
    "PUBLIC_URL": "https://localhost:{app_port}",
    "CALLBACK_URL": "https://localhost:{app_port}/api/auth/callback"
  },
  "preferred_model_type": "reasoning",
  "preview_image": "images/preview.png",
  "kamiwaza_version": ">=0.8.0"
}

docker-compose.yml

Your development compose file. This is what you use for kz-ext dev local and what the tooling transforms for App Garden deployment.

services:
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    environment:
      - BACKEND_URL=http://backend:8000

  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    ports:
      - "8000:8000"
    volumes:
      - backend_data:/app/data
    environment:
      - OPENAI_BASE_URL=${KAMIWAZA_ENDPOINT:-http://host.docker.internal:8080}
      - OPENAI_API_KEY=${OPENAI_API_KEY:-not-needed-kamiwaza}

volumes:
  backend_data:

When deploying to Kamiwaza, the CLI automatically transforms this for the cluster: host ports removed, bind mounts removed, build contexts replaced with image references, tags synced from kamiwaza.json. References to host.docker.internal (used for local dev to reach the host machine) are replaced by cluster-internal service URLs that the platform injects automatically via environment variables.

Key Patterns

Health endpoint

Apps must expose a health check:

@app.get("/health")
async def health_check():
    return {"status": "healthy"}

Frontend-backend communication

In Docker, services communicate by service name:

const BACKEND_URL = process.env.BACKEND_URL || 'http://backend:8000';

export async function GET(request: NextRequest) {
  const response = await fetch(`${BACKEND_URL}${request.nextUrl.pathname}`);
  return NextResponse.json(await response.json());
}

MCP tools

Tools serve MCP over streamable HTTP at /mcp on port 8000, plus a readiness route. The working minimum, which kz-ext create --type tool generates in kamiwaza-sdk releases after 1.3.x (1.3.x scaffolds need the edits in step 2):

from mcp.server.fastmcp import FastMCP
from starlette.requests import Request
from starlette.responses import JSONResponse

# host/port go on the constructor, not on run(). 0.0.0.0 is required in a container.
mcp = FastMCP("tool-hello", host="0.0.0.0", port=8000)


@mcp.custom_route("/health", methods=["GET"])
async def health(request: Request) -> JSONResponse:
    return JSONResponse({"status": "ok"})


@mcp.tool()
def hello(name: str = "world") -> str:
    """Say hello — replace this with your tool implementation."""
    return f"Hello, {name}!"


if __name__ == "__main__":
    mcp.run(transport="streamable-http")   # serves /mcp; never "sse"

with mcp==1.30.0 in requirements.txt. See Building an MCP tool for the full walkthrough.

Building an MCP tool

This walkthrough builds a tool with one function, hello(name), deploys it to the Tool Shed, and calls it from Kaizen.

1. Scaffold

pip install kamiwaza-sdk
kz-ext login https://kamiwaza.example.com/api   # your host; the URL must end in /api
mkdir tool-hello && cd tool-hello
kz-ext create --type tool --name tool-hello

2. Check the three files that make it work

With kamiwaza-sdk 1.3.x and earlier, the scaffold gets all three of these wrong, and the tool never works: it crashes on start, serves the wrong transport, or never passes its probe. Edit them to match. Later releases generate them correctly, so just check them. kz-ext update does not fix an existing tool for you: it leaves an edited requirements.txt or src/server.py alone, and it never adds the probe block to kamiwaza.json.

requirements.txt: pin the MCP SDK below 2.x. mcp 2.x removed mcp.server.fastmcp. An mcp requirement without an upper bound below 2.0 now installs 2.x. So can a fastmcp requirement without an mcp line: fastmcp 4.x requires mcp 2.x, and some earlier fastmcp releases (for example 2.14.0) leave mcp uncapped. The container then crashes on start with ModuleNotFoundError: No module named 'mcp.server.fastmcp'.

mcp==1.30.0
kamiwaza-extensions-lib>=0.6,<0.7

The exact pin keeps builds reproducible. When this guide was written, 1.30.0 was the newest 1.x release and had no published advisories; check the mcp advisories before you pin. Any 1.x from 1.30 on works (mcp>=1.30,<2). starlette, which src/server.py imports for the /health route, comes with mcp.

src/server.py: serve streamable HTTP at /mcp, and answer /health. Make it match the server in MCP tools above, keeping the name you scaffolded in FastMCP("..."). transport="sse" serves /sse, which Kaizen never calls.

kamiwaza.json: declare the health probe. Add a top-level services key and leave the other generated fields as they are. The example shows only name, type and the new key:

{
    "name": "tool-hello",
    "type": "tool",
    "services": {
        "tool": {
            "healthCheck": {
                "httpGet": { "path": "/health", "port": 8000 }
            }
        }
    }
}

This is where the scaffold declares it, and kz-ext dev uses it. A copy deployed from the Tool Shed card (step 5) carries only the compose file and runs without a probe, so it is marked ready as soon as its container starts. To probe those copies too, declare the same block under x-kamiwaza.healthCheck on the tool service in docker-compose.yml instead.

3. Test locally

docker build -t tool-hello:local .
docker run --rm -d --name tool-hello -p 8000:8000 tool-hello:local
curl -s --retry 10 --retry-all-errors --retry-delay 1 localhost:8000/health   # {"status":"ok"}

# MCP initialize: expect HTTP 200 and a JSON-RPC "result"
curl -si --max-time 10 -X POST localhost:8000/mcp \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}'
docker rm -f tool-hello

The scaffold's server also returns an mcp-session-id header; a server built with stateless_http=True does not, and that is fine. --retry-all-errors needs curl 7.71 or later.

4. Validate and deploy

A registry such as ghcr.io needs an owner namespace in the image path. Put it in KAMIWAZA_REGISTRY, as below, so the image becomes ghcr.io/acme/tool-hello-tool:<tag>. A registry that accepts root-level repositories, such as a cluster-local one, needs nothing.

kz-ext validate
export KAMIWAZA_REGISTRY=ghcr.io/acme   # registry and namespace your cluster can pull from
docker login ghcr.io
kz-ext dev
kz-ext status          # Phase: Running   tool  yes  1/1

See Image registry and CPU architecture for the registry, pull secret and DOCKER_DEFAULT_PLATFORM. In the UI, Tool Shed → Your Tool Servers shows the server Running with a URL like https://<host>/runtime/tools/tool-hello-dev-<id>.

5. Put the tool where Kaizen can see it

Kaizen sees tool servers in its own workroom and in No Workroom, which every workroom can see unless the platform runs with WORKROOM_ALLOW_GLOBAL_ACCESS=false. On such an install, deploy the tool into the workroom itself.

Where Kaizen runs	Where the tool must run
No Workroom	No Workroom. `kz-ext dev` deploys here.
Workroom X	Workroom X, or No Workroom

To deploy into a workroom: enter the workroom, open Tool Shed, and click Deploy on the tool's card. kz-ext dev writes the card; if it is missing, check the kz-ext dev output for the ✓ Catalog overlay line or the reason it was skipped, then rerun kz-ext dev without --service or --no-push.

Kaizen must be deployed from the App Garden by an administrator. Kaizen deployed any other way (with kz-ext dev, by a non-admin, or with a restricted token) does not receive the user's session, so it cannot list tools, and MCP is missing from its Admin → Tools page.

6. Enable the tool in Kaizen

Steps 6 and 7 were tested with Kaizen 0.8.4; menus may differ in other Kaizen versions.

Kaizen 0.8 enables no MCP tools by default.

Open Kaizen from the same workroom.
Click your name at the bottom left, then Admin area, then Tools.
Expand Kamiwaza → MCP.
Tick the tool's surface. Kaizen names it <deployment> · tool-hello, for example tool-hello-dev · tool-hello for the kz-ext dev deployment tool-hello-dev-8d041e (Kaizen drops the trailing id), or just tool-hello when the deployment has the server's name.

7. Call it

Start a new session in Kaizen and send: Use the hello tool to greet "Acme". Reply with exactly what the tool returned. The reply shows one tool-call step, then Hello, Acme!.

To clean up, use Tool Shed → Your Tool Servers → Stop. kz-ext dev --unload also removes the Tool Shed card; stop running copies first.

Troubleshooting MCP tools

Deployment names below look like tool-hello-dev-8d041e. Extension pods run in the kamiwaza namespace on single-namespace installs and in kamiwaza-extensions on multi-namespace ones; adjust -n to match. kz-ext logs finds the namespace itself.

Stuck at Provisioning

Provisioning means the pod has never become ready. Neither the Tool Shed nor kz-ext status shows why, and the platform never times it out.

Read the container log first:

kz-ext logs --tail 50
# or: kubectl -n kamiwaza logs deploy/<deployment>-tool -c tool --tail 50

You see	Fix
`ModuleNotFoundError: No module named 'mcp.server.fastmcp'. This is mcp 2.x ...`	Pin `mcp==1.30.0` in `requirements.txt`.
`exec format error`	Rebuild for the cluster's architecture: `DOCKER_DEFAULT_PLATFORM=linux/amd64` (or `linux/arm64`).
`GET /sse HTTP/1.1" 404`, repeating, then `Shutting down`	The probe hits `/sse`. Add the `/health` route and the `healthCheck` block.

Then read the pod events:

kubectl -n kamiwaza describe pod -l extensions.kamiwaza.io/deployment-id=<deployment> | sed -n '/^Events:/,$p'

Event	Cause and fix
`Startup probe failed: HTTP probe failed with statuscode: 404`	The probe path doesn't exist. Point `healthCheck` at a route the server answers.
`Startup probe failed: ... context deadline exceeded`	The probe times out on an SSE stream, which never ends. Serve streamable HTTP and probe `/health`.
`ErrImagePull` / `ImagePullBackOff`	The cluster can't pull the image. Set `KAMIWAZA_REGISTRY` to a registry it can reach, and add a pull secret for a private one.
`FailedScheduling ... Insufficient cpu` (or `memory`)	The cluster is full. Stop unused apps or tools, or add capacity.

If kz-ext dev fails at push time, you probably didn't set KAMIWAZA_REGISTRY, so the CLI tried registry.<your-host>.

Running in the Tool Shed but not in Kaizen

Each of these failures is silent in Kaizen: the tool is simply missing. Check them in order.

Wrong transport or path (the most common cause). Run the initialize call against the platform route with your token. kz-ext login keeps the token in ~/.kamiwaza/connections/<connection>/token.json; replace default below if you use a named connection.
```
export KAMIWAZA_TOKEN=$(python3 -c 'import json, os; print(json.load(open(os.path.expanduser("~/.kamiwaza/connections/default/token.json")))["access_token"])')
curl -si --max-time 10 -X POST "https://kamiwaza.example.com/runtime/tools/tool-hello-dev-8d041e/mcp" \
  -H "Authorization: Bearer $KAMIWAZA_TOKEN" \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}'
```
Use your host and deployment name, and add -k only for a development instance with a self-signed certificate. The token authorizes the whole platform API: run this on your own machine, not a shared host, and unset KAMIWAZA_TOKEN afterwards. The token file layout belongs to the SDK and may change between releases. A working server answers HTTP 200 with a JSON-RPC result. A 404 means it serves SSE or another path. A 307 or 308 means it answers at /mcp/ instead of /mcp; Kaizen does not follow the redirect. A 401 or 403 means the token is wrong or expired; run kz-ext login again. Fix a transport or path problem with mcp.run(transport="streamable-http") and the default path.
Not enabled in Kaizen. Tick it under Admin area → Tools → Kamiwaza → MCP.
Different workrooms. Inside the workroom you open Kaizen from, Tool Shed → Your Tool Servers must list the server. If not, deploy it there or to No Workroom.
Kaizen not deployed from the App Garden by an administrator. MCP is then missing from Admin → Tools. An administrator can confirm with kubectl -n kamiwaza logs deploy/core-api -c core-api | grep "forward/validate/api/extensions": a 401 Unauthorized there confirms it. Redeploy Kaizen from the App Garden as an administrator.
See why Kaizen dropped the tool. Kaizen 0.8 and later log MCP discovery failed on every endpoint for server '<name>': kubectl -n kamiwaza logs deploy/<kaizen-deployment>-api-backend | grep "MCP discovery". Kaizen 0.7 logs it only at DEBUG level.

Command Reference

Command	What It Does
`kz-ext login <url>/api`	Connect to a Kamiwaza instance (the URL must end in `/api`)
`kz-ext create --type app --name my-app`	Create a new extension (`--type app`, `tool` or `service`; `connector` in kamiwaza-sdk releases after 1.3.x)
`kz-ext dev local`	Run locally with docker-compose + SDK (auto-detects port conflicts)
`kz-ext dev local --auth`	Run locally with the ForwardAuth identity bridge enabled
`kz-ext dev local --sdk-repo <path>`	Run locally with local SDK runtime libs (volume mount)
`kz-ext dev`	Build, push, and deploy to Kamiwaza
`kz-ext dev --sdk-repo <path>`	Build with local SDK libs baked into images, then deploy
`kz-ext dev --no-build`	Push and deploy (skip build)
`kz-ext dev --service backend`	Rebuild one service only
`kz-ext dev --no-push`	Build and deploy without pushing (the cluster must already reach the image; skips the catalog card)
`kz-ext dev --unload`	Remove the development catalog card
`kz-ext update`	Reconcile a scaffold against the current template (interactive by default; supports `--dry-run` / `--force` / `--non-interactive` / `--bootstrap`)
`kz-ext bump <major\|minor\|patch>`	Bump the version in `kamiwaza.json`
`kz-ext validate`	Check extension requirements
`kz-ext doctor`	Check CLI compatibility, tools, connection, SDK override config, and per-failure-class auth hints
`kz-ext convert /path`	AI-powered conversion of existing app to extension
`kz-ext config publish-profile <name>`	Configure a named publish profile
`kz-ext config publish-profile --list`	List all publish profiles
`kz-ext publish --stage <profile> --catalog-schema compat-v1`	Low-level publish to an isolated compat-v1 catalog; without this option the schema defaults to legacy v3
`kz-ext publish --stage <profile> --catalog-schema compat-v1 --dry-run`	Preview; does not resolve live registry digests
`kz-ext publish --stage <profile> --catalog-schema compat-v1 --revision <sha>`	Set revision tag and catalog revision; it does not provide a digest pin
`kz-ext status`	Show deployment status and URL
`kz-ext logs [--service name] [--follow] [--tail N]`	Stream logs from deployed extension (needs `kubectl` access to the cluster)
`kz-ext shell [--service name]`	Exec into a running container
`kz-ext port-forward [--service name] [--port N]`	Forward a port locally

The low-level publisher is not a canonical release route. Do not use it with production credentials, point it at the shared dev-info bucket, or treat a catalog write as a Kajiya promotion receipt. Canonical releases require authenticated promotion evidence.

FAQ

Do I need to bump the version to test a change?

No. During development, kz-ext dev generates a unique image tag on every build using the git SHA and a timestamp. Version bumps are only needed when you release.

Can I develop without a Kamiwaza instance?

Partially. kz-ext dev local runs your containers with plain docker-compose, but most extensions need Kamiwaza features (model access, auth). Connect to a Kamiwaza instance for the best local development experience.

How do I iterate on the runtime libraries?

Use kz-ext dev local --sdk-repo ~/repos/kamiwaza-sdk or create a .kz-ext/local.yaml config. This overrides the published packages with your local SDK source — no compose surgery needed. See the Local SDK development section.

How do I update my tooling?

pip install --upgrade kamiwaza-sdk                     # CLI + SDK
pip install --upgrade kamiwaza-extensions-lib           # Python runtime library
npm update @kamiwaza-ai/extensions-lib                  # TypeScript runtime library

After upgrading, run kz-ext doctor to verify compatibility with your extension's declared kz_ext_version range. The intelligence lives in installed packages, not copied files in your repo — but a small set of generated wiring (proxy routes, AGENTS.md, etc.) is template-owned. To pick up template changes shipped with a newer CLI, run kz-ext update (see Updating Scaffolds When the Template Evolves).

How do I handle authentication?

You don't. The platform authenticates requests before they reach your extension and injects identity headers. Use the SDK middleware to read user identity. See the Authentication section.

How do I access models?

Use the get_model_client() helper from kamiwaza-extensions-lib. It returns an OpenAI-compatible async client. See the Model Integration section.

What is the difference between `kz-ext dev` and `kz-ext publish`?

kz-ext dev is the fast inner loop — no release build or release catalog artifacts, unique development tags, and immediate deployment. The only catalog entry it writes is a development card on your own instance (see Deployed development). kz-ext publish is a low-level SDK command for an isolated non-production catalog. The shared workflow alone writes development discovery. Neither command authorizes a canonical release. Release publication requires Kajiya's authenticated promotion-receipt path, which is not available yet.

How do I convert an existing app?

Run kz-ext convert /path/to/app. The AI agent analyzes your existing code, generates kamiwaza.json, and wires in SDK integration. All changes are git-tracked. See the Bringing an Existing App to Kamiwaza section.

Does `kz-ext` work in CI?

Yes. Use KAMIWAZA_API_KEY and KAMIWAZA_URL environment variables for headless authentication. See the CI/CD section.

What Is An Extension?​

Extension Requirements​

Per-service healthCheck​

Getting Started​

Install the CLI​

Connect to a Kamiwaza instance​

Create an extension​

The Development Loop​

Local development​

Port auto-detection​

Hot reload​

Local auth bridge (--auth)​

Deployed development​

Image registry and CPU architecture​

Iterating​

Local SDK development (--sdk-repo)​

Repo-local config​

Remote deploy with local SDK​

Validating the override​

Authentication​

How it works​

Login redirects​

Reading user identity​

Python (FastAPI)​

TypeScript (Next.js)​

Calling Kamiwaza APIs from your backend​

Local development​

Workroom context​

Model Integration​

How models work with extensions​

Declaring model requirements​

Using models in your code​

Letting users select a model​

Streaming​

Local development​

Runtime Environment​

Platform runtime contract (stable)​

Platform gateway variables​

Model variables​

Template variables in kamiwaza.json​

Deployment inspection​

Workroom-scoped runtime sessions​

Writing portable code​

Debugging​

Logs​

Shell access​

Status​

Port forwarding​

Bringing an Existing App to Kamiwaza​

Updating Scaffolds When the Template Evolves​

Validation​

Isolated catalog testing (low-level publisher)​

Profile JSON shape​

Endpoint and credentials must match​

Preparing for a future release​

CI/CD​

Extension Anatomy​

kamiwaza.json​

docker-compose.yml​

Key Patterns​

Health endpoint​

Frontend-backend communication​

MCP tools​

Building an MCP tool​

1. Scaffold​

2. Check the three files that make it work​

3. Test locally​

4. Validate and deploy​

5. Put the tool where Kaizen can see it​

6. Enable the tool in Kaizen​

7. Call it​

Troubleshooting MCP tools​

Stuck at Provisioning​

Running in the Tool Shed but not in Kaizen​

Command Reference​

FAQ​

Do I need to bump the version to test a change?​

Can I develop without a Kamiwaza instance?​

How do I iterate on the runtime libraries?​

How do I update my tooling?​

What Is An Extension?

Extension Requirements

Per-service `healthCheck`

Getting Started

Install the CLI

Connect to a Kamiwaza instance

Create an extension

The Development Loop

Local development

Port auto-detection

Hot reload

Local auth bridge (`--auth`)

Deployed development

Image registry and CPU architecture

Iterating

Local SDK development (`--sdk-repo`)

Repo-local config

Remote deploy with local SDK

Validating the override

Authentication

How it works

Login redirects

Reading user identity

Python (FastAPI)

TypeScript (Next.js)

Calling Kamiwaza APIs from your backend

Local development

Workroom context

Model Integration

How models work with extensions

Declaring model requirements

Using models in your code

Letting users select a model

Streaming

Local development

Runtime Environment

Platform runtime contract (stable)

Platform gateway variables

Model variables

Template variables in kamiwaza.json

Deployment inspection

Workroom-scoped runtime sessions

Writing portable code

Debugging

Logs

Shell access

Status

Port forwarding

Bringing an Existing App to Kamiwaza

Updating Scaffolds When the Template Evolves

Validation

Isolated catalog testing (low-level publisher)

Profile JSON shape

Endpoint and credentials must match

Preparing for a future release

CI/CD

Extension Anatomy

kamiwaza.json

docker-compose.yml

Key Patterns

Health endpoint

Frontend-backend communication

MCP tools

Building an MCP tool

1. Scaffold

2. Check the three files that make it work

3. Test locally

4. Validate and deploy

5. Put the tool where Kaizen can see it

6. Enable the tool in Kaizen

7. Call it

Troubleshooting MCP tools

Stuck at Provisioning

Running in the Tool Shed but not in Kaizen

Command Reference

FAQ

Do I need to bump the version to test a change?

Can I develop without a Kamiwaza instance?

How do I iterate on the runtime libraries?

How do I update my tooling?