Deployment topology
Kaizen ships as a set of cooperating services declared in one canonical compose topology. This page lists every process, its port, its dependencies, and its health contract.
The services
| Service | Port | Health contract | Depends on |
|---|---|---|---|
| postgres | 5432 | pg_isready -U postgres -d tomo, 5s / 10 retries | — |
| valkey | 6379 | valkey-cli ping, 5s / 10 retries | — |
| api-backend | 8000 | GET /healthz, 10s / 6 retries, 15m start period | postgres, sandbox-controller |
| background-worker | 9100 (health, metrics, Pi callbacks) | GET /health, 10s interval / 10s start period | postgres, valkey |
| document-worker | 9100 (health, metrics, Pi callbacks) | GET /health, 10s interval / 10s start period | postgres, valkey |
| seaweedfs | 8333 | GET /healthz, 10s / 12 retries | — |
| document-extractor | 9998 | GET /version, 10s / 12 retries, 3m start period | — |
| frontend | 8080 | kaizen-healthcheck, 30s / 3 retries | api-backend |
| agent-runtime | 8090 | — | api-backend |
| package-tool-runner | 8090 | — | — |
| sandbox-controller | 8085 | GET /health, 10s / 5 retries | — |
| sandbox-worker | — | Created on demand by the controller | — |
| sandbox-egress-proxy | 8000 (opaque TCP / HTTP CONNECT) | — | — |
Termination and hard-loss behavior
api-backend handles SIGTERM and SIGINT with a bounded graceful window. It
stops accepting new detached orchestration work, drains private chat,
collaborative-room, and prompt-chain producers, closes runtime streams, and
disposes database connections. The process allows up to 20 seconds for its
ASGI shutdown; the packaged deployment gives containers 30 seconds before a
forced kill.
agent-runtime marks health unavailable while draining, refuses new write
requests, aborts registered runs, and waits up to 15 seconds before forcibly
closing remaining connections. Read-only health observation remains available
long enough for the orchestrator to remove it from service.
These hooks improve ordinary rollouts; they are not disaster recovery. A
SIGKILL, OOM kill, crashed process, failed node, or power loss runs no hook.
PostgreSQL lease fencing, the background recovery coordinator, artifact
checkpoint reconciliation, and fresh browser authorization cover that path.
Never shorten the orchestration lease to make recovery look faster without
also proving that a partitioned old executor cannot still publish.
About the 15-minute start period
api-backend allows a long start period on purpose: first boot runs migrations
and verifies dependencies including blob storage.
This has a practical operational consequence. If a network policy blocks the backend from reaching the object store, the service sits at 1/2 ready for the whole window rather than failing loudly. If a deploy hangs at "api-backend not ready", check object-store reachability before anything else. See Troubleshooting.
Dependency shape
Trace startup dependencies and runtime paths across the application, data, worker, and sandbox services.
Images
Every Kaizen-owned image is built as one coherent release set for the extension
version, currently 0.7.6; catalog installs pin each image by digest (see
Install & upgrade). That set includes the purpose-built
PostgreSQL, non-persistent Valkey, API and workers, object store, extractor,
web, agent runtime, sandbox worker, and egress proxy images.
Supported architectures
The release set publishes both linux/amd64 and linux/arm64, so one version
installs on x86-64 hosts and on ARM64 hosts including Apple Silicon.
One limit applies today. When the controller creates a sandbox worker it names
linux/amd64 explicitly rather than letting the node resolve the multi-architecture
manifest, so agent code execution does not yet run on an ARM64-only node. Every
other service does. An operator running a mixed-architecture cluster, where the
worker's architecture must be chosen deliberately rather than inferred, is
unaffected by that limit.
The sandbox-controller is the sole external runtime image. It comes from
the platform operators repository and is pinned by digest. Its source commit
and digest are recorded in release-dependencies.json and validated with the
rest of the release contract. The sandbox-worker is created on demand by
that controller and remains revision-coupled to api-backend.
The revision-coupling rule
sandbox-worker is not an extension Deployment; the controller creates its
pods on demand from whatever image tag the deployment selected.
Therefore an api-only rollout is safe only after the matching worker tag has
been pushed. Otherwise the controller stays healthy while every new worker
enters ImagePullBackOff, and Kaizen reports the resulting create timeout as
sandbox unavailability — a confusing symptom with a simple cause.
Use a full deployment whenever the matching worker image has not already been published. The repository wrapper checks the registry before a filtered single-service rollout and refuses when the companion tag is missing.
Standalone Docker Compose
docker compose up loads the repository's docker-compose.override.yml as a
local-only layer. It starts the application and data services without the
operator-native sandbox controller, because that controller requires Kubernetes
credentials and platform-managed network policy. In this mode Kaizen sets
SANDBOX_ENABLED=false, so code execution, artifact runs, package tests, and
browser sessions are unavailable rather than falling back to a weaker local
executor. Use the unmodified canonical topology through Kamiwaza to enable the
controller-managed sandbox.
Resource envelope
Limits and reservations per instance:
| Service | CPU limit | Memory limit | CPU reserved | Memory reserved |
|---|---|---|---|---|
| document-worker | 4 | 4 GB | 0.5 | 1 GB |
| postgres | 2 | 4 GB | 0.25 | 512 MB |
| api-backend | 2 | 2 GB | 0.25 | 512 MB |
| background-worker | 2 | 2 GB | 0.25 | 512 MB |
| agent-runtime | 2 | 2 GB | 0.25 | 256 MB |
| package-tool-runner | 2 | 2 GB | 0.25 | 256 MB |
| seaweedfs | 2 | 2 GB | 0.25 | 512 MB |
| document-extractor | 2 | 2 GB | 0.25 | 512 MB |
| sandbox-controller | 1 | 2 GB | 0.05 | 256 MB |
| valkey | 1 | 768 MB | 0.1 | 128 MB |
| sandbox-worker (live pod) | 2 | 4 GiB | 0.05 | 256 MiB |
| frontend | 1 | 512 MB | 0.1 | 64 MB |
| sandbox-egress-proxy | 0.5 | 128 MB | 0.05 | 32 MB |
Totals including one on-demand sandbox worker: about 23.5 CPU / 27.7 GB at limits and 2.60 CPU / 4.74 GB reserved. The baseline before any sandbox worker is created is 21.5 CPU / 23.4 GB at limits and 2.55 CPU / 4.47 GB reserved.
The sandbox-worker row shows what the sandbox controller applies to each live
worker pod (SANDBOX_RESOURCE_*), not the image's Compose declaration, which
no running pod carries. Plan the pool from
Requirements & capacity planning.
Hardening posture
- frontend and sandbox-egress-proxy run with read-only root filesystems.
- No service runs privileged; capabilities are dropped by default.
- valkey has no persistence, so a compromise or a restart exposes and loses only ephemeral state.
- package-tool-runner deliberately holds none of the backend's secrets.
Every deployed service declares the full restricted context — no privilege
escalation, not privileged, all capabilities dropped, RuntimeDefault seccomp —
and the running containers carry it.
Reading the posture back
Compose→Kext translation drops false-valued booleans, so the KamiwazaExtension
resource echoes back only capabilities and seccompProfile. Do not read the
absence of allowPrivilegeEscalation or privileged there as a missing
control: it is a lossy projection of the declaration, not the state of the
workload.
Verify the posture against the pods, which is where it is actually enforced:
kubectl -n <namespace> get pods -l extensions.kamiwaza.io/deployment-id=<kext> \
-o jsonpath='{range .items[*].spec.containers[*]}{.name}{" "}{.securityContext}{"\n"}{end}'
The release UAT does exactly this. It holds the Compose declaration and the live pod specs to the complete restricted context, and requires only that the resource in between never echo a weakened value.
Network requirements
| From | To | Why |
|---|---|---|
| api-backend | postgres:5432 | System of record |
| api-backend | seaweedfs:8333 | Checked at startup — blocking this stalls readiness |
| api-backend | sandbox-controller:8085 | Startup dependency |
| api-backend | Kamiwaza platform | Identity, models, connectors |
| background-worker, document-worker | postgres:5432, valkey:6379 | Queue and ledger |
| background-worker | document-extractor:9998 | Text extraction |
| agent-runtime | api-backend:8000 | Every callback |
| sandbox-worker | sandbox-egress-proxy:8000 | Only when egress is enabled |
| frontend | api-backend:8000 | API proxy |
The egress proxy serves 8000 rather than a port of its own choosing. The platform operator writes the sandbox NetworkPolicy and permits exactly one destination: TCP/8000 to pods belonging to this deployment. Nothing in the manifest or the extension SDK can widen it, so a proxy listening anywhere else is unreachable — a sandbox never gets to present its grant, and every outbound request hangs until it times out rather than being refused.
Idle timeouts in front of the API
Several of Kaizen's responses are long-lived streams rather than short requests. They stay open for the whole of an agent run — minutes, occasionally longer — and are legitimately silent for much of it: while the model produces its first token, while a tool runs, and for as long as a write sits waiting on a member's approval. Every hop on the path must therefore tolerate a connection that is open, healthy, and sending nothing.
Each such stream carries its own keepalive, and they are configured separately:
| Stream | Keepalive setting | Default |
|---|---|---|
Member conversation events (GET /conversations/{id}/events) | SHARED_CHAT_PRESENCE_HEARTBEAT_SECONDS | 10s |
| Agent-builder and Studio sandbox chat | MEMBER_CHAT_STREAM_HEARTBEAT_SECONDS | 15s |
| Prompt-chain run streams | fixed comment heartbeat | — |
The bundled frontend nginx already tolerates a quiet stream. Its /api
location sets proxy_read_timeout and proxy_send_timeout to 86400s and
turns proxy_buffering off, so the hop Kaizen ships will not close one and
will not hold frames back until a buffer fills.
What Kaizen cannot configure is whatever an operator puts in front of that container — an ingress controller, a service mesh sidecar, a corporate proxy, a CDN. These commonly default to an idle timeout in the tens of seconds; a hundred seconds is a widespread default. When one closes a silent response, the member's turn stops streaming mid-answer.
Your proxy's idle timeout must exceed the longest of the intervals above, with room to spare — treat anything under 60 seconds as needing attention, and either raise it or lower the intervals to match. Also disable response buffering for the API path; a buffering proxy defeats streaming even when it never times out.
A proxy that closes a conversation stream costs more than a reconnect. The client re-establishes and is served the turn from the journal, so thinking and tool steps come back, and in-progress artifact content returns from a short-lived snapshot — but the answer's prose arrives in one block instead of streaming, because assistant text fragments are deliberately not journaled. See Streaming & event contracts for why, and what the member sees.
An operator debugging "streaming works, then stops" should measure the idle timeout on every hop before looking at Kaizen, and should check which stream is affected: ordinary chat and the agent builder keepalive on different settings.
Delegated workload identity
The platform publishes its delegated-workload contract and readiness surface by
default, but publication alone grants no authority. Kaizen always declares
roles for only the two Task services, while the separate Tasks rollout remains
closed until the complete capability graph is ready. api-backend
projects the task-control service account at the fixed assertion path and may
create/read consent intents, reserve or observe cancellation for runs, and read
readiness. background-worker projects the separate task-executor service
account and may claim, start, heartbeat, observe cancellation, and transition
fenced runs; reserve, execute, and transition exact effects; request brokered
credential use; and read readiness.
Both roles use the fixed kamiwaza-workload-sts audience and assertions no
older than ten minutes. Core prefers live Kubernetes TokenReview and may select
only the registered offline projected-token baseline as a portable fallback.
The projected volume supplies an assertion, not a reusable Core bearer token.
DPoP private keys and every run/effect capability remain in process memory.
The frontend, agent runtime, package runner, document worker, and queue receive
no delegated-workload credential or role.
Required network policy permits the two declared services to reach the Core workload STS, readiness, run/effect, resource-guard, quota, approval, audit, model-admission, and credential-broker endpoints they use. It does not expose those internal endpoints to the frontend or package process.
A restrictive default-deny policy that omits the object-store rule is the most common cause of a deployment that never becomes ready.