Requirements & capacity planning

Sizing Kaizen against a real member population, rather than against a headcount.

Baseline footprint

A single-replica deployment before any sandbox worker is created declares 21.5 CPU / 23.4 GB at limits and 2.55 CPU / 4.47 GB reserved. Each on-demand worker adds 2 CPU / 4 GiB (4.3 GB) at limits and 0.05 CPU / 256 MiB reserved; with one worker, the totals are about 23.5 CPU / 27.7 GB and 2.60 CPU / 4.74 GB.

Limits are ceilings, not steady-state consumption. Plan the node pool against reservations plus measured peak, not against the sum of limits — a cluster provisioned for the limit total will be idle most of the time.

Running inside a namespace quota

A namespace-admin install lands in someone else's ResourceQuota, and that quota usually counts limits.cpu rather than reservations. The one-worker total above therefore reads as 23.5 CPU to the admission controller even though the deployment idles near 2.6. When the quota is smaller than that, pods are not throttled — they are never created. The symptom is a no healthy upstream route with exceeded quota on the ReplicaSet events, which looks like a routing fault and is not one.

Reduce limits deliberately, in this order:

1. Lower the burst ceilings, not the reservations. The reservations are what the services actually need to run; the ceilings exist for peaks.
2. Include the sandbox pool. SANDBOX_RESOURCE_CPU_LIMIT and SANDBOX_RESOURCE_MEMORY_LIMIT are charged to the same quota for every worker the controller creates — 2 CPU / 4 GB each by default — and a single worker can be the reason a frontend pod cannot be admitted. A quota that counts limits fits few workers; SANDBOX_RESOURCE_CPU_LIMIT=1 and SANDBOX_RESOURCE_MEMORY_LIMIT=2Gi halve that charge.
3. Re-check the startup budgets. Cutting CPU lengthens cold starts. document-extractor compiles and binds Tika on first start — roughly 26 seconds on a one-core slice — which is why it declares a 180-second startup window. A CPU reduction without headroom in the startup budget produces a crash loop that reads as an unhealthy service rather than a sizing mistake.
4. Leave rollout headroom. A RollingUpdate with a surge needs room for one extra pod during every upgrade. A quota with no surge headroom stalls the rollout while reporting the old pod as healthy.

warning

Do not lower a limit below its reservation. The pod is admitted and then starved, which is harder to diagnose than the refusal it replaces. For sandbox workers the controller refuses it outright: lowering a worker ceiling below the shipped 50m / 256Mi reservation stops the controller at startup until the matching SANDBOX_RESOURCE_*_REQUEST is lowered too (see Configuration).

Storage

Store	Grows with	Guidance
PostgreSQL	Conversations, journals, passages, embeddings, task ledger, audit	The dominant consumer. Vector columns and audit rows both grow monotonically
Object store	Uploads, exports, artifacts, workspace snapshots	Grows with member activity; large uploads dominate
Valkey	Nothing durable	768 MB is ample; it is a lease store

Three things grow with activity and deserve explicit retention policy: the audit trail, task run events, and external capability replay receipts. Their size is a policy decision rather than a capacity surprise.

External capability sources add small indexed PostgreSQL metadata rows and one credential-free replay receipt per attempted endpoint call. They do not add schema documents, endpoint responses, or credentials to PostgreSQL or Valkey. Replay receipts are intentionally retained for idempotency and audit correlation, so size their retention with invocation volume. Explicitly remembered credentials live in Kamiwaza DataHub/Catalog, not Kaizen storage.

Schema metadata is process-local and shared across members by URL, with ETag, single-flight refresh, 60-second freshness, 15-minute stale fallback, and 16-way bounded refresh. Exact endpoint selections encode the source id and use an indexed targeted query; they do not enumerate every workroom source. When running multiple agent-runtime replicas, budget the 16 MiB worst-case schema bound per concurrently refreshed source per replica, and prefer exact or source selectors over the all-external parent in very large workrooms.

Multipart upload cleanup

Budget for orphaned session receipts and unregistered object-store bytes that remain for expiry cleanup. Their reclamation requires an eligible session and successful cleanup; expiry alone does not prove provider storage was reclaimed.

Cleanup bound	Capacity implication
24-hour session TTL and at least 1 hour idle	Retained receipts and unregistered bytes remain eligible only after both conditions hold
At most 64 candidate scans every 15 minutes	At most 6,144 scans/day per scheduling worker at uninterrupted cadence; not guaranteed distinct removals
Provider failure moves expiry to 1 hour after failure	Failed reclamation occupies storage until a later successful pass

Large backlogs can outpace cleanup; provider failures, revisited sessions and slow passes lower actual throughput. Monitor receipt and unregistered-byte growth and multipart_upload.cleanup_retry warnings. Cleanup requires document import to be enabled. See the exact cleanup bounds and the coordinated writer cutover.

Sizing the sandbox pool

Size the worker ceiling from measured concurrent execution demand, not total users:

concurrent workers ≈ peak execution arrivals/second × p95 execution seconds
headroom           = reset/create failures + burst reserve + node disruption

1,000 signed-in users do not imply 1,000 workers. If 2% run code concurrently, the starting requirement is roughly 20 workers plus measured headroom.

Queue wait should stay below the caller's end-to-end tool timeout — if members regularly wait SANDBOX_ACQUIRE_WAIT_SECONDS (30) and then fail, the ceiling is too low, not the timeout too short.

Each worker is 2 CPU / 4 GiB at limits, against 0.05 CPU / 256 MiB reserved. Twenty concurrent workers is therefore ~40 CPU / 80 GiB of burst capacity that must exist somewhere in the pool, while the scheduler only ever sees ~1 CPU / 5 GiB of it — sixteen to one on memory.

That gap is deliberate. A run_code worker rarely approaches its ceiling, so reserving for the worst case would strand most of a node; a worker holding a browser session, or an agent working a moderately hard job, can sit near it for a long time. The ceiling is 4 GB because a browser alone needs about 1.4 GB (the worker refuses below 1.5 GB) and a coding agent's build or test run can need several GB — the same pool serves all of them, and the controller applies one resource profile to every worker it creates.

So size the burst capacity for the number of concurrent heavy workers — browser sessions and demanding agent jobs — you expect, not for the pool ceiling. Only the reservation is guaranteed — 16:1 on memory and 40:1 on CPU at the shipped values. When a node runs short of memory, the kubelet evicts pods using most beyond their request, which is a live browser or a busy agent job rather than an idle worker, and CPU is shared among busy workers rather than delivered at the ceiling. Disk usually binds before memory does. With SANDBOX_PERSISTENCE off (the default), the workspace (/tmp/work) is a scratch volume capped by SANDBOX_STORAGE_SIZE, which ships at 1Gi, and the worker's writable layer, /tmp and that workspace share the controller's ephemeral-storage limit of 2 GiB. A build that writes more than about 1 GiB into its workspace is evicted long before it nears the 4 GiB memory ceiling. Raising SANDBOX_STORAGE_SIZE helps only up to that shared 2 GiB: Kaizen does not pass the controller's ephemeral-storage settings, so 2 GiB of disk per worker is the real bound in this release.

See Sandbox pool sizing.

Sizing the workers

Ingestion is the heaviest regular workload. document-worker is deliberately larger (4 CPU / 4 GB) than background-worker (2 / 2) because extraction, chunking, and embedding dominate.

Setting	Default	Effect
WORKER_CONCURRENCY	4	Jobs per worker process
WORKER_JOB_TIMEOUT_SECONDS	43200	Ceiling for a long ingestion
INGEST_EMBED_BATCH_SIZE	16	Embedding batch
INGEST_EMBED_MAX_BATCH_TOKENS	4096	Token ceiling per batch
DOCUMENT_WORKER_SEPARATE_QUEUE_ENABLED	off	Isolate document work from task dispatch

Turn on the separate document queue when a large reindex starts delaying task dispatch. That is the symptom it exists to fix.

Sizing PostgreSQL

PostgreSQL carries retrieval, the journal, and the task ledger, and it is the schedule authority. It is the service to over-provision first.

Watch:

• Vector index memory. HNSW recall is bought with memory and HNSW_EF_SEARCH (default 80).
• Connection count. The backend, both workers, and each additional replica all hold pools.
• Due-claim latency. Task dispatch uses FOR UPDATE SKIP LOCKED every TASKS_DISPATCH_CADENCE_SECONDS (5).

If dispatch lag grows while workers are Ready, look at database health before worker capacity.

Multipart upload connections

Each admitted multipart part PUT holds a PostgreSQL transaction and connection through its object-store PUT and ETag commit so cancellation and completion cannot overtake an in-flight part. MULTIPART_PART_DB_CONNECTION_BUDGET limits those calls across API replicas; the default is 2. Excess parts receive a retryable 429 upload_busy response. This budget does not cap other database users or the number of upload requests reading part bodies, so reserve connection headroom for terminal upload operations, member requests, and workers before raising it. Use the same setting on every API replica, and measure peak connections and object-store latency on the target deployment. The browser can issue several parts at once, so a low budget may cause retries and longer uploads during bursts. Admission happens after the entire part body is buffered, so a rejected part must resend up to 16 MiB on retry. Sustained contention can exhaust the browser's bounded retries and fail the file; lowering the budget trades connection headroom for upload throughput and extra bandwidth. Each replica separately admits 8 part bodies, with at most 4 for one verified member across all their sessions. The client shares four active part transfers across folder files and independent uploads in the same renderer; Desktop uses at most two of those slots. Queued files take turns at part boundaries, so a folder remains concurrent without multiplying its per-file Uppy limits. A failed attempt releases its client slot before Uppy's existing bounded backoff. Queued cancellation sends no part; an active bridge retains its slot until its transport promise settles. The queue holds at most 64 attempts and expires each waiting attempt after 45 minutes. Expiry fails the file for an explicit retry; it does not extend the backend body deadline or prove termination of provider work. Other tabs, renderers and devices still contend through the server's bounded 429 policy. This leaves server capacity for another member; it does not reserve capacity across clients. The memory and member permits remain held until storage and ETag persistence finish, including while cancellation waits for provider work to stop. Completed, rejected, and cancelled requests release their member counters. A body may continue beyond two minutes while receiving bytes. The idle limit is 120 seconds since the last nonempty chunk; empty chunks do not extend it. A separate 30-minute total receive limit prevents indefinite trickling. Either limit returns 503 upload_timeout, which the supported multipart browser and Desktop transports retry within their bounded sequence. The bundled unbuffered nginx /api/ hop has a finite 150-second request-body inter-read timeout, giving the backend idle response 30 seconds of margin. Its send/read upstream timeouts govern other directions. The platform edge and any external proxies must be qualified separately; a proxy-generated 408 is still terminal on the supported Uppy part rail. No provider write starts for an expired body. For a 16 MiB part, the hard limit requires about 0.075 Mbit/s per part, or about 0.3 Mbit/s of shared uplink for four browser parts; slower transfers can still fail and retries resend the part. Allow memory for eight 16 MiB buffers plus temporary provider copies and normal API traffic. These fixed body bounds are independent of the database setting. The valid range is in Limits & quotas.

Model capacity

Kaizen does not host models — it routes to Kamiwaza deployments. Plan there for:

• Chat model throughput at peak concurrent turns. Ordinary turns may take up to AGENT_LOOP_MAX_CYCLES (10) general tool-bearing rounds. A staged creation may add at most 52 begin/append/finalize protocol rounds, plus the runtime's small terminal reserve; include that case when artifact creation is common.
• Embedding model throughput for both interactive query embedding and bulk ingestion. These have separate latency budgets on purpose; a large reindex should not slow interactive search.
• Reranker, if selected. With none selected, Kaizen makes no reranking call.

LLM_PER_USER_DAILY_TOKENS defaults to 0, so daily member model usage has no cap. Set a positive per-member limit before a pilot if model cost is a constraint. Per-turn and per-run bounds still apply.

Corpus-job capacity

Corpus Tasks bound memory by partition and database page, but they deliberately move total state into PostgreSQL and the object store. Size from admitted source bytes, partitions, canonical records, provenance fan-out, and export bytes — not from the graph-node count in a browser.

The release benchmark commits 38,000 unique ontology nodes from 40 mixed CSV, Markdown, and JSON sources, restarts halfway, validates first/last lineage, and repeats deterministic JSONL/GraphML streaming under a 96 MiB process-memory gate. This proves a supported reference point, not a universal ceiling. Edges, large properties, many-to-many lineage, slower models, and simultaneous owners increase storage and elapsed time independently.

One parent run maps one partition at a time for deterministic checkpoint order; different owners still use the normal bounded Task worker pool. Watch kaizen_task_corpus_operation, database write latency, object-store throughput, and model capacity before raising source, partition, model-call, record, or storage ceilings.

Delegated workload capacity

Task admission has two independent ceilings. The member remains the charge owner under the normal model budget, while the registered workload revision is the concurrency and spend subject. Core reserves both in one correlation before provider I/O; raising a Kaizen task cap cannot bypass either platform ledger.

Size Core claim, proof, policy, quota, resource-guard, audit-outbox, approval, and credential-broker capacity from peak effects per second, not only from worker count. Each worker attempt authenticates, claims, heartbeats, observes cancellation, and can reserve multiple model or tool effects. Keep the claim lease longer than a normal heartbeat interval but shorter than the recovery objective, and keep every derived capability inside the run's immutable authority deadline. More worker replicas do not increase useful capacity once the workload ceiling, member charge, or protected-resource quota is saturated.

Monitor rejected/fenced claims, proof latency, heartbeat lag, effect reservation latency, approval age, quota settlement backlog, credential-lease expiry, audit outbox age, and readiness freshness. Never add member, workload instance, run, resource, or credential IDs as metric labels; use bounded outcome categories and follow an alert into the correlated audit record.

Worked example — 500 members

Assumptions: 500 signed-in, ~50 active in a busy hour, ~5 concurrent chat turns, 2% concurrent code execution, 100k document corpus.

Component	Starting point
api-backend	2 replicas × (2 CPU / 2 GB)
agent-runtime	2 replicas × (2 CPU / 2 GB)
background-worker	1 × (2 CPU / 2 GB)
document-worker	1–2 × (4 CPU / 4 GB)
postgres	4 CPU / 8–16 GB, over-provisioned deliberately
sandbox pool	SANDBOX_POOL_MAX_ALLOCATED ≈ 16, warm min 3
object store	Sized against upload volume, not member count

Then measure. The numbers that actually move are concurrent execution demand and corpus size; member count is the weakest predictor of either.

What to measure before scaling

1. kaizen_task_dispatch_lag_seconds — is the scheduler keeping up?
2. Sandbox queue wait and acquire failures — is the pool ceiling right?
3. Retrieval branch latency against its 8s budget.
4. Embedding latency against its two separate budgets.
5. Postgres connection saturation and slow-query profile.
6. Corpus operation outcomes and checkpoint age during large transforms.
7. Core claim/effect latency and workload quota saturation for delegated Tasks; separate these from Kaizen queue depth and member model charge.

Scaling a replica count without these is guesswork. See Scaling & performance and Observability.