Scaling & performance

What scales horizontally, what does not, which knob actually moves the number, and how to tell which one you need.

What scales horizontally

Component	Scales	Why it is safe
frontend	Freely	Stateless, holds no secrets
api-backend	Yes	Room lanes live in the journal; dynamic sandbox admission and ownership live in Valkey
agent-runtime	Yes	Secretless; all authority is re-checked on callback
background-worker	Yes	Due claims use FOR UPDATE SKIP LOCKED; cron dedup gives exactly-once-per-interval across replicas
document-worker	Yes	Same queue semantics
package-tool-runner	Yes	Stateless
sandbox-worker	On demand	Created by the controller, bounded by the pool ceiling

What does not

Component	Constraint
PostgreSQL	Single primary. It is the system of record and the schedule authority. Scale it vertically and with read discipline, not by adding writers
sandbox-controller	One active controller service manages worker Pods; API replicas coordinate its deployment-scoped pool through Valkey
valkey	Required for queues, leases, and dynamic sandbox allocation; reachability loss closes new admission, while keyspace loss causes all controller inventory to be fenced and deleted before pool rebuild

PostgreSQL is the component to over-provision first. Almost every latency problem in Kaizen eventually reduces to it.

Where time goes

A search request

Stage	Typical dominant cost	Budget
Query embedding	Model round trip	SEARCH_EMBED_LATENCY_BUDGET_SECONDS 5.0
Lexical branches	Index scan	SEARCH_BRANCH_LATENCY_BUDGET_SECONDS 8.0
Vector branch	Recall setting	HNSW_EF_SEARCH 80
Fusion	Negligible	—
Rerank	Model round trip	Only if selected

No model writes a search answer, which is why Search is fast. If Search feels slow, the cause is almost always embedding latency or index health — not "the AI".

A chat turn

Stage	Dominant cost
Authority resolution	PostgreSQL
Model steps	Up to AGENT_LOOP_MAX_CYCLES (10) general tool-bearing rounds; staged creation can add 52 bounded protocol rounds plus the runtime's small terminal reserve
Tool calls	Whatever the tool does — run_code can be 30s
Compaction	Off the critical path, when warm candidates are enabled
Journal commit	PostgreSQL

Chat latency is dominated by model round trips. Adding backend replicas does not make a turn faster; it makes more turns possible.

The knobs that actually matter

Throughput

Knob	Default	Raise when
api-backend replicas	1	Request concurrency saturates
agent-runtime replicas	1	Concurrent turns queue
WORKER_CONCURRENCY	4	Ingestion backlog grows with idle CPU
SANDBOX_POOL_MAX_ALLOCATED	32	Members wait then fail to acquire
TASKS_MAX_CONCURRENT_RUNS_PER_OWNER	10	One owner's tasks serialize unacceptably

Latency

Knob	Default	Effect
SANDBOX_WARM_POOL_MIN	3	Removes cold-start from the first execution
CONTEXT_COMPACTION_WARM_CANDIDATES_ENABLED	on	Moves compaction off the critical path
PROMPT_CACHE_ENABLED	off	Cuts repeated prefix cost when turned on
AGENT_LOOP_MAX_PARALLEL_TOOLS	1	Raising helps genuinely independent reads
HNSW_EF_SEARCH	80	Lower is faster, less recall

Quality, at a cost

Knob	Default	Cost
SEARCH_POOL_SIZE	50	More candidates to fuse
Reranker selected	none	A model round trip per query
SEARCH_EXPANSION_ENABLED	off	Extra retrieval work
INGEST_CONTEXTUAL_RAG_ENABLED	off	A model call per chunk at ingest
AGENT_LOOP_MAX_CYCLES	10	More general tool-bearing rounds per turn; does not change the derived staged-artifact allowance

Sizing the sandbox pool

Size from measured concurrent execution demand, not member count:

concurrent workers ≈ peak execution arrivals/second × p95 execution seconds
headroom           = reset/create failures + burst reserve + node disruption

1,000 signed-in users do not imply 1,000 workers. At 2% concurrent execution, the starting point is roughly 20 workers plus headroom. Queue wait must stay below the caller's end-to-end tool timeout.

Cold creates reserve capacity under the allocator lock and perform the network request after releasing it, so independent creates run concurrently and releases are not blocked behind a slow pod creation.

The allocator lock, FIFO, warm target, hard ceiling, and per-member cap are deployment-scoped in Valkey. Increasing API replicas does not multiply sandbox limits. If Valkey or controller inventory is unavailable, replicas reject new leases instead of creating independent local pools. If the Valkey keyspace is lost rather than briefly unreachable, reconciliation cannot trust any controller inventory: it deletes all Kaizen workers, including active workers, then rebuilds the pool before admission reopens.

The symptom-to-knob table

Symptom	Look at	Then
Search slow, chat fine	Embedding latency, index health	Lower HNSW_EF_SEARCH, check the embedding deployment
Search returns nothing semantic	"embedding failed; hybrid unavailable"	Fix the embedding role — see Models
Chat slow	Model round-trip latency	Model capacity, or lower effort/cycles
run_code waits then fails	Acquire wait and queue depth	Raise SANDBOX_POOL_MAX_ALLOCATED
First run_code always slow	Warm pool exhausted	Raise SANDBOX_WARM_POOL_MIN
Tasks fire late	kaizen_task_dispatch_lag_seconds	Database health first, then worker cadence
Ingestion backlog	Worker CPU, embedding budget	Raise concurrency, or split the document queue
Ingestion delays task dispatch	Shared queue	DOCUMENT_WORKER_SEPARATE_QUEUE_ENABLED
Everything slow	PostgreSQL	Connections, slow queries, vector index memory

Measure before you scale

1. kaizen_task_dispatch_lag_seconds — is the scheduler keeping up?
2. Sandbox queue wait and acquire failures — is the ceiling right?
3. Retrieval branch latency against its 8s budget.
4. Embedding latency against both budgets separately.
5. PostgreSQL connection saturation and slow-query profile.
6. Model round-trip latency at the gateway.

Changing a replica count without these is guesswork, and it usually moves the bottleneck rather than removing it.

Cost levers

• LLM_PER_USER_DAILY_TOKENS defaults to 0 (unlimited); set a positive member daily cap when model cost needs a ceiling.
• AGENT_LOOP_MAX_CYCLES bounds general tool-bearing rounds. Capacity planning must add the staged-artifact protocol allowance for creation-heavy workloads.
• LLM_DEFAULT_EFFORT sets the deployment-wide reasoning default.
• Leave INGEST_CONTEXTUAL_RAG_ENABLED off unless measured recall justifies a model call per chunk.
• Member compression is opt-in and member-owned — it is not a deployment cost control.

Related

• Requirements & capacity planning
• Observability
• Sandbox pool sizing