Skip to main content

Sandbox pool sizing

The sandbox pool trades idle capacity for predictable latency. This page covers sizing it, the invariants that keep it safe, and the failure modes worth knowing.

Why a pool exists​

Booting an isolated environment per request would make every run_code call start with a multi-second cold create. Kaizen keeps a small pool allocated so a lease is usually instant.

The knobs​

SettingDefaultMeaning
SANDBOX_WARM_POOL_MIN3Minimum allocated workers
SANDBOX_POOL_MAX_ALLOCATED32Hard ceiling across warm, leased, resetting, and creating
SANDBOX_POOL_TARGET_IDLE1Clean reserve maintained during demand
SANDBOX_BURST_IDLE_TTL_SECONDS300Cooldown before clean surplus is deleted
SANDBOX_MAX_CONCURRENT_CREATES3Deployment-wide Kaizen create concurrency
SANDBOX_ACQUIRE_WAIT_SECONDS30Bounded FIFO wait for a worker being created or reset
SANDBOX_ACQUIRE_QUEUE_MAX256Hard queue-memory bound
SANDBOX_MAX_CONCURRENT_PER_MEMBER2Per-member fairness

All eight settings are live controls in Admin → Settings → Agents & tools. An accepted edit publishes one versioned policy to Valkey for every API replica; no restart is required. Controller URL, Valkey URL, lease backend, deployment identity, worker image, and lease timing remain deployment wiring and require rollout or restart.

Lowering a bound never interrupts an active execution. New admission observes the lower member cap immediately, surplus clean workers retire after current leases reset, and a smaller queue limit rejects new tickets without deleting existing tickets. SANDBOX_ACQUIRE_WAIT_SECONDS = 0 restores immediate-busy behavior when no clean worker or new create is available.

Each worker is 2 CPU / 4 GiB at limits, so the ceiling is also a statement about burst capacity your cluster must be able to satisfy: at the shipped SANDBOX_POOL_MAX_ALLOCATED of 32, a full pool is charged 64 CPU / 128 GiB against a quota on limits, while reserving only 1.6 CPU / 8 GiB.

Capacity model​

Size from measured concurrent execution demand — not total users:

concurrent workers ≈ peak execution arrivals/second × p95 execution seconds
headroom           = reset/create failures + burst reserve + node disruption

1,000 signed-in users do not imply 1,000 workers. If 2% run code concurrently, the starting requirement is roughly 20 workers plus measured headroom.

Queue wait should stay below the caller's end-to-end tool timeout. If members regularly wait the full SANDBOX_ACQUIRE_WAIT_SECONDS and then fail, the ceiling is too low — raising the timeout only makes the failure slower.

Remember that artifact runs, package tests, and workspace-persisted agent calls each take a lease too. run_code is not the only consumer.

Persistent Python and Node cells do not pin workers when workspace persistence is enabled. Each cell checkpoints supported state into the same bounded workspace snapshot and crosses the unconditional release boundary. Capacity therefore remains a function of concurrent cells, not the number of conversations that have used a kernel. The restored checkpoint still counts toward SANDBOX_WORKSPACE_MAX_BYTES.

If SANDBOX_WORKSPACE_PERSIST is disabled, persistent kernels use the legacy conversation-affinity path and hold a worker. Size that mode separately or keep the scalable default enabled.

Allocation behavior​

Cold creates reserve capacity under the allocator lock and perform the network request after releasing it. Independent creates therefore run concurrently, and releases and resets are not blocked behind a slow pod creation.

Valkey holds one deployment-scoped FIFO and worker state map. The warm minimum, allocated ceiling, create bound, queue, and per-member cap therefore keep the same meaning when api-backend has multiple replicas.

Every execution path — restore, policy, dispatch, snapshot, timeout, transport error, cancellation — crosses one unconditional release boundary. There is no path that leaks a lease.

Reset or discard​

After use, Kaizen kills leftover processes and strictly wipes the workspace before returning the instance to the pool.

A sandbox is reused only when the reset can be proven clean. A reset failure evicts the worker rather than returning uncertain state to another member. The reuse decision requires positive evidence, not merely the absence of an error.

Leases, crash takeover, and coordination outages​

Pool ownership lives in deployment-scoped Valkey state. Leases carry SANDBOX_LEASE_TTL_SECONDS (900) and heartbeat every SANDBOX_LEASE_HEARTBEAT_SECONDS (60). Another replica can continue a routed session by atomically taking over its ownership record.

Expired ownership is handled conservatively. Because workers do not validate Kaizen's operation token, an expired endpoint is never handed to a new owner. Kaizen marks it retiring, requests controller deletion, confirms 404, then reopens capacity and creates a clean replacement. A late release or reset from the former owner cannot publish that replacement.

Creation failures are also conservative: a reservation remains counted when the HTTP response is ambiguous. Inventory reconciliation locates the worker by its deterministic requested name and confirms deletion before removing the reservation. Recovery depends on the controller preserving Kaizen's requested name unchanged in inventory; both supported controller backends provide that contract.

warning

SANDBOX_LEASE_BACKEND=redis and a reachable SANDBOX_REDIS_URL are mandatory for controller-created workers. A temporary Valkey reachability or controller-inventory outage makes new sandbox admission fail closed; Kaizen does not start a process-local fallback pool. Loss of the Valkey keyspace is more disruptive: controller inventory becomes untrusted, so reconciliation fences and deletes every Kaizen worker it finds, including workers serving active executions, before rebuilding the pool.

The revision-coupling invariant​

sandbox-worker is not an extension Deployment. The controller creates its pods on demand from whatever image tag the deployment selected, and the API and worker are revision-coupled.

An api-only rollout is safe only after the matching worker tag has been pushed. Otherwise the controller stays healthy while every new worker enters ImagePullBackOff, and Kaizen reports the resulting create timeout as sandbox unavailability — a confusing symptom with a simple cause.

Use a full deployment whenever the matching worker image has not been published. The repository wrapper checks the registry before a filtered single-service rollout and refuses when the companion tag is missing. Do not bypass that check.

During a rolling deployment, Kaizen records the selected worker image in shared pool coordination. Idle workers from the previous revision are fenced and replaced immediately. A worker already serving an execution is allowed to finish and reset, then is replaced before it can accept another lease. This keeps active work intact while converging the entire warm pool on the selected release image. Reconciliation republishes the active replica's image selection on every pass. This recovers both from Valkey being unavailable at startup and from an older replica restarting late in a rolling update and overwriting the selector. A transient rollout ordering cannot leave the surviving API Ready with an empty pool tied to the previous release. Published catalog deployments resolve that image to an immutable digest; development deployments use a unique revision tag.

Workspace storage​

Compressed workspace bytes use the S3-compatible object store by default, under an owner- and session-hashed key (SANDBOX_WORKSPACE_BACKEND = object_store, SANDBOX_WORKSPACE_MAX_BYTES 8 MiB).

The retained Redis store remains available, but the canonical path deliberately avoids putting multi-megabyte values in the shared Valkey memory budget.

Hashline snapshot and named-register authority uses a separate compressed object under the same composite identity, bounded by SANDBOX_HASHLINE_STATE_MAX_BYTES (64 MiB). Unlike workspace persistence, authority reads and commits fail closed: losing the state may require a fresh read_workspace, but Kaizen never guesses what a four-hex tag or named register meant. Local affinity tests use an in-process store; multi-replica deployments use the selected shared workspace backend.

Current known cost: the worker RPC uses base64 JSON and creates a full gzip-tar per persisted execution. Binary streaming and incremental manifests are follow-up work.

Symptoms and responses​

SymptomLikely causeResponse
First execution always slowWarm pool exhaustedRaise SANDBOX_WARM_POOL_MIN
Waits ~30s then failsCeiling too lowRaise SANDBOX_POOL_MAX_ALLOCATED
"Sandbox unavailable" after a partial deployWorker tag missingPublish the companion tag; full rollout
Workers churn constantlyReset failuresInvestigate reset errors — eviction is correct, the churn is the signal
One member monopolizes capacityPer-member capLower SANDBOX_MAX_CONCURRENT_PER_MEMBER
All replicas reject new leasesValkey or controller inventory unavailableRestore shared coordination; do not switch to memory in controller mode
All workers are deleted after Valkey recoveryValkey keyspace was lostLet reconciliation finish and pool rebuild before reopening member traffic
Valkey memory pressureRetained workspace storeConfirm SANDBOX_WORKSPACE_BACKEND = object_store