Troubleshooting

Symptom, check, fix — for the failures that actually happen.

A service rejects a retired configuration name

Symptom. A backend process exits with retired product configuration name(s) are not accepted by Kaizen, or agent-runtime and agent-runtime restart because a retired Pi callback name is configured.

Cause. A pre-0.4 process environment or .env still contains a removed product-prefixed name. Compose forwards PI_TOMO_CALLBACK_URL and PI_TOMO_CALLBACK_TIMEOUT_MS to the Pi sidecar only as rejection sentinels, so the API can be healthy while agent-runtime refuses startup.

The old template wrote KAIZEN_MODEL_CONTEXT_WINDOW_OVERRIDES={} as an active line, so a .env copied from that template must rename it to KAIZEN_MODEL_CONTEXT_WINDOW_OVERRIDES even when the value is still empty.

Fix. Rename that model-context override, remove other retired names, or migrate the Pi callbacks to PI_KAIZEN_CALLBACK_URL and PI_KAIZEN_CALLBACK_TIMEOUT_MS. If both KAIZEN_AUTO_MIGRATE and KAIZEN_AUTO_MIGRATE are temporarily present, make their values agree before restarting; do not bypass the conflict to force a migration.

API or worker rejects the tokenizer cache setting

Symptom. An API, background-worker, or document-worker process exits with TIKTOKEN_CACHE_DIR is image-owned and must equal /opt/kaizen/tiktoken-cache, with DATA_GYM_CACHE_DIR must be unset; Kaizen uses the image-owned tokenizer cache, or reports that the image-owned tokenizer cache is missing or invalid. The process exits with configuration error code 78; orchestrators may surface that numeric code instead of the full message.

Cause. A carried-forward .env, Compose override, Helm value, or raw process environment changed the image-owned cache path, retained the legacy DATA_GYM_CACHE_DIR override, or the deployed image does not contain the verified cl100k_base data. Kaizen rejects this state instead of downloading tokenizer data during startup.

Fix. Remove both tokenizer-cache overrides and deploy a complete, matched API-family image. Do not create a writable cache volume or add a runtime egress exception. If the integrity error remains with no override, replace the image from the trusted release rather than copying cache bytes into a running container.

Deployment hangs at api-backend 1/2 ready

Symptom. A fresh deploy never completes. The backend sits at not-ready for the whole 15-minute start period. Logs show blob-store retry errors rather than an obvious denial.

Cause. A restrictive network policy that does not allow the backend to reach the object store on seaweedfs:8333. The backend verifies blob storage during startup, so the check stalls rather than failing loudly.

Check. Confirm the deployment's network policy permits api-backend → seaweedfs:8333. A default-deny policy allowing only the platform's common ports will omit it.

Fix. Add the per-deployment allow rule for the object store. As a temporary unblock you can relax the policy, but the durable fix is the explicit rule — this recurs on every fresh deploy otherwise.

Search returns nothing semantic

Symptom. Exact keyword matches work; paraphrased queries return nothing. Search reports "embedding failed; hybrid unavailable". Ingest events show 404 on the embeddings path.

Cause. The embedding role points at a deployment that does not serve an embeddings endpoint — commonly a general chat deployment that is otherwise healthy.

Check. Correlate a document ingest failure with a 404 on the embeddings path, then confirm the selected target's declared capabilities.

Fix. Point the embedding role at a deployment that actually serves embeddings. Then reindex: vectors from a different model are not comparable.

This is the single most common misconfiguration in a new deployment.

If the warning reads cause=tls_verify_failed, the embedding call could not verify the platform's certificate. Set KAMIWAZA_INTERNAL_API_URL to Core's in-cluster Service (Core's extension runtime injects it by default), or supply the platform's trust bundle, or set KAMIWAZA_VERIFY_SSL=false. See How embedding calls reach Core.

"Sandbox unavailable" after a partial deploy

Symptom. run_code fails with a create timeout. The sandbox controller is healthy. New workers are in ImagePullBackOff.

Cause. An api-only rollout without the matching sandbox-worker image tag. The controller creates worker pods on demand from the selected revision, and the API and worker are revision-coupled.

Check. Confirm the companion worker tag exists in the registry for the deployed revision.

Fix. Publish the companion tag and do a full rollout. The repository wrapper checks the registry before a filtered single-service rollout and refuses when the tag is missing — do not bypass it.

Code execution refused because the chat's files changed

Symptom. run_code returns "The files shared in this chat changed since this reply started". The chat.tool.run_code audit outcome is resource_denied:resource_manifest_changed, with a member.conversation.sandbox.isolation_denied row beside it.

Cause. Between the turn starting and the code starting, the chat's files changed by anything other than the run itself: a file the turn was admitted with was removed, another participant or another turn linked one, or the member attached or updated a file by hand while their own turn was running. A member who lost access to the chat mid-run gets the same refusal when the turn was admitted with files, because the files they can see no longer match. New files the same run linked itself — its own earlier code output, a tool result, or an artifact it created — never cause this.

A related refusal names a file that "was updated earlier in this reply", with the chat.tool.run_code outcome resource_denied:resource_revised. The run itself revised an artifact or document the turn started with, and code would otherwise have read the previous version. In a new message both versions are available to code: each manifest entry carries its version, and one of the two file names is prefixed to keep them apart.

Check. Compare the room's resource links and their turns with the refused run's turn.

Fix. Nothing to repair: the refusal is the isolation check working. Ask again in a new message. A retry inside the same reply is refused again, because the check compares the room with the files that reply started with. A refusal on a run that only used its own new outputs is a defect; keep the two audit rows.

A chat.tool.run_code outcome of unavailable:<reason> names the resource-authorization check that refused (resource_scope_unavailable, resource_manifest_mismatch, resource_projector_unavailable, or resource_projection_failed). A bare unavailable came from the tier, controller, worker, or workspace path.

Background worker will not start

Symptom. The worker restarts in a loop and the deployment never completes.

Cause. Historically, a worker that treated the embedding target as a startup requirement. The embedding model is a runtime choice, not a boot dependency.

Check. Look for a startup-time model or embedding validation in the crash loop.

Fix. Worker startup must not gate on model configuration. A missing or misconfigured embedding target should degrade ingestion, not prevent the worker from becoming Ready.

A safe test stops before the model

Symptom. A safe test reaches a terminal policy_blocked state within a few seconds and never calls a model. Which error_category it carries says whose problem it is.

• draft_test_model_grant_unavailable — the deployment has no platform service identity, so it authorizes safe tests through the member's own single-use draft test model grant, and this test reached the worker without a usable one. Either the member's request carried no platform bearer, the test started after the grant expired (TASKS_LOCAL_TEST_GRANT_TTL_SECONDS), or an earlier execution attempt of the same run (a worker crash, a lease re-claim) already spent it. The member sees a summary asking them to test the draft again, plus Test again. The audit rows for the run (target_id is the run id) name the case:
  • .issued then .consumed earlier, then this refusal: an earlier execution attempt spent the grant and the run was executed again (worker crash or lease re-claim).
  • .issued then .refused with a reason of expired, binding_mismatch or invalid: the grant was opened and rejected, and is spent.
  • .issued only: the run never reached execution, or the unconsumed row was purged after expiry when the member issued another grant.
  • No task.run.model_grant.* row at all: no grant was sealed, because the member's request carried no platform bearer.
• platform_authority_unavailable — the deployment can neither call the model as itself nor seal a member grant. The member is offered no recovery control, only a summary naming an administrator and the usual troubleshooting link, because testing again produces an identical result. On a deployment authorized by workload attestation, scheduled runs keep succeeding while this happens, because attestation does not reach the safe-test path.

Cause. A capability grant authorizes a run to execute; it does not authorize the platform calls that run makes, and every task reaches a model. Confirm the refusals from the runs themselves:

SELECT status, error_category, count(*)
FROM task_run GROUP BY 1, 2 ORDER BY 3 DESC;

A refused run carries no actor_snapshot, because the refusal happens before a run records one — so do not look for authority_source on these rows. A deployment that predates this refusal shows the older shape instead: failed runs whose error_category names a generic runtime fault, each having spent a model call to discover the same thing.

The worker also logs worker.platform_service_identity_absent once at startup when it has no identity of its own.

Fix. For draft_test_model_grant_unavailable, the member tests the draft again from a live session; a new test seals a new grant. For platform_authority_unavailable, give the deployment an authorization of its own: either let the platform inject the workload identity it declares in the extension manifest, or set KAMIWAZA_SERVICE_CLIENT_ID, KAMIWAZA_SERVICE_CLIENT_SECRET and KAMIWAZA_SERVICE_TOKEN_ENDPOINT explicitly. See Configuration.

Do not treat a deployment-wide platform_authority_unavailable as a per-run fault: retrying individual drafts only spends runs to reach the same fixed outcome.

Tasks fire late, or not at all

Symptom. kaizen_task_dispatch_lag_seconds exceeds two dispatch cadences, or queued available_at age grows while workers are Ready.

Check, in order.

1. Database health. PostgreSQL owns due selection; nothing else can compensate.
2. Worker timezone-data identity across all replicas. A split version produces disagreeing occurrence decisions.
3. Worker cadence — is the worker actually waking?
4. Delegated-workload readiness freshness — are every required component, attestation profile, model guard, and dataset guard still ready?

Fix. After readiness recovers, trigger repair_task_handoffs, which safely re-enqueues the same committed Core run references with deterministic job IDs. Core claim fencing ensures exactly one executor remains current.

Do not delete ledger rows or manually mint replacement runs. Broker enqueue failure after a database commit is expected to leave visible queued work — that is the design, and repair is the response.

Authority denied on a scheduled run

Symptom. Runs terminate with an authority outcome rather than executing.

The three outcomes mean different things:

Outcome	Meaning	Response
denied	Current member/grant, workload revision, target access, resource registration, policy, quota, approval, audience, claim, or fence rejected it	Preserve the block. Correct current authority. Never retry with workload identity
failed	Attestation, DPoP, transport, policy, or resource-guard validation failed	Verify fixed audience, clock, projected-token rotation, active registrations, proof-key binding, and guard version. Queued work stays durable
unavailable	Database, Core component, acceptable attestation profile, registered guard, or fresh readiness report is absent	Keep all new Task boundaries fail-closed until the authoritative path recovers

Only failed is a bug hunt. denied is the system working.

Use only safe identifiers and categories when investigating: correlation ID, opaque grant/run/claim/effect/binding references, workload/resource revision, fence digest, capability family, component, profile, and closed reason code. Do not print the projected assertion, DPoP key or proof, capability, nonce, credential lease, provider secret, request body, prompt, model output, or raw provider error.

Turning a task on, or a chat background launch, fails on consent

Symptom. Turn on in Tasks answers 503, or a chat turn's run_in_background / create_loop call fails, and the model's later attempts in the same turn fail the same way.

Kaizen approves the member's Core consent intent with that member's own live session. When the approval cannot complete, the error code says which side failed. It appears on the chat.tool.write audit row as commit_raised: <message>, in the api-backend warning pi tool failed: … tool=manage_tasks … error_code=<code>, and as the HTTP error code on the Tasks route:

Code	Meaning	Response
consent_session_unavailable	The member's request reached Kaizen without a platform bearer, so there is no session to approve with. Kaizen-side	Check that the platform edge forwards the member's Authorization or X-Kamiwaza-Access-Token to the extension
consent_url_invalid	Core's intent response carried no usable one-time consent link: it is missing, is not an absolute HTTPS URL, carries a fragment, or does not hold exactly one consent nonce	A Core contract or platform URL problem; compare the platform's advertised public URL with the one Core returns
consent_request_failed	Core refused or failed the exchange. The message names the step (view or decision) and either the HTTP status or the transport error class, or says the view returned no usable CSRF token	A 401/403 is Core rejecting the member's session or Kaizen's workload on its consent surface. A 5xx or a transport error is Core or the network being unavailable
consent_pending	A pending intent from an earlier or concurrent attempt holds the only consent link	Wait for it to settle or expire; a retry after expiry starts a fresh intent

The message never carries the response body, the consent link, the nonce, or a credential.

A failed approval forgets the intent it could not approve, because Core hands out the one-time link only when an intent is created. The next attempt creates a fresh intent and reports its own failure instead of a missing link. Only the request that created an intent forgets it, and only while it is still pending. When Core's answer to the decision itself was lost or was a server error (no response, or any 5xx from Core or a gateway in front of it), the intent is kept, because Core may already have approved it; the next attempt reads its outcome. A decision Core answers with HTTP 500 is therefore followed by consent_pending until the intent settles or expires. An attempt that finds a pending intent it did not create answers 409 consent_pending instead of approving or forgetting it. That is either a concurrent activation or an intent kept after a lost answer. It clears once Core settles the intent. Once it is past its expiry and Core still reports it pending, Kaizen replaces it with a fresh one; a record past its expiry that Core reports approved is kept and used. Inside one chat turn, a retry with different wording settles the turn's first request again and reports that failure. It does not report reused input. Once the first request can start, different work in the same turn is still refused as idempotency_conflict, because one turn starts at most one background job.

Do not retry from Kaizen to work around consent_request_failed with a 401 or 403. Retrying spends turns to reach the same refusal. Fix the platform side that refused it.

Tasks disappear after a platform dependency change

Symptom. Retained history remains visible, but the Tasks rail, route, scheduler, worker, model, credential, and resource operations close together.

Cause. Kaizen received an absent, stale, degraded, unavailable, or incompatible aggregate readiness report. A missing capability family, unhealthy attestation profile, resource-guard mismatch, revocation, or Core component loss intentionally has the same fail-closed result.

Check. Inspect the server-side readiness categories and Core audit correlation. Confirm the report contains exactly the required v1 families, all component statuses are ready, an acceptable attestation profile is healthy, model and dataset registrations are compatible, and the validity window has not expired.

Fix. Restore or reconcile the failed platform dependency and wait for a fresh aggregate report. Do not enable a local fallback, weaken the acceptable profile set, fabricate a report, or bypass the browser gate. Recovery resumes only safely retryable queued work; an effect already marked ambiguous remains terminal.

Kaizen keeps the request that observed a transport failure closed, discards that failed readiness transport, and builds a fresh one on the next check. This re-resolves Core after a head pod or service endpoint rotates; a healthy replacement should therefore restore Tasks without restarting Kaizen. If repeated checks still target a removed address, inspect service discovery and mesh routing rather than recycling workload identities or registrations.

Members wait then fail to run code

Symptom. run_code waits about 30 seconds, then fails to acquire.

Cause. The pool ceiling is too low for concurrent demand.

Fix. Raise SANDBOX_POOL_MAX_ALLOCATED. Raising SANDBOX_ACQUIRE_WAIT_SECONDS only makes the failure slower.

Size from measured concurrent execution demand, not member count. Remember artifact runs and package tests also take leases.

Run code says a package could not be installed

Symptom. A member's run_code fails within seconds with "the package name could not be installed because the package registry is unreachable".

Cause. The code imports a package the sandbox image does not bake, and the configured package registry cannot be reached — the normal state of an air-gapped installation that kept the public registry defaults. The code never ran, and the member's workspace files are unchanged.

Fix. Nothing is broken. Use a baked package or the standard library, or point SANDBOX_PACKAGE_REGISTRIES at an internal mirror. See Air-gapped installations.

First code execution is always slow

Cause. The warm pool is exhausted, so every first execution pays a cold create.

Fix. Raise SANDBOX_WARM_POOL_MIN, and check SANDBOX_BURST_IDLE_TTL_SECONDS if surplus capacity is being reclaimed too eagerly between bursts.

Sandbox workers churn constantly

Cause. Reset failures. A sandbox is reused only when the reset can be proven clean; a failed reset evicts the worker.

The eviction is correct behavior. The churn is the signal — investigate what is failing to reset rather than trying to make reuse more permissive.

Ingestion delays task dispatch

Cause. Document work and task dispatch share a queue.

Fix. Enable DOCUMENT_WORKER_SEPARATE_QUEUE_ENABLED. This is precisely the symptom it exists to fix.

Two members get different search results

This is correct. Every connector call runs as the signed-in member, and retrieval enforces access inside the query. Two people legitimately see different results for the same query.

Before investigating, confirm the difference is not simply the product working.

Search says "partial"

This is also designed behavior. If one independent provider is unavailable, Search labels the set partial and returns what the others authorized rather than failing the whole query.

An occasional partial label is not an incident. A persistent one names a provider worth checking.

A member cannot see a chat they expect

Work through the access paths:

1. Are they the owner?
2. Do they have a direct grant?
3. Are they an active member of a linked Project?

Effective access is the union. Access ends only when the removed path was their final path. Also confirm the Projects flag has not been turned off, which removes the Project-derived path entirely — resolve any Project-only viewer before disabling it.

A turn fails with "the assistant is temporarily unavailable"

Symptom. Every chat turn ends with the unavailable message, and the journal shows agent_run_failed with runtime_failed rather than any model output.

Read the upstream status, not just the code. The api-backend log carries pi model proxy denied or unavailable: code=... status=... upstream_status=.... The upstream status is the diagnostic:

upstream_status	Code	What it means
401	model_credential_rejected	The platform refused the credential itself — absent, malformed, or expired.
403	model_access_denied	The credential authenticated, but the caller was not entitled to that model.

A platform reply of Authenticated workroom context missing under a 403 means a workroom assertion reached a route that does not accept one. Kaizen does not send a workroom on model calls, so seeing this points at something re-adding it — check anything that builds model-call headers without going through the shared forwarded-header set.

A turn ends saying it lost contact with the Kaizen service

Symptom. A turn stops partway through with:

I lost contact with the Kaizen service partway through this turn, so I stopped instead of guessing at the rest. Please try again; if it keeps happening, an administrator can check the service's health.

What happened. The agent runtime calls back into the API backend between model steps. Idempotent callbacks (the step-snapshot binding, tool discovery, and session reads) retry a fast 502, 503 or 504, or a refused or reset connection, up to three attempts, within one overall budget of three seconds, and stop early if the turn ends. A failure that took longer than that budget (typically a gateway 504 that arrived after its own timeout) is not retried, so a slow outage is named rather than tripled. The turn ends with this message when such a callback is still failing.

Callbacks that save part of the turn (session appends, compaction, tool execution) are never retried, because the write may have committed behind the lost response. When one of those fails at the gateway, the turn says it could not tell whether that last step was recorded and asks the member to check the conversation before asking again.

Diagnose. The agent-runtime log names each failed attempt as [kaizen-callback] <callback> returned <status> (attempt N), or was unreachable (<socket error code>). The api-backend log carries the terminal frame, kaizen callback unavailable: <callback> returned <status> after N attempts. The backend's own callback routes never answer 502, 503 or 504; when the api-backend shows no matching request, the failure is in the service mesh or the network between the two services, not in Kaizen.

A turn ends saying the conversation is too long for the model

Symptom. A turn that had been working ends with:

This conversation is now too long for the selected model to read in one turn. Start a new conversation, remove some of the attached content, or choose a model with a larger context window.

This is a bound doing its job, and it replaces a wrong answer. This cause used to present as "the assistant is temporarily unavailable", which was both inaccurate — nothing was unavailable — and unactionable, because retrying the same conversation produced the same failure. Two paths now end here. Kaizen refuses a prompt that cannot fit before it is sent, answering the internal model route with 413 context_length_exceeded naming the measured prompt size and the model's input budget; and a provider rejection that names a size problem is no longer replayed, because the replay changes streaming, never the prompt.

Where the budget comes from. The input budget is the selected deployment's declared context window, less the reserved output and CHAT_CONTEXT_SAFETY_MARGIN_TOKENS. The window is resolved per request: the deployment's declared window first, then KAIZEN_MODEL_CONTEXT_WINDOW_OVERRIDES, then KAMIWAZA_MODEL_CONTEXT_DEFAULT_WINDOW_TOKENS. A model whose serving layer declares no window falls through to that default, and the warning context window for platform model … did not arrive with the served model says so at the time — treat a run of these refusals on one model as a missing declaration rather than a member problem.

Operator signals. Each refusal writes an audit row with outcome context_overflow and error class ContextBudgetExceeded, carrying the prompt hash and never the prompt, and increments kaizen_llm_request with outcome="context_overflow" and failure="budget".

A turn ends saying the model kept returning empty replies

Symptom. A chat turn stops early and the member is told the model kept returning empty replies. The run frame carries provider_empty_response_exhausted.

This is a bound doing its job. The provider answered with HTTP success but no text and no tool call, repeatedly. Every such call is logged as a success, so the condition leaves no error anywhere; what it leaves is a run that is busy and going nowhere.

Two separate bounds emit this code, and they have different tells in the model-call audit:

• The same request repeated. Prompt size stops moving while responses collapse to a token or two. The retry budget for that payload is spent and latches rather than re-arming, because replaying a request the provider has already answered with nothing cannot produce a different result.
• The request drifting while the replies stay empty. Prompt size keeps changing, so nothing looks frozen — the tell is instead a run of consecutive calls whose response length is one or two tokens. A turn ends after twice the configured retry count's worth of these, regardless of how much the payload moves.

Before these bounds existed, either condition spun for as long as the run was allowed to live and ended in the generic unavailable message.

What to check. Look at the model and the reasoning effort on the failing agent first; a model that returns nothing for one prompt shape usually keeps doing so. A prompt that is very large relative to the deployment's window is the common trigger, so compare the request size against the model's context window before assuming a provider outage. If a specific deployment produces this across several agents, it is worth checking on the provider side; a single agent reproducing it points at that agent's prompt or tool grant.

A turn ends saying a tool kept failing the same way

Symptom. A chat turn stops early and the member is told a tool kept failing the same way with the same input. The run frame carries repeated_tool_failure, and the journal shows the same tool settling with outcome: error several times in a row, each with the same error class.

This is a bound doing its job. The model re-issued one tool call with byte-identical arguments after it had already failed, and the sidecar closed the turn on the third identical failure. Only calls that actually ran count: a call the sidecar denied because the captured step had not advertised the tool never executed and carries its own retry guidance, so binding the tool and retrying the same arguments is not a repeated failure. Before this bound existed the loop ran until the tool-round ceiling ended the turn, and that ceiling's answer asked the member to continue — which resumes the same loop.

What to check. Read the error class on the failed step in the transcript. An invalid_argument class means the tool rejected what the model sent, so the fix is in the request or the agent's tool guidance, not in retrying; a timeout or provider_error class points at the service behind the tool. Retrying the turn unchanged fails the same way.

A turn fails with "the model provider is temporarily unavailable"

Symptom. Chat returns the bounded provider-unavailable message, and the member sees nothing about why.

This is the normalization working. The provider's own wording never reaches a member. The cause is in the api-backend log instead: search for provider_error_message, which appears on every one of these warnings, and read provider_status and provider_error_type next to it. A bridge retry that also failed logs its own line — platform non-streaming bridge retry did not return a completion — carrying the retry's HTTP status and response body.

Read the status first. A 5xx, an unnamed status, and the 4xx statuses that are themselves transient — 408, 425, and 429 — are replayed once before the gateway gives up, so two upstream lines appear. A recurring one names a provider or deployment worth checking; a run of 429s is a rate limit, not a defect. Any other 4xx is a rejection of the request itself and is answered from the first attempt, so exactly one line appears and the provider's message names the parameter or combination it refused. The exception is a rejection that blames streaming, which is replayed because turning streaming off is what the replay changes. A structured 402 arrives as its own credit category with member-facing guidance rather than as a generic failure. A provider that names a 402 only in its prose is replayed first, so that one shows two lines: the gateway will not build member billing guidance out of provider text it has not had confirmed.

A scheduled or other server-side run logs a different line. Those turns run under delegated authority and take no bridge replay, so their failures read no bridge replay is available and carry the same provider fields. One upstream line is expected there in every case, transient or not.

Model calls hang, then fail as model_unavailable

Symptom. Chat and scheduled runs fail with model_unavailable and upstream_status=None while the selected deployment is current and DEPLOYED, and the same model answers when called from outside the cluster.

Cause. The platform transport chain resolved to the deployment's own public origin, so every server-side model call leaves the pod for its own ingress and hangs there. TASKS_PLATFORM_GATEWAY_URL resolves to KAMIWAZA_PLATFORM_GATEWAY_URL and then to KAMIWAZA_ORIGIN, so an empty value is not "no gateway" — it is the public origin.

Fix. Look for the boot warning platform_model_route_is_public_origin; it is logged by every service when the chain lands on the public origin inside Kubernetes. Point the deployment at the platform's internal transport origin. Which setting carries it depends on how the deployment was created. Where the platform operator manages the extension, the operator injects KAMIWAZA_PLATFORM_GATEWAY_URL and the fix is to make sure it is configured. On a namespace-admin install using the scheduler's direct apply path, nothing injects it, so set TASKS_PLATFORM_GATEWAY_URL explicitly on api-backend, background-worker and document-worker — all three make server-side model calls, and a value set on only the API leaves both workers on the public route. An internal origin is safe: the model's registered authority still supplies the Host header and the TLS server name, so authorization and certificate validation are unchanged.

An external model endpoint sees more requests than turns

Symptom. A rate-limited external endpoint reports load that does not match the number of member turns.

Cause. Composer-time prewarm issues a speculative request per admission for eligible models, before the member sends anything.

Fix. Turn off SPECULATIVE_PROMPT_CACHE_ENABLED. That is the narrow control for prewarm load, and it leaves the cache discount on the turn itself in place. You only see this symptom on an installation that turned PROMPT_CACHE_ENABLED on, since prewarm needs it; turning that back off also stops prewarm, but it gives up the discount too.

A turn fails with "This run could not be completed" mid-answer

Symptom. A turn that was visibly working ends failed, and the journal shows agent_run_failed with reason input_lease_lost.

The run stopped renewing its input lease. While a turn runs, its runner renews a PostgreSQL lease every third of SHARED_CHAT_INPUT_LEASE_SECONDS, at most every 15 seconds (so every 15 seconds at the default 180-second lease). When a renewal cannot land, the runner stops the runtime and ends the turn rather than risk a second process executing it. The api-backend log carries conversation runtime lost its input lease for the room. A single renewal that cannot be written is enough: the usual causes are an event loop starved long enough to let the lease expire, or one failed PostgreSQL write, even a transient one. A single occurrence under load is recoverable by sending the message again; a recurring one is a capacity or database connectivity problem worth chasing.

A chat stays busy long after its process died

Symptom. A room still shows a running turn minutes after an api-backend pod was killed, and nobody has reopened it.

Wait for the background worker. Every minute it recovers turns whose lease expired more than one further lease period ago, and logs worker.conversation_runs.recovered with the room ids. A turn that was already running is salvaged; a turn claimed but never started goes back to the queue. With the default 180-second lease, a dead run therefore clears within about seven minutes. The worker does not execute turns, so a message still queued in that room starts when a member next opens or writes to the room. A room that never clears points at the worker: confirm it is running, and search its log for worker.conversation_runs.recovery_failed, which names rooms whose recovery raised.

A queued input can also remain blocked after a Desktop process loses its local admission. Its original submitter can cancel it through the authenticated member control, including an older input without a Desktop marker. Recovery settles that exact input under the room writer lease and publishes the cancellation; another member cannot cancel it. The worker never executes queued inputs or treats age as permission to cancel them. A remaining input still needs its own fresh local admission when the Desktop authority requirement is enabled. The durable cancellation reason is desktop_owner_cancelled_queued only when the input carries that requirement. Otherwise it is owner_cancelled_queued: an unmarked input, including an older Desktop input, does not prove its origin.

Everything feels slow

Check PostgreSQL first: connection saturation, slow-query profile, vector index memory. Almost every latency problem in Kaizen eventually reduces to the database.

Then check model round-trip latency at the gateway. Adding backend replicas does not make a turn faster; it makes more turns possible.

Cluster teardown leaves storage pinned

Symptom. A dev cluster teardown reports success but leaves substantial disk consumed.

Cause. Teardown can swallow removal errors while storage-wedged workers pin volumes.

Fix. Verify volumes are actually released after teardown rather than trusting the exit status, and clean up wedged workers before reprovisioning.

The app URL returns the platform dashboard instead of Kaizen

Symptom. /runtime/apps/<slug> renders the platform's own page, while /runtime/apps/<slug>/ — the same path with a trailing slash — loads Kaizen.

Cause. The published dynamic route matched only the prefix form, so the bare base path was never claimed and fell through to the default frontend route. The same shape affected model base paths.

Fix. This is route publication, not an application fault: nothing in the deployment is unhealthy, so do not restart Kaizen looking for it. Confirm the platform release publishes an exact-match companion route for each dynamic route, and check that the route for this deployment id is actually present in the live routing bundle rather than only in the generated source.

Sandbox leases stop renewing after workers start

Symptom. Sandbox creation succeeds and warm workers appear, then every lease renewal logs a forbidden error against the sandbox-controller service account, naming pods.

Cause. The platform generates the sandbox-manager Role. A generator that grants patch on services but only get, list, watch, create and delete on pods is sufficient to create a sandbox and insufficient to renew its lease, because renewal patches pod metadata. The failure is therefore invisible until the first worker exists.

Fix. Inspect the generated *-sandbox-mgr Role and confirm pods carries patch. Kaizen cannot widen its own RBAC; a platform release that omits the verb needs the platform fix, and patching the live Role is a stopgap that the next reconcile may revert.

Sandbox creation fails against the cluster API

Symptom. The sandbox controller reaches its Kubernetes API override path and still cannot create sandboxes, either with a connection failure or a certificate verification error.

Cause. Two independent prerequisites on a namespace-admin install. Egress policy may be enforced after Service DNAT, in which case allowing the API service VIP alone is not enough and the real control-plane addresses must be allowed too. Separately, the projected service-account CA does not necessarily validate an external API endpoint, so the controller needs the platform trust bundle instead.

Fix. Allow the control-plane endpoints the cluster actually resolves to, and point the controller at the external API URL with the mounted trust bundle as its CA. Both are platform-injected settings, not Kaizen configuration.

Outbound model calls fail certificate verification

Symptom. Model endpoints protected by a site CA fail TLS from Kaizen pods even though the platform's own services trust that CA.

Cause. Direct-runtime extension workloads are created outside the chart-owned pod templates that mount the trust bundle, so the injection has to happen on the path that actually creates them.

Fix. Confirm the platform release injects trust material into extension Deployments and Jobs, then check a running Kaizen pod for the merged bundle and the CA environment variables before investigating Kaizen's own TLS handling.

A turn says Kaizen can't verify the platform's TLS certificate

Symptom. Every chat turn ends with an answer that starts "Kaizen can't verify the Kamiwaza platform's TLS certificate" and says the turn never reached the model, then names the fix that matches why verification failed:

Logged reason=	What the answer and the log say to do
An unknown issuer (unable to get local issuer certificate) or a self-signed certificate	Add the platform's CA certificate to the kamiwaza-edge-trust ConfigMap, then redeploy Kaizen.
certificate has expired	Renew the expired certificate in the platform's chain: its own, an intermediate, or the CA. If the CA was renewed, publish the new one in kamiwaza-edge-trust and redeploy Kaizen.
Hostname mismatch or IP address mismatch	Reissue the platform's certificate with the logged host in its subject alternative names, or correct the platform or model URL that names that host. For a model call the logged host is the model's registered authority, which can differ from the gateway Kaizen dials.
Anything else	Check the platform's TLS certificate against the logged reason.

The api-backend log carries platform TLS certificate verification failed: host=<platform host> code=platform_tls_untrusted reason='<OpenSSL reason>'; <remedy> at ERROR, at most once per host and reason every five minutes in each api-backend process, and each refused call logs pi model proxy denied or unavailable: code=platform_tls_untrusted. The default Turn Engine does not retry the refused model call within the turn, because the certificate cannot change mid-turn; the compatibility runtime keeps its configured provider retries.

Cause. The model call to the platform failed certificate verification against the trust bundle Kaizen reads from SSL_CERT_FILE. The common case is an install whose edge certificate is not issued from the internal CA: the platform's edge anchor must be published in the kamiwaza-edge-trust ConfigMap (key ca-certificates.crt) so Core folds it into the trust store it mounts into extension pods. When that ConfigMap is missing, the bundle cannot verify the platform's own origin. This used to present as "the assistant is temporarily unavailable", which hid the cause and invited a retry that cannot succeed.

Fix. Follow the row for the logged reason. For a missing anchor, create the kamiwaza-edge-trust ConfigMap with the platform CA as the install documentation describes, then redeploy Kaizen so its pods mount the rebuilt bundle. Other connection failures (refused, DNS, timeouts) keep the generic unavailable answer.

A superseded deployment keeps a Pending pod alive

Symptom. An older Kaizen runtime's backend stays Pending and reappears after deletion, consuming quota while the current runtime serves traffic.

Cause. The superseded deployment still desires a replica while the volume it claims no longer exists, so its pod can never be scheduled.

Fix. Scale the superseded deployment to zero and let the platform garbage collect the runtime. Confirm which deployment is current first — a Pending pod next to a healthy one is not evidence of which is which.

A pod will not start after a hand-patched deployment

Symptom. A service stops starting with configmap references non-existent config key, without any repository change that could explain it.

Cause. A generated deployment was patched by hand to mount a key that the declarative source never owned. Once the source evolves, the manual mount becomes unresolvable and breaks the next pod start.

Fix. Remove the live-only volume, mount and checksum annotation so the deployment matches its declarative source. Runtime state for a deployed extension belongs on the extension resource, never on a patched Deployment: the operator reconciles that drift away without rolling the pod, so the patch reads as applied while nothing changed.

Related

• Observability — the leading signals
• Scaling & performance — the symptom-to-knob table
• Task scheduler operations