Observability
Logs, metrics, and traces — what Kaizen emits, how it is redacted, and which signals predict an incident rather than describe one.
AI metrics command center
Administrators can open Admin area → AI metrics for a content-free production snapshot. The backend combines the API process's canonical Prometheus registry with six-hour aggregates from the append-only LLM and audit ledgers, task runs, connector state, and indexed-source revisions. Each source is queried independently. A failed source produces Not reporting and partial coverage; it never becomes a fabricated zero or healthy result.
Browser polling cannot fan out into unbounded database work. Each API process
serves a 30-second single-flight snapshot cache, so concurrent administrators
share one refresh. A refresh runs at most two source queries concurrently;
each query uses a read-only transaction, a 1.5-second statement timeout, and a
250-millisecond lock timeout inside a two-second end-to-end deadline. Task-run
aggregation restricts the planner to the six-hour window plus currently live
states instead of scanning historical runs. Time-window and live-state scans
use indexes built concurrently, so installing the dashboard does not block
append-heavy ledger writes. A timed-out source becomes partial evidence and
cannot hold a request or scarce database connection indefinitely.
The query cost itself is exported as
kaizen_ops_metrics_source_query_seconds{source,outcome} to Prometheus and OTLP.
The Health tab is organized around the questions an operator must answer during a shift: Is Kaizen healthy now? Are members affected? Which request path is degrading? Will capacity, queues, or freshness fail next? Can the dashboard's own evidence be trusted? The highest-severity measured objective leads the page, followed by the production paths that contribute to those objectives. Compact means removing unused tracks and uneven card gaps, not shrinking operational copy below a readable size. A successful query with no observations is an observed zero; a failed query remains Not reporting.
The command center covers nine bounded signal families:
- Member request-path success and tail latency.
- Database, scheduler, queue, object-store, extractor, sandbox-controller, and model-platform dependency readiness.
- Model-route reliability, latency, throttling, capacity, and token volume.
- Connector health, indexed inventory, freshness, and retrieval effectiveness.
- Task outcomes, tail latency, pending age, approval waits, tokens, and attributed cost.
- Tool and sandbox execution reliability, acquire latency, and queue pressure.
- Signal-only member feedback and production-evaluation score and coverage.
- Model tokens and cost recorded by durable task runtimes.
- Prometheus, durable-ledger, and OTLP evidence-source confidence.
Request-path success measures requests that avoid a server-side 5xx response; expected authorization denials and other 4xx responses do not lower the service-reliability objective. Connector health is based on indexed sources that can currently serve retrieval. A connector with durable chunks remains serving while a refresh is in progress, while a configured connector that has never produced an index is disclosed separately and excluded from the health denominator. Retrieval p95 remains visible immediately, but it is not scored against the objective until the API process has observed 20 searches. This prevents a handful of cold-start requests from becoming a critical tail-latency claim.
The Pi runtime tab shows the Pi 0.84 execution layer without creating a
second telemetry service. It reads the same six-hour /ops/metrics snapshot
and reports instrumented-run coverage, typed span volume and error rate,
provider-request and whole-run p95 latency, and a per-operation p50/p95 table.
The closed operation set covers provider requests, assistant steps, tools,
lifecycle hooks, compaction, journal checkpoints, turns, and runs. A Pi run
from the reporting window that predates instrumentation makes coverage
Partial instead of being silently omitted.
Pi itself supplies telemetry schemas and exporter plumbing, not a bundled web
dashboard. The tab's overview-card and operation-table concepts are adapted
from oh-my-pi's MIT-licensed omp-stats dashboard.
Kaizen does not copy omp-stats' local JSONL scanner or SQLite database.
Pi spans enter the existing chat.agent.run audit funnel and PostgreSQL remains
the only durable aggregate authority. This avoids a process-local dashboard
that would disagree after a restart or horizontal scale-out. A Pi-run-only
partial audit index bounds the candidate ledger rows on busy installations
without indexing unrelated audit actions.
The sidecar projects only Pi's declared schema names, closed status values, bounded durations, and a reviewed scalar-attribute allowlist. The Python boundary validates that projection again. Prompts, response text, reasoning, tool arguments and results, credentials, and raw run, session, turn, operation, or tool-call identifiers never reach the Metrics API. Identifiers needed for in-turn correlation are digested before they leave the sidecar.
Provider failures carried inside an otherwise successful HTTP streaming response are normalized before they reach Pi and finalize the LLM ledger as an error, not a transport success. The ledger retains only the fixed safe error class; provider messages and account detail are neither audited nor displayed.
Some metrics are intentionally visible before their producer exists. Production evaluation coverage, for example, reports Not reporting until an evaluator calls the bounded quality metric setter. That gap is an operator fact. The dashboard does not infer groundedness from feedback or estimate direct-model spend without an authoritative pricing record.
The separate Usage tab answers product-operating questions. Its aggregate trends remain anonymous. A meaningful active day is one on which a member submits a message or steering input; login, presence, refreshes, worker activity, and background model calls do not count. The tab reports UTC DAU, WAU, MAU, DAU/MAU stickiness, new and returning active members, conversations, completed inputs, median time to first completed input, exact-day D1/D7/D30 cohort retention, and a daily activity series. It also reports user-attributed model calls and token distributions, tool and sub-agent adoption and success, approval outcomes, grounded-answer coverage, and feedback ratios. Hovering, tapping, or keyboard-focusing a daily activity point updates a dedicated exact-value strip above the chart with its active users, input outcomes, model calls, and token totals. The strip stays outside the plot so it never covers another selectable day. Each bar carries the same figures, with the same labels and in the same order, as its accessible name, so a screen reader announces exactly what the strip shows — including a day whose counts are all zero.
Generation and embedding traffic are ranked separately. This prevents a high-volume embedding deployment from being presented as the most popular answer model. Up to five models in each class carry call share, success, and token totals. System and background calls are excluded from per-active-user consumption. Model-source availability is reported separately from the overall snapshot, so a failed model aggregate reads Not reporting while a healthy query with no calls reads as an observed zero.
The browser can select only the reviewed 7-, 30-, and 90-day windows. Each window has its own 60-second single-flight cache. Four read-only aggregate queries fan out with at most two running concurrently and inherit the command center's statement, lock, and acquisition deadlines. Two concurrent partial indexes support activity-window and first-activity access without blocking the append-heavy conversation-input ledger during installation.
The Usage tab also has a separate Member operational usage table. It is
served by its own admin-role-gated endpoint and every view emits
admin.usage.members.viewed. At most 100 rows are ordered by token volume and
carry a display name, an opaque server-derived member key, active days,
conversation and input counts, completion ratio, model calls, token totals,
and no activity timestamps. Email-shaped display values fall back to the
opaque member label.
Neither metrics endpoint returns prompts, answers, conversation ids, source activity, tool arguments, raw platform subjects, emails, or model outputs. The audit event records only the viewing administrator, selected window, returned row count, and source availability; it never copies a member row. Model deployment names appear only in bounded administrator JSON tables and never as Prometheus or OTLP labels.
Logging
Kaizen uses a structured pipeline that owns the root logger. Third-party loggers — the web server, HTTP client, database layer — are routed through the same processor chain, so they emit the same shape, the same redaction, and the same service metadata. There is no second formatter to keep in sync.
| Setting | Default | Choices |
|---|---|---|
LOG_LEVEL | INFO | DEBUG … CRITICAL |
LOG_FORMAT | auto | json, console, auto (TTY → console, pipe → json) |
LOG_ACCESS_LOG_DROP_PREFIXES | /healthz,/health/readiness,/health/liveness | Comma-separated prefixes to silence |
SERVICE_NAME | kaizen | Static |
SERVICE_VERSION | installed version | Override per rollout |
SERVICE_ENVIRONMENT | development | development / staging / production |
Leave LOG_FORMAT=auto: readable locally, one JSON object per line in
containers and CI.
Chatty third-party loggers are demoted to WARNING regardless of LOG_LEVEL,
so DEBUG locally does not drown Kaizen's own events.
Schema migrations preserve loggers that the application has already configured. An in-process startup migration therefore cannot silently disable later runtime warnings, audit diagnostics, or connector failure records.
The health-probe filter
Without the drop-prefixes default, health probes are roughly 95% of log volume on a quiet deployment. Add a path to silence it; remove one to surface it. Matching is case-insensitive against the formatted access-log line.
Redaction
Every record passes a three-layer scrub before render:
- Secret-named keys — keys on the explicit secret allowlist, keys named
exactly
token/password/secret, or any key suffixed_token,_secret,_key,_password,_credential,_auth. Replaced with***REDACTED***. - Secret-shaped values — regex over every string value:
Authorization: Bearer …,api_key=/password:/token:assignments, AWS access key IDs, GitHub tokens, JWTs, PEM private-key markers, and URL-embedded credentials. For URLs the userinfo is replaced while scheme and host are preserved, so an operator can still see where the leak would have been. - Recursive — both passes walk nested dicts, lists, and tuples, so a secret inside an exception payload or an HTTP body fragment gets the same treatment.
Redaction is not a substitute for not logging secrets. It is the backstop for when something logs one anyway.
Metrics
The background worker exposes metrics on WORKER_METRICS_PORT (9100).
Prometheus and OpenTelemetry
The API exposes Prometheus at GET /metrics. Each worker exposes the same
process-owned registry on WORKER_METRICS_PORT (9100 by default). Enable
OTEL_METRICS_ENABLED=true to mirror every canonical observation from the API,
background worker, and document worker to OTLP over HTTP/protobuf. This includes
the existing queue, task, sandbox, collaboration, history, compression, and chat
pipeline families as well as the command-center families.
The exporter reads the standard OpenTelemetry variables:
| Setting | Default | Purpose |
|---|---|---|
OTEL_METRICS_ENABLED | false | Opt in to the OTLP metric mirror |
OTEL_METRIC_EXPORT_INTERVAL_MILLIS | 60000 | Periodic export cadence |
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT | SDK default | Metrics endpoint, normally a Collector /v1/metrics URL |
OTEL_EXPORTER_OTLP_ENDPOINT | SDK default | Base endpoint when the metric-specific endpoint is unset |
OTEL_EXPORTER_OTLP_HEADERS | unset | Collector authentication headers; treat as secret deployment input |
Use an OpenTelemetry Collector in production. Export is fail-soft: exporter configuration or delivery failure is logged and its latest periodic delivery result is reflected in the command center, but never blocks a member request, worker job, policy decision, or audit write. Prometheus remains available when OTLP is disabled.
Kaizen 0.4 telemetry transition
Kaizen 0.4 makes a hard telemetry identity cut at the same boundary as the application deployment. It does not dual-emit the retired metric names. Update Prometheus recording rules, alert expressions, Grafana panels, SLOs, and log selectors before the first Kaizen 0.4 scrape:
| Before 0.4 | Kaizen 0.4 |
|---|---|
tomo_task_schedules_due | kaizen_task_schedules_due |
tomo_task_dispatch_lag_seconds | kaizen_task_dispatch_lag_seconds |
tomo_task_oldest_pending_seconds | kaizen_task_oldest_pending_seconds |
tomo_task_occurrences | kaizen_task_occurrences |
tomo_task_runs_current | kaizen_task_runs_current |
tomo_task_runs | kaizen_task_runs |
tomo_task_retry | kaizen_task_retry |
tomo_task_run_duration_seconds | kaizen_task_run_duration_seconds |
tomo_task_run_tokens | kaizen_task_run_tokens |
tomo_task_run_cost_microunits | kaizen_task_run_cost_microunits |
tomo_task_authority_exchange | kaizen_task_authority_exchange |
tomo_task_approval_waits_current | kaizen_task_approval_waits_current |
tomo_task_cancellation_latency_seconds | kaizen_task_cancellation_latency_seconds |
tomo_task_corpus_operation | kaizen_task_corpus_operation |
These are Prometheus family names. Keep the exporter's standard counter and
histogram suffixes (_total, _bucket, _count, _sum, and _created) when
translating a query. Structured logs likewise change the exact selector
service.name="kaizen" to service.name="kaizen".
The boundary is the first scrape after the old deployment is quiesced and Kaizen 0.4 starts. The old series receive no further samples. Instant queries and alerts that retain an old selector therefore become stale or no-data after Prometheus's configured lookback; range queries can still show the retained earlier history until normal TSDB retention removes it. Alert rules that treat no-data as healthy can silently resolve, while rules that treat it as an error can page. Validate both behaviors as part of the deployment transition.
The cardinality rule
Metrics use bounded status, trigger, result, and reason-category labels. They never label by owner, task, run, target, prompt, model output, email, or any other unbounded identity.
Do not add cardinality so a dashboard can identify a member or a task. Correlate aggregates with owner-scoped audit records and safe run diagnostics instead. This is a compliance property as much as a cost one.
Task metrics
| Metric | Meaning | Alert signal |
|---|---|---|
kaizen_task_schedules_due | Enabled schedules currently due | Non-zero beyond two dispatch cadences |
kaizen_task_dispatch_lag_seconds | Age of oldest due schedule | > 2 × TASKS_DISPATCH_CADENCE_SECONDS |
kaizen_task_oldest_pending_seconds | Age of oldest queued or retry-wait run | Sustained growth while worker is Ready |
kaizen_task_occurrences | Committed occurrence decisions | Rising blocked / expired / skipped |
kaizen_task_runs_current | Current runs by status | Queue growth or concurrency at cap |
kaizen_task_runs | Terminal runs by status and trigger | Failure or block ratio change |
kaizen_task_retry | Retry decisions by category | Repeated exhausted or non-retryable |
kaizen_task_run_duration_seconds | Terminal duration | p95 near the effective timeout |
kaizen_task_run_tokens | Terminal tokens | Sustained approach to caps |
kaizen_task_run_cost_microunits | Terminal cost | Sustained approach to caps |
kaizen_task_authority_exchange | Authority results | Any failed / denied / unavailable |
kaizen_task_approval_waits_current | Current approval waits | Unexpected growth or age |
kaizen_task_cancellation_latency_seconds | Request → terminal | p95 beyond the objective |
The signals that predict incidents
Most of Kaizen's failure modes announce themselves before members notice.
| Leading signal | What it predicts |
|---|---|
kaizen_task_dispatch_lag_seconds rising | Database pressure, before anything else shows it |
| Sandbox acquire wait creeping toward 30s | Pool ceiling about to start failing executions |
| Embedding latency approaching its budget | Search about to degrade to lexical-only |
Queued available_at age growing while workers are Ready | Broker handoff needs repair |
| Worker churn in the sandbox pool | Reset failures — eviction is correct, the churn is the signal |
| PostgreSQL connection saturation | Everything, shortly |
Alert on the leading signals. The lagging ones — error rates, failed turns — tell you it already happened.
Correlation
Records carry request-scoped correlation so a member report can be traced through the stack.
For a chat turn: conversation, input, and run correlate the journal event, the audit rows, and the runtime events. For a task: occurrence, run, and attempt correlate the ledger, the audit rows, and the member-visible run trace.
The member-facing trace and the audit trail are deliberately different records. Use the trace to answer "what did this run do?", and the audit trail to answer "who attempted what, and what was the outcome?".
Command-center API
GET /api/ops/metrics returns the same content-free Health and Pi runtime
snapshot used by the screen. It requires Kaizen App Admin authority in the
current workroom. It is not a Prometheus proxy and does not accept PromQL,
label selectors, member identifiers, or time ranges from the browser.
GET /api/ops/usage?days=30 returns the aggregate-only Usage tab snapshot.
days accepts only 7, 30, or 90; it never accepts a member, conversation,
team, or arbitrary segment selector.
GET /api/ops/usage/members?days=30 returns the separately authorized and
audited, content-free member operational table for the same bounded presets.
It accepts no member selector.
Health endpoints
| Service | Endpoint |
|---|---|
| api-backend | GET /healthz |
| seaweedfs | GET /healthz |
| document-extractor | GET /version |
| sandbox-controller | GET /health |
| frontend | its own healthcheck binary |
The api-backend allows a 15-minute start period because first boot runs migrations and verifies dependencies. A deployment stuck at 1/2 ready is usually a blocked dependency, not a crash — check object-store reachability first.
Upgrades reconcile two required columns for databases that used the original native assistant-profile migration history. A recorded attachment-input revision alone does not prove that corpus capture generation and retrieval-only document references were installed. The forward repair adds missing canonical schema without replaying earlier revisions or changing application rows. Existing compatible columns, values and constraints remain intact.
An incompatible column type, nullability, default, or capture-generation check stops the upgrade and rolls back the repair. Investigate the reported schema condition before retrying; do not rewrite migration stamps or coerce retained data to make startup pass. Downgrading the repair preserves these columns because the current migration ancestors require them.
Multipart and uploaded-blob histories join the forward repair through no-op merge revisions. Reversing only that merge splits the recorded branch versions while preserving repaired columns and retained rows; it does not undo the repair.
When TASKS_ENABLED=true lacks static delegated-workload registration
prerequisites, GET /healthz remains HTTP 200 and adds
"tasks_scheduler":"held". Once those prerequisites exist, it reports ok.
The probe never contacts Core, so live Task release changes do not make API
health flap. Task admission still checks Core's current readiness before work.
What is deliberately not observable
- Presence. Heartbeats, tab counts, devices, IP addresses, typing text, and last-seen history are not retained anywhere.
- Chat content in metrics or audit. Not a gap; a design decision. One narrow qualification, stated rather than left to be discovered: a provider that quotes part of the rejected request back in its own error prose — a refused tool argument, a fragment of the offending content — puts that quotation into the upstream-failure log line described below. Kaizen does not choose what a provider echoes, and the line is bounded and redacted, but a deployment treating server logs as free of chat-derived text should know it.
- Raw provider errors. Normalized into safe categories before they reach a member response, an audit payload, a task trace, or a metric. The server log is the deliberate exception: a model call that fails upstream logs the provider's error type, its message, and the status recovered from the error frame, each collapsed to one line and truncated at 512 characters. A bridge retry that does not return a completion logs its HTTP status and response body the same way, and a delegated run — which takes no bridge replay — logs the same fields against its own line. Without that, an upstream rejection is only diagnosable by capturing traffic inside the pod. The redaction passes above still apply, including to provider keys a provider echoes back in its own error prose, and none of this detail reaches a member.
- Member identity in metric labels. See the cardinality rule.