Skip to main content

Backup & disaster recovery

What to back up, what is deliberately disposable, and the restore order that works.

What to back up​

StoreBack upWhy
PostgreSQLYesEverything authoritative: journals, Library, task ledger, audit
Object storeYesBytes that cannot be regenerated: uploads, exports, artifacts, workspace snapshots
ValkeyNoEphemeral by design; runs without persistence
Search indexNoA projection inside PostgreSQL; rebuildable by reindex

Two stores, both essential, and a clear rule for everything else: if it can be recomputed from PostgreSQL and the object store, it is not a backup target.

What backups inherit​

A PostgreSQL backup contains conversation journals — every message, in order. It is exactly as sensitive as the live database and must be protected to the same standard.

This is one of the out-of-band assumptions in the threat model: Kaizen cannot enforce how its backups are stored.

Kaizen 0.4 A2A identity-cut recovery​

All three forward identity-cut revisions from 0110 through 0112 refuse downgrade because they cannot reconstruct overwritten or post-upgrade-authored state honestly. Operational rollback means restoring a verified pre-upgrade backup into the retired deployment, never running Alembic downgrade in place.

Migration 0111_kaizen_admin_setting_keys deliberately deletes the eight prior-release A2A override rows and revokes every prior-release Agent Studio connection key while preserving its audit row. The migration refuses downgrade because it cannot reconstruct the prior credential state honestly. The only way to recover it is to restore a verified pre-upgrade PostgreSQL backup into an isolated copy of the retired pre-0.4 deployment.

Do not copy a recovered shared secret, inbound registry, or connection key into Kaizen 0.4. Generate new Kaizen credentials, re-register every peer, and explicitly set all five KAIZEN_A2A_*_ENABLED flags to the desired posture; they default to off when no canonical override exists, so a restore that sets none of them comes back with A2A closed.

Consistency between the two stores​

PostgreSQL holds metadata; the object store holds bytes. A restore that mixes eras produces dangling references in one direction or orphaned bytes in the other.

Prefer backups taken close together, and record the pairing. On restore, PostgreSQL is the truth: an object with no metadata row is unreachable and harmless; a metadata row with no object surfaces as an unavailable resource, which Kaizen already handles as a safe state.

Restore order​

  1. PostgreSQL — bring the primary up and verify it is reachable.
  2. Object store — restore and verify bucket accessibility.
  3. Valkey — start empty. Do not attempt to restore it.
  4. api-backend — allow the full 15-minute start period; migrations run here.
  5. Workers — start after the backend is Ready.
  6. Sandbox controller and workers — the pool rebuilds itself.
  7. Frontend — last; it depends on the backend.

Why Valkey starts empty​

Everything in Valkey is a lease, a hint, or a queue entry whose durable record already exists in PostgreSQL. Starting empty means:

  • presence disappears and re-establishes as members reconnect;
  • in-flight sandbox leases expire and workers are reclaimed;
  • committed-but-unenqueued task runs are visible as queued work and are recovered by handoff repair, not by replaying a Valkey backup;
  • stale uploaded-document rows still in the durable queued phase are re-enqueued with deterministic job identities by the recovery worker.

Restoring stale Valkey state would be actively harmful — it could reintroduce leases for workers that no longer exist.

After a restore​

  1. Trigger handoff repair. Committed queued runs and stale queued document ingests re-enqueue with deterministic job IDs; lease fencing ensures exactly one executor claims each. Do not delete ledger rows or mint replacement runs.
  2. Verify timezone-data identity across all scheduler replicas. A split version produces disagreeing occurrence decisions.
  3. Check dispatch lag — kaizen_task_dispatch_lag_seconds should settle quickly.
  4. Verify retrieval. If embeddings look wrong, confirm the embedding role points at the same deployment as before. A different embedding model makes existing vectors incomparable and requires a reindex.
  5. Verify the object store by opening a document preview and an artifact.
  6. Spot-check private and shared chat recovery. Open a conversation with an expired active lease. Confirm the worker records a resumable pause, exact unfinished artifacts are interrupted rather than building, and a currently authorized browser reconnects. Journal replay and membership must both resolve.

What a restore does not do​

  • It does not resurrect revoked access. Grants are re-checked at read time.
  • It does not replay notifications as new. Read state travels with the record.
  • It does not re-run missed scheduled work automatically. Recurring work skips missed windows by default; late one-time work expires. Explicit backfill is bounded by window and count.
  • It does not re-execute ambiguous writes. A write whose effect cannot be proven absent is terminal and never replayed.

That last point is worth planning around: after a long outage, some external side effects will simply not have happened, and Kaizen will not guess. Reconcile those deliberately.

Recovery objectives​

Kaizen does not impose an RPO or RTO — those come from your backup cadence and cluster provisioning. What the design gives you:

PropertyGuarantee
No lost accepted inputJournal append precedes any model call
No duplicate model request on restartStable input idempotency
No competing turns per roomOne FIFO lane, positions in the journal
Exactly one terminal outcome per runIdempotent, authority-checked publication
No replay of ambiguous writesRequires explicit no-effect evidence
Queued work survives broker lossThe ledger commits before enqueue

The last one is the load-bearing property for disaster recovery: because the database transaction commits occurrence, run, and cursor before the broker handoff, losing the broker entirely loses no scheduled work.

Testing recovery​

Test the restore, not the backup. A backup that has never been restored is a hypothesis.

A useful drill: restore to an isolated environment, start Valkey empty, run handoff repair, and confirm that a scheduled task fires, a queued upload is ingested, and a chat replays. Separately force-delete the API and agent-runtime pods during a disposable run-labeled turn. Recovery must succeed without a shutdown hook, produce one terminal answer, and leave no artifact version in building state.