Conversation journal & durability

A private or shared web chat has one append-only conversation journal in PostgreSQL. It is the durable source for transcript order, attribution, replay, model context, membership and resource history, and run recovery.

Journal versus cache

The bounded turns cache is a convenience projection, nothing more. Compaction can shorten that cache; it never edits or shortens the journal.

This distinction is what lets Kaizen compact aggressively for the model while keeping a complete, ordered, auditable record for members.

Pi session projection

The Pi sidecar keeps a bounded session tree for rollback compatibility and compaction. That tree is a projection of accepted transcript state, not an independent journal. A restart reconstructs served model memory from the runtime journal first and can repair the Pi projection idempotently.

Pi 0.84 adds named lanes, operation records, shared sequence numbers, and mutable facts. Kaizen does not let the sidecar activate those writes on its own: doing so would create a competing order and recovery truth across replicas. They remain fail-closed until one backend transaction can assign their shared sequence, bind the active writer, append the runtime-journal correlation, and return an idempotent acknowledgement. This preserves the existing guarantee that ambiguous writes are never replayed merely because a sidecar restarted.

Positions, not timestamps

Browsers receive journal positions in strict journal order. A client remembers the last position it applied, ignores exact duplicates, and reconnects from there.

Valkey carries only short-lived viewing and typing leases, "the room head changed" hints, and bounded live artifact recovery material. Reconnect always re-authorizes active membership and fills any durable gap from PostgreSQL — a hint or transient artifact snapshot is an optimization, never the journal.

If a cursor is no longer usable, Kaizen reloads an authorized transcript snapshot before resuming live updates.

Each viewer has a bounded delivery queue. Publication never waits on a slow browser. If transient traffic fills that queue, Kaizen coalesces the newest durable head, emits stream_reset, and makes that viewer re-authorize and rebuild from the journal plus current artifact snapshots. Dropping an arbitrary suffix of state-changing deltas is not allowed.

What is projected, and what is not

Internal runtime events are a separate, more detailed journal. Kaizen projects only closed, member-safe facts into the conversation journal.

Never projected:

• hidden model reasoning,
• raw tool arguments or results,
• credentials,
• hidden prompts,
• presence.

A bounded member-visible Thinking trace is appended through durable agent progress as it is released. Kaizen commits the initial visible status, the first meaningful text, and bounded later checkpoints instead of one journal row per provider fragment, then folds the exact bounded trace into the terminal assistant message. This keeps the trace available when a run is paused at approval or the browser reloads before an answer without multiplying database writes by tokens and viewers. It is distinct from hidden reasoning, starts collapsed when rehydrated, and follows the same conversation ACL as the answer.

Tool activity is projected the same way, and the distinction matters: a room sees a tool's name, status, outcome, and result count, never the provider's own call identifier. On a failed step the room also sees a short error class — one word from a fixed vocabulary such as invalid_argument or timeout, carried inside the tool object beside the name — and never the error text, the arguments, or the result. An unrecognised class is dropped at the projection boundary rather than published. The transcript still needs a key to fold a tool's start and its completion into one row, so the projection publishes a digest of that call identifier bound to the run — stable for the pair, meaningless outside the room, and not reversible to provider-local identity. Withholding the key entirely was worse than it looked: the client had nothing to key a row on, so a room rendered no tool activity at all, live or on reload, while the journal named every tool that ran.

Approval requests and their approved or denied outcomes are projected as member-safe, non-model-visible facts. They carry the request identity, agent, tool, permission class, and canonical argument digest—not the arguments or credentials themselves. The decision is a later event caused by the request; neither event is updated when the operational request advances or is consumed.

A run has exactly one terminal conversation event, even when callbacks, timeouts, cancellation, salvage, or process recovery race each other. An assistant answer is staged until terminal publication. The answer and its completed terminal event commit in one database transaction, so a reload cannot observe a successful answer paired with a failed run. If an adapter stops after producing a usable answer, recovery commits that answer with a salvaged terminal outcome instead of replacing it with a generic failure.

A response rejected by a conditional stream-correction rule never becomes an assistant event and never enters the conversation projection. The runtime journal records only the fixed server-authored correction as a steering message before capturing the replacement provider step. Diagnostics retain the rule ID and counts, not the matching model text. Every newly accepted private or shared turn has a durable input identifier. Changing a room from private to collaborative therefore changes its audience, not its execution or replay model.

Three identities stay separate

Boundary	Owns	Does not grant
Collaboration scope	Active named membership, audience version, common policy, pinned shared-resource manifest	A principal, credentials, approvals, or a sandbox
Conversation stream	Journal order, input lane, selected agent, shared model context, presence	Access to another conversation in the same Project
Actor scope	The initiating member's connector ACL, credentials, approvals, audit attribution, sandbox	The owner's or another participant's authority

Collapsing any two of these would be the shortest path to a privilege bug, so they are represented separately in the data model rather than derived from one another.

Code execution follows from this

Because actor scope and conversation stream are distinct, code execution uses an actor-plus-conversation workspace key.

Conversation resources are mounted separately as a bounded read-only projection. /work, installed packages, environment, credentials, processes, and snapshots remain participant-private. A validated file explicitly generated for a shared turn enters the room through one metadata-only journal reference and the existing resource guard. Raw stdout, stderr, bytes, and unretained files do not enter the journal.

That linked resource is also the durable reload projection for the answer and Workspace. The journal does not duplicate native-file metadata inside the assistant payload; authorized transcript projection joins the current linked reference back to its originating turn.

Projects add a pointer, not a copy

A Project adds one versioned pointer from a grouping record to a conversation.

Effective access is the union of owner, direct-chat, and active Project paths. The pointer never copies or re-identifies the chat. Each conversation keeps its own journal, input lane, model context, presence, retention, direct grants, resources, and actor-scoped sandboxes.

That is why unlinking or deleting a Project destroys nothing: removing a pointer cannot remove the thing it pointed at.

Room content and search projections

The conversation journal is room content. Every current room member receives the same retained messages, artifacts, citations, and linked resources.

The same transaction that stores an index-eligible journal event writes a projection outbox row. Background workers then update or tombstone passage and Project indexes.

Those indexes are disposable projections. They never become access capabilities — a stale index row cannot resurrect a revoked grant, because the grant is re-checked when the anchor is opened.

Durability guarantees

Guarantee	Mechanism
No lost accepted input	Journal append with an idempotency key, before any model call
No concurrent model attempts on restart	The old lease expires before a replacement attempt can claim the accepted turn
No competing turns per room	One FIFO lane, positions in the journal
Exactly one terminal outcome	Terminal publication is idempotent and authority-checked
No answer/terminal split-brain	The staged answer and terminal event commit atomically
No stale success after cancellation	Cancellation is durable first; the run loses side-effect authority
No replay of an ambiguous write	Retry requires explicit no-effect evidence

Recovery deliberately separates observation from authority. Hydration may wake the exact active input in a private or shared room, but the backend re-resolves current write access and requires the authenticated member to be the input's original actor. The room's database lease still serializes execution across replicas. An old browser credential or inaccessible conversation cannot resume it, and ambiguous running model or tool work is salvaged from its journal instead of being blindly replayed.

The browser repeats that wake at a bounded interval while the active input remains visible, including in the tab that originated it. Each request is authorization-checked and idempotent; it can schedule only the existing accepted input after the prior lease expires. A successful wake is therefore not remembered forever in browser memory, because the process that accepted it may be replaced later. No wake appends a second member message or creates a second journal lane.

The only dispatched writes that reconstruction may repair as interrupted are exact artifact operations with an admitted durable checkpoint: ordinary create_artifact and update_artifact calls, plus the staged begin_artifact/append_artifact/finalize_artifact/discard_artifact transaction. Updates and staged writes commit their owner-, turn-, and call-fenced checkpoint before displaying partial content. A create admits its row only after the complete unique-key arguments prove the call ordinary; an earlier interruption fences the call without a row and is not resumable artifact content. Once a create row exists, an exact owner, logical-turn, and conversation match may rebind its single interrupted first version to a recovered provider's new call identifier and preserve the root. Multiple matches fail closed as ambiguous. Every other dispatched write remains a terminal ambiguity.

The logical run identity remains stable across that repair so a late terminal from the abandoned attempt still races through the single-terminal fence. Member-safe progress and answer projections from the replacement attempt use a new opaque attempt namespace, preventing harmlessly different provider frame boundaries from colliding with the append-only facts retained before the crash.

Conversation input acceptance stores the member message in its first append-only conversation event. If the run ends as cancelled, failed, or degraded, the next model call rehydrates that message with an explicit unfinished-turn boundary. Reconstruction reads the journal rather than mutable worker state and applies to semantic follow-ups, not a particular command word. A later completed terminal becomes the newest result and prevents stale interrupted intent from being injected again.

The runtime journal binds that stable, member-authored input separately from the provider-ready message. Server-authored document, artifact, routing, and orchestration context can legitimately be rebuilt to a newer snapshot after a crash; that enrichment does not turn the retry into a different logical turn. Recovery still feeds the provider the exact enriched message already committed to the journal, while a genuinely different member input fails the recovery fence instead of being substituted into the accepted turn.

Artifact content has a related materialized checkpoint. The internal runtime journal remains append-only; once an approved or ungated artifact call has enough server-validated identity, Kaizen opens the initiating actor's next owner-only artifact checkpoint and projects admitted deltas into that one mutable building row. An update to the actor's own root becomes its next version. An update targeting another participant's exact active room grant instead opens a new actor-owned root whose remixed_from points to that immutable source version. It never acquires write authority over the source. A create reaches its checkpoint only after its complete unique-key JSON object passes the reserved-language boundary; incomplete creates remain buffered and a terminal signal fences them without materialization. Stop retains an already-open checkpoint as interrupted, and a follow-up rebinds the fenced call identity and completes the same version or remix. Completed rows never mutate. This projection does not put raw tool arguments in the conversation journal, and unfinished rows are never made visible through publish or room-resource grants.

Finalizing a room remix appends its own attributed resource reference and terminal facts to the shared journal. Owner and collaborator therefore replay the same completed resource set while audit can distinguish the source owner, the initiating actor, and the exact lineage edge without rewriting an earlier message, artifact version, or grant.

Staged creation adds a sequence and content digest to that materialized row and stores a metadata-only receipt for each accepted append. The logical input and turn remain stable across recovery, but a replacement execution rotates a private fence. Only the current fence can append, finalize, discard, or rebind the draft; a delayed callback from the expired process therefore cannot mutate the recovered transaction. Finalization is idempotent at the exact terminal cursor and is the only point at which room linking occurs.

The room runner explicitly closes its upstream runtime stream after durable cancellation, including when cancellation arrives while a member-safe frame is being persisted. Closing that stream runs the runtime's interruption cleanup; aborting only the browser observer or only the input lease would otherwise leave an already-open artifact checkpoint in building state.

During an orderly SIGTERM or SIGINT, the backend stops accepting detached work, closes active room and prompt-chain producers, and gives artifact cleanup a bounded drain window. Accepted room inputs remain in PostgreSQL for fenced recovery. The agent runtime rejects new writes while draining and aborts its live runs so their owner callbacks can terminalize or checkpoint them.

A crash, OOM kill, forced pod deletion, node loss, or power failure cannot run shutdown code. In that case PostgreSQL is the recovery coordinator: expired input leases are fenced into resumable state, exact in-scope building artifact rows become interrupted, and a newly authenticated browser can wake the accepted input. This path never assumes the dead process cleaned up after itself. Artifact fencing is part of the same transaction that grants an expired lease, so an authenticated resume cannot race ahead of the periodic recovery sweep and create a second artifact root.

Nobody may come back to the room, so recovery cannot depend on a browser. Every minute the background worker looks for claimed or running inputs whose lease expired at least one full lease period ago (SHARED_CHAT_INPUT_LEASE_SECONDS). For each room it takes the same room writer lease a runner holds, skips rooms that a live runner still owns, and applies the same fenced recovery a draining runner applies: a claim that never started returns to the queue, an existing terminal is attached, anything else is salvaged from its journal, and the room's active input is cleared. The worker never executes a turn, so queued inputs wait for an authorized browser wake or the next message. Each room it changes records one member.conversation.run.recovered audit event carrying only those counts. The sweep carries no member authority, so it never resumes a run; the extra lease period leaves a returning member's authorized resume window intact. It then publishes the new room head so open viewers update without reloading.

Corpus Tasks add content-minimized facts for definition revision, manifest digest, phases, checkpoints, leases, retry categories, pause/resume controls, budget exhaustion, export publication, and final outcome. Source text, model answers, record properties, unrestricted locators, credentials, and hidden reasoning remain outside the Task journal. Coverage, provenance, and output bytes are separate owner-authorized projections.

A chat-promoted background job adds one immutable, owner-scoped correlation between the originating conversation and input, exact selected agent, request digest, Task, and run. It does not copy the member message, agent instructions, tool payloads, or output into that binding. The chat job card rehydrates bounded status from the Task projection after navigation or refresh; the output conversation, trace, usage, diagnosis, and artifacts remain canonical Task records and keep their normal authorization checks.

Prompt-chain execution uses the same durable-observer principle without pretending a prompt is a conversation. An interactive run has an owner-scoped, lease-fenced orchestration row and append-only event journal. The detached producer records each member-safe chain frame before an SSE observer can replay it. Browser disconnect therefore changes only observation; explicit cancel terminalizes the journal and fences execution.

When the same prompt chain runs from a Task, the Task definition, occurrence, run, and run-event journal remain authoritative for the entire chain. The Prompts library projects a pointer to that Task run beside interactive entries; it does not duplicate the output or manufacture a second terminal record.

Recovery

A tool callback, resumed model stream, timeout salvage, or terminal publication is rejected if it presents a different member, conversation, audience, agent, or resource scope than the run was authorized with.

Membership and policy are checked again before publication. If membership was revoked mid-run, the run does not publish — it terminates safely instead.

Retention

The journal is the retained record. Deleting a chat, unlinking it from a Project, or removing a participant does not rewrite history that already happened; it changes who can reach it going forward.

See Data handling & retention.

Related

• Anatomy of a chat turn
• Streaming & event contracts
• Data model & storage