Durable tasks: scheduler & worker

Kaizen keeps authored intent, scheduled opportunity, actual execution, and compliance evidence separate. This prevents an edited task or an inferred runtime path from rewriting what happened.

A scheduled task. PostgreSQL owns Kaizen's schedule decision; Core reserves and fences execution; the broker carries only a Core-issued opaque run reference.

Four separate records

Record	Holds	Mutability
Task definition	Owner intent: target, inputs, destination, schedule, limits, notifications	Mutable
Occurrence	One scheduled, manual, backfill, or test opportunity, with its exact calendar resolution and dispatch decision	Immutable
Run	One execution identity, current state, exact target version, bounded usage, outputs, safe diagnosis	Immutable identity
Run event	One append-only, gap-free, member-safe trace fact	Append-only
Chat job binding	Owner, originating conversation and input, exact agent identity, request digest, Task, and run correlation	Immutable identity
Goal input request	Exact run and Agent binding, question digest, Task-thread input reference, and later accepted member-input reference	Pending to answered or cancelled
Platform authority mapping	Opaque grant, run, claim, resource, and credential-binding IDs plus immutable revision/fence metadata	Append-only or compare-and-set

Run events reference existing runtime sources. They do not copy prompts, completions, tool payloads, or private reasoning.

Corpus work is child state, not another scheduler

An optional input_template.corpus turns an ordinary Task run into a bounded corpus run. The parent definition, occurrence, lease, authority, cancellation, retry policy, journal, and terminal result remain authoritative. Child tables hold the exact selector snapshot, input manifest, partitions, canonical records, lineage, export state, phase, control request, and checkpoints.

Execution advances through enumerate, map, reconcile, reduce, export, publish, and complete. Source enumeration is deterministic and independent of ranked retrieval. Each map call uses a stable partition identity. A worker restart replays the same conversation input instead of repeating a completed model turn. Only a completed response that fails strict JSON Lines validation advances a durable repair generation and receives a new input identity. The generation changes in the same fenced transaction that requeues the partition; claims and model-call ceilings use the stored run definition. Commits remain fenced and merge by declared identity and conflict rules. A terminal replay reads durable exports and never calls the model or publisher again.

One parent Task lease runs bounded partition executors concurrently when the operator raises TASKS_CORPUS_MAX_CONCURRENCY above its default of one. Claim admission, output ceilings, and counters serialize on the run row; partition leases renew during model calls. Database pages, model input, output parsing, coverage, and provenance stay bounded. JSON Lines and GraphML exports stream canonical records in stable order through the object store.

If no partition succeeds and the run fails, Kaizen completes the failed run without publishing an empty Artifact or export. Partial runs with committed partitions retain their published records and report incomplete coverage.

Collection corpus enumeration selects one ACL-visible head chunk per source before calculating byte and token totals from that source's ACL-visible revision. The source limit applies before those aggregates, avoiding a chunk-by-chunk join that multiplied work and inflated size estimates.

The graph Artifact contains a run reference, not cached source-derived records. The complete graph is served page by page from the run itself (GET /api/task-runs/{run_id}/corpus/graph, keyset-ordered by record kind and stable id). Every page is re-authorized against the member's current access to the sources each record came from: a record whose source was revoked after the run committed is withheld and counted, not served. Element counts no longer cap the graph — the viewer draws only what the viewport contains — and the JSON Lines and GraphML exports remain the canonical complete output.

PostgreSQL is the schedule authority

Browser time, browser timers, worker wall clocks, and Valkey are not schedule authority.

Database time selects bounded due rows with FOR UPDATE SKIP LOCKED, records deterministic occurrences and runs, advances the future cursor, and commits before any broker handoff.

Preview and dispatch use one IANA evaluator with explicit gap and fold policy and an exact deployed timezone-data identity. Each immutable occurrence preserves its UTC instant, local wall time, offset, fold, schedule revision, and timezone-data version.

Duplicate replicas and redelivery converge on deterministic occurrence and run identities. Occurrence uniqueness and one-run-per-occurrence constraints defeat duplicate wakes.

The worker registers its Task executor and scheduler wakeups from static rollout configuration, not from a one-time platform health snapshot. A fresh install may start the worker before delegated workload and protected-resource registration converge; that temporary hold therefore does not require a worker restart. Every wakeup and execution still re-evaluates current delegated readiness and remains fail-closed until Core reports the complete graph ready.

Production Task model calls use Core's guarded inference API at /api/serving/delegated-models/<deployment-id>/v1/chat/completions or /v1/responses. Core validates the run-bound capability and request proof, consumes the authorized effect before calling the already deployed engine, and settles its usage afterward. This covers host-served engines as well as replica-backed deployments without handing delegated credentials to the engine or changing its lifecycle. Draft tests and interactive calls retain their member-authenticated model route. A Task needs a Core build exposing this guarded API; a successful draft test alone does not prove production access.

Chat adapts intent; the task service stays strict

The chat tool publishes the exact canonical rule shapes for one-time, interval, and cron schedules. A small adapter accepts bounded conversational shorthand — such as a one-time starts_at value or interval seconds — and canonicalizes it before schema validation. A simple daily UTC cron expression is stored as its equivalent 86,400-second interval so model-authored "every day" requests converge on one durable representation. It does not weaken the durable schedule evaluator or invent fields after persistence.

An open prompt canvas has an artifact identity for editing and a separate Library Prompt identity for execution. Focused context exposes the runnable identity, and the chat adapter can map the focused artifact ID to that Prompt only after an owner-scoped database lookup. Arbitrary artifact IDs are never accepted as task targets. Schema and schedule failures return bounded field or rule details so the model can correct one request without guessing repeatedly.

The same shared chat tool can promote arbitrary substantial work to a durable, one-time Task. The server derives the owner, conversation, stable input identity, selected agent, and version from the accepted or durable turn context; the model supplies only the bounded title, work message, reason, limits, and optional exhaustive corpus. Durable orchestration keeps its task-qualified journal key while Kaizen derives a separate deterministic UUID for the job binding, so recovery replays converge without exposing identity to the model.

The background run copies the source conversation's persisted pipeline ID, version, assignment source, and policy version into its private Task thread in the dispatch transaction. A missing source aborts dispatch; ordinary Tasks receive their own policy assignment. Each Task turn still passes normal pipeline admission and falls back to the reviewed baseline if its assignment is disabled or incompatible.

Before inserting the production run, the server requests or polls Core authority and settles first-use consent from the initiating member's live verified bearer. Missing or expired authority fails without creating an unstartable queued run. The definition's schedule stays disabled, so a one-shot job cannot become a recurring Agent schedule. Replaying the same input converges on the same Task and run, and a Task-launched agent cannot promote another background job. An unqualified status or control resolves only when the conversation has one binding; multiple jobs require the exact run from the corresponding card or tool result.

Library agents pin their immutable component version directly. Built-in source agents use an owner-scoped protected Library mirror only to satisfy the Task ledger's target foreign key and platform revision mapping. Worker execution resolves the real source agent by its server-selected id and policy digest; a changed or missing source definition fails closed instead of executing the mirror.

The chat card is a bounded projection of the existing Task run detail and control routes. It does not create another scheduler, output store, or browser authority. An optional corpus changes only the execution adapter: ordinary work uses the same agent runtime once, while exhaustive folder, Collection, or Document work uses the partitioned corpus child state described above.

Goals and bounded Agent loops

A Goal is versioned metadata on one Task definition. Its private envelope binds the normalized outcome, constraints, and verification digest to the exact Task, accepted source input, pinned Agent component and version, and definition revision. The API projects only the member-safe Goal. Editing it increments the Task revision, returns the loop to draft, disables its schedule, and revokes stale test and execution authority.

create_loop is the only recurring Agent admission path. It creates a marked definition with explicit ends_at and max_occurrences, missed_policy=skip, zero backfill, and overlap_policy=skip, then settles Core authority from the initiating member's live verified bearer and activates in the same server operation. The chat model and member do not test, consent, poll, or activate it separately. Dispatch, execution, journal, approvals, notifications, and cancellation remain the same shared Task pipeline. Generic Agent targets remain rejected by public authoring and runtime policy.

A Goal result is one closed disposition: continue, complete, needs input, or blocked. Needs-input writes an owner/run/revision/Agent-bound request, preserves the member-readable question in the private Task thread, clears execution leases, pauses the loop schedule, and records an input-required notification. An exact accepted owner response requeues the same run. A continued resumed run atomically re-enables the next future occurrence; completion leaves the schedule disabled. No worker or sandbox remains idle while the member decides.

The broker carries one opaque platform reference

Before handoff, Kaizen maps its immutable occurrence to Core's idempotent run reservation. Valkey receives only the resulting opaque run_reference, with a deterministic job ID. It never carries Kaizen schedules or definitions, member identity, capabilities, workload assertions, DPoP keys, credentials, or effect authority. Possession of the reference grants no read, cancel, claim, or execution permission.

Broker enqueue failure after the Kaizen and Core reservations commit is expected to leave visible queued work. That is by design: the durable ledgers are the truth, and handoff repair re-enqueues the same Core reference. Core's atomic claim and monotonic fencing ensure exactly one executor is current across replicas and redelivery.

Never delete ledger rows or hand-mint replacement runs to "fix" a stuck handoff.

The worker proves authority every time

A worker receives only an opaque Core run reference. It then:

1. reads a fresh projected workload assertion and creates a fresh DPoP proof,
2. atomically claims the Core run and receives a short-lived, sender-constrained run capability,
3. binds the returned Core run, claim, authority deadline, and fencing token to the immutable Kaizen occurrence and revision,
4. resolves the exact target version under current member-plus-workload authority, and only then
5. enters the same accepted-input and runtime path as interactive work.

Core owns consent, grant state, current member entitlement, active workload revision, capability consumption, cancellation, and durable audit. Kaizen is a policy-enforcement point, not a second policy decision point. Its database may retain only opaque Core IDs and safe evidence; the run/effect capabilities, assertion, proof key, broker handle, and one-use consumption material stay in process memory and are non-serializable. The former Kaizen-local grant table is legacy evidence only and cannot authorize production.

The run capability lasts about a minute and the claim lease about two, while a run lasts as long as its steps. For the whole worker job, Kaizen heartbeats the lease every 20 seconds. It exchanges the capability through Core's successor route whenever less than 30 seconds remain, and each chain step checks its binding against that live capability window rather than the one returned at claim time. A capability that has already lapsed cannot be renewed, so a missed renewal fails the run's next platform call closed.

A terminal outcome is recorded locally even when Core refuses the matching platform transition, for example after the authority deadline, because Core ends that run itself. If a worker leaves while cancellation is pending, handoff repair cancels the run once its lease has expired and records the usual cancellation event and audit.

Every model call, credential use, or protected resource action first reserves an exact effect under the current run fence. For a model request, Kaizen binds the receiver-visible method, public HTTPS authority, URI, content type, and exact serialized body digest; the body is sent as canonical JSON (sorted keys, compact separators, ASCII escapes) because that is the form Kamiwaza digests, and its output ceiling is bounded to 8192 tokens. A deployment may carry that request over a separate transport URL — the route derives from the injected platform origin rather than from any named ingress controller — but the transport cannot replace the registered resource audience. Kaizen then sends one attempt carrying only the effect capability, a DPoP proof for the public URL, and the workload assertion. The Kamiwaza model endpoint is the guard: it authorizes and consumes the effect and derives the sealed requester context itself, so Kaizen never presents a consumption token or context headers. Kaizen does not perform model-alias discovery or timeout replay. The resource's registered descriptor determines action classification, canonical digest, current-access adapter, quota dimensions, and approval requirement. Model admission charges the member and enforces the distinct workload ceiling in one correlated transaction. A mutation is consumed before protected code and ends ambiguous when the external outcome cannot be proven.

The legacy local mode remains only for safe-test and fixture paths. There a credential-free grant — bound to exact owner, task, definition, target, actions, policy version, and expiry — is evaluated by embedded Cedar against AuthZEN-shaped principal, action, resource, and context values drawn from current PostgreSQL state. Snapshot strings are evidence only, and the local principal has no platform roles. Its authorization headers come only from the deployment's platform service identity or, for a safe test's execution on a deployment without one, from the member's single-use draft test model grant. Normal grants include Kaizen write authority only behind the resolved member permission mode, live lease, current definition and target, and tool resource guard. Safe-test grants remain read-only, and sensitive actions deny because a headless worker cannot satisfy MFA.

Every retry stays on the same run and receives a new attempt number.

Approval waits release execution capacity

A normal task write first crosses the same member approval guardrail as an interactive Pi run. The effective mode comes from the task override or the current workspace, owner, and per-agent defaults. Safe tests always retain their separate side-effect restrictions.

When approval is required, the runtime writes an owner-scoped request bound to the run, input, agent, tool, and canonical argument digest. It releases the conversation-affinity sandbox before marking that request ready. The executor then appends approval.waiting and changes running to awaiting_approval in one fenced transaction, marking any other still-pending request from that run as failed so only the request the run waits on stays open. Cancelling a run retires its open requests for the same reason; the run transition clears lease_owner, lease_token, and lease_expires_at, which also frees the owner's concurrency slot.

A chain task run writes its own records to the same append-only orchestration log a member's chain run uses: each completed step's output, the exact call parked for approval, and the outcome of applying that call. Writing is best effort, and a run whose records are lost still completes; it just cannot be replayed. Records are bounded, and a step or call too large to record exactly is never replayed as if it were.

Approval changes the run from awaiting_approval to queued and enqueues it again. The resumed attempt folds those records before it starts: steps before the waiting step keep their recorded output, and the approved call is applied exactly as the member saw it. The tool's successful result becomes the waiting step's output; later steps run again against that result rather than reusing outputs written before approval. The approval is consumed first, which is a one-shot transition, so an interrupted resume can lose that write but can never perform it twice. A run whose approved call did not happen fails rather than reporting success without it. A background run is granted only the atomic artifact writes for the same reason: a staged write spans several calls and its first one carries no content, so a run that paused on it would have nothing to apply. A parked call the run cannot apply fails the run with approved_call_not_applied rather than reporting success without the write. The resume may land while the paused job is still finishing, so the enqueue retries briefly, and task jobs keep no result that would block the same job id. A new claim and attempt are required before execution resumes. When the paused claim's capability has expired, Core fences that claim and issues a successor with a higher fencing token, which Kaizen records in place of the old one. The matching receipt is consumed once at tool execution. Denial changes the run to policy_blocked. Neither outcome reuses the released task or sandbox lease.

Missed work, overlap, and replay

Catch-up is never unbounded. Grace, backfill window and count, queue, concurrency, runtime, token, cost, and output limits are all server-capped.

Situation	Default behavior
Recurring work missed a window	Skip
One-time work is late	Expire
Explicit backfill	Oldest-first, bounded by window and count
Overlap, queue-one	Only one pending occurrence
Overlap, cancel-replace	Requires conservative replace-safe classification, else policy-block
Ambiguous external effect	Terminal; never replayed

Flow is authored; run trace is observed

Flow resolves the exact immutable Library version under current access. A prompt chain uses the existing static chain visual language. Editing opens the Library; the task inspector never mutates the source.

Run trace orders durable events by server sequence. It shows actual branches, loops, retries, duration, bounded usage, state changes, and safe source provenance. It never animates the authored canvas as though runtime behavior had changed the definition.

A running step also appends step.progress while its runtime is still producing frames: model-wait heartbeats, tokens, or tool state. The executing worker records at most one per ten seconds, only between the step's start and finish, and the trace folds them into the step's live row instead of listing each one. The events carry no content. A provider stream that sends nothing is still ended by the runtime's own inactivity budget, so these events cannot keep a dead step looking alive past that bound. The signal is process-local: the worker that executes a run also drives its runtime stream.

Refreshing or reconnecting resumes after the last durable sequence. Browser connection state does not start, stop, or authorize execution. Terminal replay closes only after the final durable state is visible. The terminal run event is authoritative for the Flow overlay, so a briefly stale run summary cannot leave an authored node painted as running after the durable trace has settled.

For a prompt-chain target, this parent Task run owns the whole workflow rather than a sequence of browser-owned requests. Its lease heartbeat spans every chain step, step lifecycle events enter the same Task trace, and retry or recovery reuses the same run identity. The Prompts library may link to this run in its workflow history, but the Task trace, output references, controls, and terminal status remain the canonical records.

A run's Output is a third projection beside those two, and it invents no new store. Every run opens one deterministic private conversation — its task thread — and the runtime writes the answer and any artifacts there exactly as a chat turn does. GET /task-runs/{run_id}/output resolves ownership from the forwarded envelope, then reads that conversation back through the same member-safe journal projection the Chat surface uses, and reduces it to a bounded answer preview, artifact handles pinned to the version the run wrote, and the correlations that identify the producer: the conversation to open, the journal position of the answer, the attempt, and the resolved target version. The run id selects; it never authorizes. A caller who does not own the run gets the same non-disclosing not-found as for any other run.

The projection's state is closed. available carries output, empty says the run produced nothing and why (still running, or finished with nothing), unavailable says the run did produce output but its source is no longer readable under retention. A failed run answers with its safe diagnosis rather than a server error, and retention expiry never rewrites the durable succeeded state. Raw reasoning, tool payloads, prompts and provider errors are not read at all: only journal fields the journal contract already declares member-safe.

Run trace resolves the same correlations. An output.published event carries a conversation_run or orchestration_run reference, and the trace now re-authorizes those at read time — the conversation through the owner-scoped conversation read, the orchestration run through its owner-scoped row — instead of redacting every runtime reference because no resolver claimed it.

Safe detail and expired sources

Trace envelopes contain safe state, reason, timing, usage, version, and opaque source references. Kaizen re-authorizes each source when detail is opened:

• authorized — a bounded, redacted projection with provenance;
• redacted — reveals no source existence or content;
• payload_unavailable — retained trace metadata outlived its source.

System and developer instructions, hidden reasoning, credentials, tokens, cookies, raw provider errors, raw prompts and completions, unrestricted tool arguments and results, and another member's inaccessible content are always prohibited.

Cancellation

Cancellation is durable first. Queued and retry-wait work becomes cancelled immediately. Running work receives the shared conversation cancellation signal, loses side-effect authority at the task lease fence, and can no longer publish a stale success.

Automatic retry is permitted only when runtime evidence explicitly says the failure is retry-safe and no external effect occurred.

Corpus pause uses the same fence. A running partition reaches its safe commit or classification boundary before the parent run becomes paused. Resume queues that same run and retains the manifest, leases, committed records, provenance, and export checkpoints.

Failed-run recovery

A failed run carries a closed safe diagnostic: plain-language summary, affected scope, stable reason code, retry eligibility, authorized recovery actions, and an opaque troubleshooting context reference.

Opening that context re-checks the run owner and referenced sources. Task ownership never turns a recovery action into an approval.

Related

• Tasks & scheduling — the member view
• Task automation security
• Task scheduler operations