Task scheduler operations

Durable Tasks use Kamiwaza Core's complete delegated-workload identity, grant, claim, lifecycle, resource, credential, approval, quota, revocation, and audit contract. Kaizen-local authority is not a production mode.

Availability

TASKS_ENABLED defaults to false. This release holds Tasks for a later qualified release: a default installation offers members no Tasks rail entry, no task creation, no chat task tools, and no task API, runs no scheduler, and registers no task worker job. The administrator's setup checklist reports Tasks as switched off for this deployment, which is the rollout decision rather than a platform fault.

An administrator may still set TASKS_ENABLED=true. Default-off is not a prohibition, but enabling Tasks on this release is an operator choice outside its qualification: this release's acceptance testing treats the Tasks journeys as not applicable, so an enabled deployment runs workflows that were not qualified for it. Run now and scheduled runs then go through Kamiwaza Core, and on a Kamiwaza that does not yet prove the complete delegated-workload graph, the member surface, chat task tools, task API, and scheduler stay closed and the setup checklist reports the platform hold — the flag requests the rollout, it does not force the surface open.

Setting TASKS_ENABLED=true does not open Tasks until Core's full readiness decision is healthy. While any required term is unavailable, Kaizen keeps these boundaries closed:

no task activation or execution;
no scheduler wake and no task worker job;
no task navigation or member route;
no deletion of retained task, run, event, notification, or audit history.

The readiness gate is code-owned and has no environment, manifest, live-setting, or request override. Retained records remain available to the storage and owner-scoped projection layers; the hold performs no deletion or migration. Before static scheduler registration is complete, API health stays HTTP 200 and reports tasks_scheduler: held; the worker readiness command exits healthy without contacting the queue because no Task execution capability is expected. Once static registration is complete, process health checks only local dependencies. Task admission separately checks Core's current release decision before every protected effect, so live platform changes cannot make process health flap.

The extension already declares the release shape: 20 v1 capability families, compatible v1 guards for kamiwaza.model and kamiwaza.dataset, the task-control role on api-backend, and the task-executor role on background-worker. Kaizen opens only from Core's fresh aggregate readiness decision plus the local rollout flag; individual healthy subsystems are never enough.

Turning the flag on never grants execution. Normal runs require an exact live Core grant, a fenced worker claim, a current definition and target, and a member-plus-workload allow decision for every run and side effect. Legacy local mode remains inert for production.

Settings

Setting	Default	Purpose
`TASKS_ENABLED`	`false`	Master reversible rollout and emergency-stop gate. Off in this release; see Availability
`TASKS_BACKGROUND_AUTH_MODE`	`core`	Reserved run-bound platform exchange. Legacy `local` values still parse during upgrade but never satisfy authority/readiness
`TASKS_LOCAL_TEST_GRANT_TTL_SECONDS`	3600	Lifetime of a safe test's local capability grant and, on a deployment without a platform service identity, of the encrypted single-use draft test model grant that holds the member's bearer at rest until the test spends it. Lower it to shorten how long an unconsumed member credential can sit in the database
`TASKS_LOCAL_GRANT_TTL_SECONDS`	2592000	Legacy inert upgrade setting; never production authority
`KAMIWAZA_ORIGIN`	injected by the platform	Public platform authority used to derive server-side platform URLs. An absolute `http(s)` origin with no userinfo, query, or fragment; a routing prefix is supported. Anything else leaves Tasks unavailable and is reported as `tasks.platform_origin_unusable` at startup
`KAMIWAZA_PLATFORM_GATEWAY_URL`	injected by the operator when configured	Standard internal callback transport derived from immutable platform policy. It does not replace the registered path, Host, TLS authority, or authorization context
`KAMIWAZA_VERIFY_SSL`	unset, which verifies	Certificate verification for outbound platform calls; the platform injects it only to turn verification off
`TASKS_BACKGROUND_AUTH_URL`	`KAMIWAZA_ORIGIN` + `/api/v1/delegated-workloads`	Generic delegated-workload API base URL
`TASKS_PLATFORM_GATEWAY_URL`	`KAMIWAZA_PLATFORM_GATEWAY_URL`, then `KAMIWAZA_ORIGIN`	Override for the server-side platform transport origin. Empty resolves to the public origin, which a pod normally cannot reach — see the note below the table. A direct Core service is not a gateway route
`TASKS_BACKGROUND_AUTH_AUDIENCE`	empty	Legacy task-runtime audience; not required by projected workload identity
`TASKS_MAX_DEFINITIONS_PER_OWNER`	100	1–1000
`TASKS_MAX_LIST_PAGE_SIZE`	100	1–100
`TASKS_DISPATCH_CADENCE_SECONDS`	5	Due-claim wake cadence
`TASKS_DISPATCH_BATCH_SIZE`	100	Schedules locked per due transaction
`TASKS_HANDOFF_REPAIR_BATCH_SIZE`	100	Queued runs re-enqueued per repair pass
`TASKS_BLOCKED_RUN_EXPIRY_SECONDS`	3600	60–604800. A run still waiting for platform execution authority after this long expires before repair re-enqueues anything
`TASKS_MAX_INTERVAL_SECONDS`	31622400	Largest fixed interval
`TASKS_MAX_MISFIRE_GRACE_SECONDS`	604800	Maximum on-time grace
`TASKS_MAX_BACKFILL_WINDOW_SECONDS`	2592000	Maximum recovery lookback
`TASKS_MAX_BACKFILL`	100	Backfill count ceiling
`TASKS_MAX_QUEUED_RUNS_PER_OWNER`	1000	Owner queue pressure
`TASKS_MAX_CONCURRENT_RUNS_PER_OWNER`	10	Owner execution concurrency
`TASKS_MAX_RETRY_ATTEMPTS`	10	Retry ceiling
`TASKS_MAX_RETRY_DELAY_SECONDS`	86400	Retry delay ceiling
`TASKS_MAX_RUNTIME_SECONDS`	86400	Per-run runtime
`TASKS_MAX_TOKENS`	1000000	Per-run tokens
`TASKS_MAX_COST_MICROUNITS`	1000000000	Per-run cost
`TASKS_MAX_OUTPUT_BYTES`	100000000	Durable output

TASKS_PLATFORM_GATEWAY_URL resolves through three members in order — this setting, the operator-injected KAMIWAZA_PLATFORM_GATEWAY_URL (its origin only), and finally KAMIWAZA_ORIGIN — so leaving it empty does not mean "no gateway": it means the deployment's own public origin, which a pod normally cannot reach, and in a cluster that is the configuration that makes model calls hang until they fail as model_unavailable. Kaizen logs platform_model_route_is_public_origin at boot on every service when the chain resolves to the public origin inside Kubernetes.

Fixed intervals accept any positive whole number of seconds. Actual dispatch is limited by TASKS_DISPATCH_CADENCE_SECONDS; missed Agent-loop occurrences are skipped rather than backfilled. Other Task types retain their configured misfire and backfill policies.

The service identity is how a run calls the platform as the deployment rather than as the member, and only two things forward it: connector sync and model traffic. It is not delegated member authority and never substitutes for it — production runs prove themselves to the platform by workload attestation, which needs no secret at all.

These settings are optional for scheduled runs, which attestation authorizes, and optional for a member's safe test too. A capability grant authorizes a run to execute; it does not authorize the model call every task makes, and attestation does not reach the safe-test path. With a service identity, a safe test calls the model as the deployment. Without one, the API edge seals the member's own bearer into a single-use draft test model grant bound to that one test run, and the worker spends it on the run's first execution. A safe test that reaches the worker without a usable grant — the member's request carried no platform bearer, or the grant expired or was already spent — stops before the model and reports draft_test_model_grant_unavailable, which offers the member Test again. platform_authority_unavailable is reserved for a deployment that can neither call the model as itself nor seal a grant. The worker also logs worker.platform_service_identity_absent once at startup when it has no identity of its own.

The grant does not reach Run now or scheduled runs, which keep going through Core. Turning Tasks on for a deployment still needs the platform authority those runs require.

Never place browser bearer tokens, connector credentials, action tokens, approval decisions, MFA state, or copied member roles in these settings. No configuration authorizes app-local production delegation.

The historical task_capability_grant table is upgrade evidence only. It stores no token, credential, header, cookie, password, signing key, or reusable delegation artifact and cannot authorize production.

Authority model

PostgreSQL time and next_fire_at are authoritative. Cron only wakes a bounded FOR UPDATE SKIP LOCKED claim. Occurrence uniqueness and one-run-per-occurrence constraints defeat duplicate wakes. Kaizen maps an occurrence idempotently to a Core run reservation before handoff. Valkey carries only that opaque Core run reference — never schedules, definitions, member identities, workload assertions, capabilities, proof keys, or credentials. Core atomically claims the run and issues a monotonic fence; redelivery converges on the same run.

Every run goes through Core. A task can reach Core resources through the SDK or any tool, so one authority model is the only one that stays auditable.

That is affordable because an automation has a stable identity — the member and the target it runs against, not the individual task. Core settles an intent without presenting it when a prior decision covers that identity, so a member consents once to an agent running for them in the background rather than once per task. A deployment configured for implicit consent settles read-class automations without any prompt at all; Kaizen's task descriptor declares only kamiwaza.model:invoke, which is read class.

note

The identity includes the owner. Core scopes prior consent by tenant and client but not by member, so an identity without the owner would let one member's approval settle another member's automation.

Metrics and alerts

Metrics use bounded status, trigger, result, and reason-category labels. They never label by owner, task, run, target, prompt, model output, email, or other unbounded identity.

Metric	Meaning	Alert signal
`kaizen_task_schedules_due`	Enabled schedules currently due	Non-zero beyond two dispatch cadences
`kaizen_task_dispatch_lag_seconds`	Age of oldest due schedule	> `2 × TASKS_DISPATCH_CADENCE_SECONDS`
`kaizen_task_oldest_pending_seconds`	Age of oldest queued or retry-wait run	Sustained growth while the worker is Ready
`kaizen_task_occurrences`	Committed occurrence decisions	Rising blocked, expired, or skipped rate
`kaizen_task_runs_current`	Current runs by status	Queue growth, or concurrency pinned at cap
`kaizen_task_runs`	Terminal runs by status and trigger	Failure or block ratio change
`kaizen_task_retry`	Retry decisions by category	Repeated exhausted or non-retryable
`kaizen_task_run_duration_seconds`	Terminal duration	p95 near the effective timeout
`kaizen_task_run_tokens`	Terminal token usage	Sustained approach to caps
`kaizen_task_run_cost_microunits`	Terminal cost	Sustained approach to caps
`kaizen_task_authority_exchange`	Core proof, grant, claim, effect and readiness results	Any failed, denied, fenced, replay, stale, incompatible, or unavailable
`kaizen_task_approval_waits_current`	Current one-shot member approval waits	Unexpected growth or age
`kaizen_task_cancellation_latency_seconds`	Request-to-terminal latency	p95 beyond the safe-boundary objective
`kaizen_task_corpus_operation`	Corpus phase, authorization, retry, budget, and publication outcomes	Any denial or exhaustion; sustained failed/retry growth

Correlate aggregates with owner-scoped audit records and safe run diagnostics. Never add cardinality so a dashboard can identify a member or a task.

Lag and handoff recovery

Alert when the oldest enabled next_fire_at is older than two dispatch cadences, or when queued available_at age grows.

First verify database health, worker timezone-data identity, Core readiness freshness, workload attestation, and claim service health. Then trigger repair: repair_task_handoffs safely re-enqueues the same committed Core run references with deterministic job IDs, and the Core claim fence ensures exactly one executor remains current.

Broker enqueue failure after a database commit is expected to leave visible queued work. Do not delete ledger rows or manually mint replacement runs. Trigger handoff repair or restore worker cadence.

Corpus recovery follows the same parent-run handoff. An expired partition lease returns only that uncommitted partition to pending. Committed partitions, canonical records, provenance, and ready exports remain idempotent. Use the member's Pause job, Resume job, Cancel, and safe retry controls; do not delete child rows, change a manifest digest, or mint a replacement run.

Downtime behavior is task policy, not an incident:

recurring work skips missed windows by default;
one-time late work expires;
explicit backfill is oldest-first, bounded by window and count;
overlap policy applies after recovery, including queue-one and safe cancel-and-replace classification.

Authority failure response

Treat the three outcomes differently:

denied — current membership, grant, target access, resource contract, quota, approval, policy, audience, revision, claim, or fence rejected the operation. Preserve the block. Never retry with workload identity.
failed — attestation, DPoP, transport, registered guard, or Core evaluation failed. Verify the fixed audience, clocks, projected-token rotation, registration activation, and proof construction. Queued work remains durable.
unavailable — the database, Core component, healthy acceptable attestation profile, resource guard, or fresh readiness report is absent. Keep every new boundary fail-closed until the authoritative path recovers.

After repair, trigger normal bounded dispatch and handoff repair. Do not copy a browser credential, mint an ad hoc run, or mutate occurrence decisions.

Notification operations

Notification creation reads the task's current policy under the same database authority used for its durable insert. Duplicate worker delivery reuses the deterministic notification identity. Preference changes affect future material states only.

Monitor inbox query latency, unread growth, and insert failures alongside terminal-run counts. A notification failure must not rewrite run truth. Repair delivery from durable task and run state using the same idempotent mapping; never replay raw model or provider content.

Notification retention is separate from live task deletion. Apply the retention window in bounded owner-scoped batches, record removed counts, and preserve occurrence, run, trace, and audit evidence for their independent policies. Mark-read is not deletion.

Authority prerequisites

Before Tasks can become available:

Reconcile the manifest's task-control and task-executor roles to the exact deployment UID and immutable artifact/configuration/role digests.
Reconcile compatible model and dataset protected-resource registrations, guards, policies, quotas, approval rules, and credential bindings.
Prove the fixed kamiwaza-workload-sts audience and both registered attestation profiles, with TokenReview preferred and offline fallback tested.
Prove member consent, dual-principal authorization, idempotent reservation, atomic claim/fencing, effects, model attribution, credential brokering, cancellation, revocation, and durable audit end to end.
Confirm all 20 capability families and every required component report compatible ready state inside the freshness window.
Prove failure never falls back to an app-local grant, service identity alone, stored snapshot, bearer capability, or browser credential.

Core capability and nonce authority is cluster-scoped, not node-scoped. Every Core and Ray replica must mount the same preserved cluster Secret so node joins, replacement, and cross-replica routing do not invalidate an in-flight nonce or capability. Different Kamiwaza clusters use different authority material. A strict FIPS deployment must provide the cluster P-256 key from its approved KMS, HSM, or validated crypto provider; selecting P-256 without validated key generation provenance is not a FIPS compliance claim.

Legacy local tomo.task grants never become platform grants. Any later migration requires fresh member consent.

Deployment validation

./scripts/kz-ext validate .

Validation must confirm the intended TASKS_ENABLED value (the shipped default is false), TASKS_BACKGROUND_AUTH_MODE=core, bounded task settings, password metadata for the optional legacy Core secret, and explicit API and worker configuration. A true flag without complete readiness remains held and is not an enablement proof.

Rollback

Set TASKS_ENABLED=false to stop new scheduling and execution. The independent readiness gate also prevents scheduler registration and new execution whenever Core authority is incomplete.

Turning the flag off does not cancel work that is already queued, waiting, or running. With the flag off the worker registers no task job and runs no repair, so such a run is neither executed nor ended: it stays in its last status, with its definition, events, notifications, approvals, and audit records retained. The member controls that end a run are behind the same flag, so they are unavailable once it is off. Decide what happens to current work before you turn the flag off or replace the workers, using the procedure below.

Turning Tasks off with work in flight

This procedure applies to rollback and to upgrading an installation that runs Tasks to this release, which ships them off.

Check for work. On the running deployment, count every run that has not ended, straight from the database:
```
SELECT status, count(*) FROM task_run
WHERE status IN ('queued', 'running', 'retry_wait', 'paused',
                 'awaiting_approval', 'awaiting_input')
GROUP BY status ORDER BY status;
```
Use this count rather than kaizen_task_runs_current: that gauge does not report awaiting_input, so it can read zero while runs are still waiting on their owners. If the query returns no rows, there is nothing to drain; go to step 4.
Drain. Ask task owners to pause their tasks, which stops new occurrences, and to resolve every run that is waiting on them: approve or decline approvals, answer input requests. Then wait for active runs to finish, re-running the step 1 query until it returns no rows. Operators have no control that pauses or cancels another member's work; owner scoping is deliberate.
Cancel what cannot finish. For a run that cannot finish inside the window, the owner uses Cancel on the run. Cancelling a run that is executing only requests cancellation; the run ends cancelled, with its audit record, once its worker acknowledges the request, which needs Tasks still on. Confirm with the step 1 query that each cancelled run has left the non-terminal statuses before you continue. A run that still shows running with its cancellation unacknowledged is held, not cancelled: record it in step 4. A cancelled run is never re-enqueued.
Record the outcome, including any run you chose to leave non-terminal and why. A run left in place is held, not cancelled.
Turn Tasks off — set TASKS_ENABLED=false, or upgrade to a release whose default is off — and replace the workers. Preserve PostgreSQL and Valkey state.
Verify. The setup checklist reports Tasks switched off for this deployment, a member's Tasks link lands on Home, and the task API answers 404. Confirm retained custody with a read-only count, which must match what you recorded:
```
SELECT status, count(*) FROM task_run GROUP BY status ORDER BY status;
```
Right after the switch, the background worker may log function 'cron:dispatch_due_tasks' not found once or twice. Those are scheduler wakeups the previous worker queued before it stopped; the new worker drops them without running anything. A run job queued the same way is dropped too, and its run stays held as described above.

warning

A run left queued, retry_wait, or running while Tasks are off can be picked up again when they are next turned on. Handoff repair treats a running run whose worker lease has expired as retryable. Resolve every run you recorded as held before you re-enable Tasks.

info

Upgrading does not necessarily reset a value the deployment already carries. An installation whose configuration records TASKS_ENABLED=true keeps Tasks on after upgrading to this release. Read the setup checklist after the upgrade, and set TASKS_ENABLED=false explicitly to adopt this release's posture.

To reduce blast radius without a full rollback, lower the dispatch batch, backfill, queue, concurrency, runtime, token, cost, and output caps.

Never change timezone data on only part of a scheduler deployment. Drain, roll all replicas, verify the exact identity, then resume.

Keep definitions, runs, events, notifications, and audit evidence. A database downgrade is not an operational rollback.

Availability​

Settings​

Authority model​

Consent, and why it is not a prompt per task​

Metrics and alerts​

Lag and handoff recovery​

Authority failure response​

Notification operations​

Authority prerequisites​

Deployment validation​

Rollback​

Turning Tasks off with work in flight​

Related​