Version: 1.3.3 (Latest)

Job Submission

Submit and manage Ray jobs on local or remote clusters. Jobs run Python entrypoints on the cluster's Ray runtime and can return structured results via a log marker protocol.

Prerequisites

For a local job: a token for a user who may run jobs on the cluster, such as admin.
For a job on a peer: a shared-realm token for a subject the receiver has allowlisted with cluster_jobs:__all__#executor. Admin roles do not cross the federation.
Ray cluster running on the target cluster

Federated delegated jobs additionally require receiver-side allowlisting, cluster_jobs:__all__#executor, exact receiver-local resource grants, an execution gate, and delegated jobs enabled in the receiver's values file:

core:
  delegatedJobs:
    enabled: true
    admission:
      enabled: true

Apply it by rerunning the helm upgrade --install command from Install on Kubernetes with Istio with the updated values file.

Tenant-mode ReBAC

On an authenticated tenant-mode install, ordinary cluster jobs work with ReBAC enabled and the community fallback disabled. The request is authorized at the Kamiwaza API, then the Core API submits the job to the in-cluster Ray Dashboard Jobs API on port 8265. The Core API is served by the Ray head workload, so the chart's Ray-head AuthorizationPolicy must allow the core-ray service account to use that port. The public ingress rule alone is not sufficient for this east-west request, and the Ray Dashboard should not be exposed as a public job-submission endpoint.

For a fresh strict-ReBAC install, verify the complete lifecycle before making the environment available to users. ADMIN_PASSWORD is the admin password from Verify:

API=https://kamiwaza.example.com/api
TOKEN=$(curl --fail --silent --show-error \
  --data-urlencode username=admin --data-urlencode "password=$ADMIN_PASSWORD" \
  "$API/auth/token" | jq -er .access_token)

BODY=$(jq -n \
  --arg ep "python -c \"import json; print('KZ_MESH_RUN_ON_JSON::' + json.dumps({'ok': True}))\"" \
  '{entrypoint: $ep, timeout_seconds: 120}')
JOB_ID=$(curl --fail --silent --show-error -X POST "$API/cluster/jobs/submit" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "$BODY" | jq -er .job_id)

until STATUS=$(curl --fail --silent --show-error "$API/cluster/jobs/$JOB_ID/status" \
    -H "Authorization: Bearer $TOKEN" | jq -r .status) \
  && [ "$STATUS" != PENDING ] && [ "$STATUS" != RUNNING ]; do sleep 2; done
echo "$STATUS"   # Expect SUCCEEDED

curl --fail --silent --show-error "$API/cluster/jobs/$JOB_ID/result" \
  -H "Authorization: Bearer $TOKEN" | jq .
# Expect a result of {"ok": true}

The check submits a small job, observes its status transition, and retrieves the structured result. A 502 containing Ray submit failed (403) indicates a mesh authorization policy problem on the Ray Dashboard hop; inspect the rendered ray-head policy and confirm the core-ray principal and port 8265 rule before changing ReBAC fallback settings.

Submit a Job (Async)

curl -sk -X POST "https://kamiwaza.example.com/api/cluster/jobs/submit" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python train.py --epochs 10",
    "runtime_env": {"env_vars": {"REPORT_FORMAT": "json"}},
    "timeout_seconds": 300
  }'

runtime_env accepts environment variables only. Kamiwaza strips Ray execution-environment keys such as pip, working_dir, py_modules, and conda. Use the approved dependency mechanism below for Python packages.

Governed delegated access and approved dependencies

A receiver-executed job must name every dataset or model operation it needs. The receiver checks the submitting identity against those exact resources before dispatch and issues renewable, job-bound authority only for that set.

Python dependencies use exact name==version coordinates from the receiver-owned package catalog:

{
  "entrypoint": "python analysis.py",
  "timeout_seconds": 900,
  "delegated_access": {
    "datasets": [
      {
        "urn": "urn:li:dataset:(urn:li:dataPlatform:postgres,nps_verbatims,PROD)",
        "operations": ["discover", "retrieve"]
      }
    ],
    "models": []
  },
  "python_packages": ["humanize==4.13.0"]
}

Packages are not resolved from public PyPI at request time. The operator must enable the package contract, approve each exact version and wheel SHA-256, and configure one PyPI-compatible repository. The request receives no repository URL or credential and cannot add an unapproved package.

For an air-gapped or restricted receiver, point the package installer at an operator-owned mirror by mounting a Secret containing a mode-0600 pip.conf. Keep the repository URL, CA configuration, and credential out of Helm values and job payloads. Restrict the delegated driver NetworkPolicy to the repository's exact CIDRs and ports:

core:
  delegatedJobs:
    enabled: true
    admission:
      enabled: true
    pythonPackages:
      enabled: true
      catalog:
        - name: humanize
          version: 4.13.0
          sha256:
            - <exact-64-character-wheel-sha256>
      repository:
        existingSecret: private-pypi
      repositoryCIDRs:
        - 10.42.0.8/32
      repositoryPorts:
        - 8443

Kubernetes NetworkPolicy works with CIDRs, not repository DNS names. Keep the address set exact and update it when the mirror moves. A package coordinate that is absent from the catalog, has the wrong digest, or cannot be fetched fails closed before the entrypoint starts.

Response:

{"job_id": "uuid", "ray_job_id": "raysubmit_...", "status": "PENDING"}

Submit a Job to a Peer

To run a job on the receiving cluster, send the same calls through the mesh proxy on your own cluster with a shared-realm token. $FEDERATION_ID is the federation's ID on your cluster; see Federation setup. $SHARED_USER_TOKEN is described in Get an admin token.

MESH=https://kamiwaza.example.com/api/mesh/$FEDERATION_ID

JOB_ID=$(curl --fail --silent --show-error -X POST "$MESH/cluster/jobs/submit" \
  -H "Authorization: Bearer $SHARED_USER_TOKEN" -H "Content-Type: application/json" \
  -d "$BODY" | jq -er .job_id)

until STATUS=$(curl --fail --silent --show-error "$MESH/cluster/jobs/$JOB_ID/status" \
    -H "Authorization: Bearer $SHARED_USER_TOKEN" | jq -r .status) \
  && [ "$STATUS" != PENDING ] && [ "$STATUS" != RUNNING ]; do sleep 2; done

curl --fail --silent --show-error "$MESH/cluster/jobs/$JOB_ID/result" \
  -H "Authorization: Bearer $SHARED_USER_TOKEN" | jq .

$BODY is a job request such as the one in the lifecycle check above. The receiver applies its execution gate before dispatch; a 403 with no_execution_gate_configured_for_mesh means the receiver has none. /cluster/jobs/run, /logs, and /cancel work through the mesh in the same way.

Run a Job (Synchronous)

Submits and blocks until completion, returning the extracted result:

curl -sk -X POST "https://kamiwaza.example.com/api/cluster/jobs/run" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python inference.py",
    "timeout_seconds": 60
  }'

Response:

{
  "job_id": "uuid",
  "ray_job_id": "raysubmit_...",
  "status": "SUCCEEDED",
  "result": {"accuracy": 0.95},
  "duration_seconds": 12.3
}

Check Job Status

curl -sk "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/status" \
  -H "Authorization: Bearer $TOKEN"

Returns the full job record including status, timestamps, and error details if failed.

Get Job Logs

curl -sk "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/logs" \
  -H "Authorization: Bearer $TOKEN"

Returns Ray stdout/stderr for the job.

Extract Job Result

curl -sk "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/result" \
  -H "Authorization: Bearer $TOKEN"

Extracts structured JSON from the job's stdout. The job code must print a result marker:

import json
result = {"accuracy": 0.95, "model": "v2"}
print(f"KZ_MESH_RUN_ON_JSON::{json.dumps(result)}")

Status Code	Meaning
200	Result extracted successfully
409	Job has not succeeded yet
410	Ray logs expired (result no longer available)

Cancel a Job

curl -sk -X POST "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/cancel" \
  -H "Authorization: Bearer $TOKEN"

Requests cancellation of a running job. The job transitions to STOPPED status.

Job Lifecycle

Status	Description
PENDING	Submitted to Ray, not yet running
RUNNING	Executing on Ray cluster
SUCCEEDED	Completed successfully
FAILED	Exited with error
STOPPED	Canceled by user or timed out

Timeout and Auto-Cancel

Set timeout_seconds on submission. If the job exceeds this duration during a /run call, it is automatically canceled and marked with timed_out: true.

API Reference

Method	Path	Description
POST	`/cluster/jobs/submit`	Async submission
POST	`/cluster/jobs/run`	Sync submit + poll + result
GET	`/cluster/jobs/{id}/status`	Job status with Ray refresh
GET	`/cluster/jobs/{id}/result`	Extract structured result
GET	`/cluster/jobs/{id}/logs`	Ray stdout/stderr
POST	`/cluster/jobs/{id}/cancel`	Cancel running job

Prerequisites​

Tenant-mode ReBAC​

Submit a Job (Async)​

Governed delegated access and approved dependencies​

Submit a Job to a Peer​

Run a Job (Synchronous)​

Check Job Status​

Get Job Logs​

Extract Job Result​

Cancel a Job​

Job Lifecycle​

Timeout and Auto-Cancel​

API Reference​