Job Submission

Submit and manage Ray jobs on local or remote clusters. Jobs run Python entrypoints on the cluster's Ray runtime and can return structured results via a log marker protocol.

Prerequisites

• For a local job: a token for a user who may run jobs on the cluster, such as admin.
• For a job on a peer: a shared-realm token for a subject the receiver has allowlisted with cluster_jobs:__all__#executor. Admin roles do not cross the federation.
• Ray cluster running on the target cluster

Federated delegated jobs additionally require receiver-side allowlisting, cluster_jobs:__all__#executor, exact receiver-local resource grants, an execution gate, and delegated jobs enabled in the receiver's values file:

core:
  delegatedJobs:
    enabled: true
    admission:
      enabled: true

Apply it by rerunning the helm upgrade --install command from Install on Kubernetes with Istio with the updated values file.

Job runtime

Ordinary Ray jobs, including jobs from a peer, run on the cluster's job runtime. For delegated jobs, see Governed delegated access.

When you write a job:

• Call Kamiwaza through the cluster's public address with your own short-lived token, as you would from outside the cluster.
• Pass what the job needs in runtime_env.env_vars. If that includes a credential, use a short-lived, narrowly scoped one.
• By default a job can reach DNS and, on installs that use Istio or the Envoy edge, Kamiwaza. To reach anything else, see Operator settings.

Operator settings

Configure the job runtime under core.userJobs in your values file:

core:
  userJobs:
    enabled: true
    numCpus: 4
    resources:
      limits:
        cpu: "4"
        memory: 8Gi
    # Literal, non-secret values only. Secret references are rejected.
    extraEnv:
      - name: CATALOG_URL
        value: https://catalog.example.com
    egress:
      # CIDRs only, not host names. 0.0.0.0/0 and ::/0 are rejected.
      additionalAllowedHosts:
        - 10.20.30.0/24

Jobs reach Kamiwaza by default through the Istio ingress gateway when Istio is enabled, or through the release's Envoy edge when global.ingress.provider is envoy. With any other ingress provider, add the address jobs use to reach Kamiwaza to additionalAllowedHosts.

Setting enabled: false turns off ordinary Ray jobs: submissions fail with 503 and user_jobs_runtime_unconfigured.

Tenant-mode ReBAC

On an authenticated tenant-mode install, ordinary cluster jobs work with ReBAC enabled and the community fallback disabled. The request is authorized at the Kamiwaza API, then the Core API submits the job to the job runtime's Ray Dashboard Jobs API on port 8265. The chart's core-user-jobs AuthorizationPolicy allows only the core-api and core-scheduler service accounts on that port.

For a fresh strict-ReBAC install, verify the complete lifecycle before making the environment available to users. ADMIN_PASSWORD is the admin password from Verify:

API=https://kamiwaza.example.com/api
TOKEN=$(curl --fail --silent --show-error \
  --data-urlencode username=admin --data-urlencode "password=$ADMIN_PASSWORD" \
  "$API/auth/token" | jq -er .access_token)

BODY=$(jq -n \
  --arg ep "python -c \"import json; print('KZ_MESH_RUN_ON_JSON::' + json.dumps({'ok': True}))\"" \
  '{entrypoint: $ep, timeout_seconds: 120}')
JOB_ID=$(curl --fail --silent --show-error -X POST "$API/cluster/jobs/submit" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "$BODY" | jq -er .job_id)

until STATUS=$(curl --fail --silent --show-error "$API/cluster/jobs/$JOB_ID/status" \
    -H "Authorization: Bearer $TOKEN" | jq -r .status) \
  && [ "$STATUS" != PENDING ] && [ "$STATUS" != RUNNING ]; do sleep 2; done
echo "$STATUS"   # Expect SUCCEEDED

curl --fail --silent --show-error "$API/cluster/jobs/$JOB_ID/result" \
  -H "Authorization: Bearer $TOKEN" | jq .
# Expect a result of {"ok": true}

The check submits a small job, observes its status transition, and retrieves the structured result. A 503 with user_jobs_runtime_unconfigured means the job runtime is turned off; see Operator settings. A 502 containing Ray submit failed (403) indicates a mesh authorization policy problem on the dashboard hop: inspect the rendered core-user-jobs policy and confirm that the core-api and core-scheduler principals and port 8265 are allowed before changing ReBAC fallback settings.

Submit a Job (Async)

curl -sk -X POST "https://kamiwaza.example.com/api/cluster/jobs/submit" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python train.py --epochs 10",
    "runtime_env": {"env_vars": {"REPORT_FORMAT": "json"}},
    "timeout_seconds": 300
  }'

runtime_env accepts environment variables only. Kamiwaza strips Ray execution-environment keys such as pip, working_dir, py_modules, and conda. Use the approved dependency mechanism below for Python packages.

Governed delegated access and approved dependencies

A receiver-executed job must name every dataset or model operation it needs. The receiver checks the submitting identity against those exact resources before dispatch and issues renewable, job-bound authority only for that set.

Python dependencies use exact name==version coordinates from the receiver-owned package catalog:

{
  "entrypoint": "python analysis.py",
  "timeout_seconds": 900,
  "delegated_access": {
    "datasets": [
      {
        "urn": "urn:li:dataset:(urn:li:dataPlatform:postgres,nps_verbatims,PROD)",
        "operations": ["discover", "retrieve"]
      }
    ],
    "models": []
  },
  "python_packages": ["humanize==4.13.0"]
}

Packages are not resolved from public PyPI at request time. The operator must enable the package contract, approve each exact version and wheel SHA-256, and configure one PyPI-compatible repository. The request receives no repository URL or credential and cannot add an unapproved package.

For an air-gapped or restricted receiver, point the package installer at an operator-owned mirror by mounting a Secret containing a mode-0600 pip.conf. Keep the repository URL, CA configuration, and credential out of Helm values and job payloads. Restrict the delegated driver NetworkPolicy to the repository's exact CIDRs and ports:

core:
  delegatedJobs:
    enabled: true
    admission:
      enabled: true
    pythonPackages:
      enabled: true
      catalog:
        - name: humanize
          version: 4.13.0
          sha256:
            - <exact-64-character-wheel-sha256>
      repository:
        existingSecret: private-pypi
      repositoryCIDRs:
        - 10.42.0.8/32
      repositoryPorts:
        - 8443

Kubernetes NetworkPolicy works with CIDRs, not repository DNS names. Keep the address set exact and update it when the mirror moves. A package coordinate that is absent from the catalog, has the wrong digest, or cannot be fetched fails closed before the entrypoint starts.

Response:

{"job_id": "uuid", "ray_job_id": "raysubmit_...", "status": "PENDING"}

Submit a Job to a Peer

To run a job on the receiving cluster, send the same calls through the mesh proxy on your own cluster with a shared-realm token. $FEDERATION_ID is the federation's ID on your cluster; see Federation setup. $SHARED_USER_TOKEN is described in Get an admin token.

MESH=https://kamiwaza.example.com/api/mesh/$FEDERATION_ID

JOB_ID=$(curl --fail --silent --show-error -X POST "$MESH/cluster/jobs/submit" \
  -H "Authorization: Bearer $SHARED_USER_TOKEN" -H "Content-Type: application/json" \
  -d "$BODY" | jq -er .job_id)

until STATUS=$(curl --fail --silent --show-error "$MESH/cluster/jobs/$JOB_ID/status" \
    -H "Authorization: Bearer $SHARED_USER_TOKEN" | jq -r .status) \
  && [ "$STATUS" != PENDING ] && [ "$STATUS" != RUNNING ]; do sleep 2; done

curl --fail --silent --show-error "$MESH/cluster/jobs/$JOB_ID/result" \
  -H "Authorization: Bearer $SHARED_USER_TOKEN" | jq .

$BODY is a job request such as the one in the lifecycle check above. The receiver applies its execution gate before dispatch; a 403 with no_execution_gate_configured_for_mesh means the receiver has none. /cluster/jobs/run, /logs, and /cancel work through the mesh in the same way.

Run a Job (Synchronous)

Submits and blocks until completion, returning the extracted result:

curl -sk -X POST "https://kamiwaza.example.com/api/cluster/jobs/run" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python inference.py",
    "timeout_seconds": 60
  }'

Response:

{
  "job_id": "uuid",
  "ray_job_id": "raysubmit_...",
  "status": "SUCCEEDED",
  "result": {"accuracy": 0.95},
  "duration_seconds": 12.3
}

Check Job Status

curl -sk "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/status" \
  -H "Authorization: Bearer $TOKEN"

Returns the full job record including status, timestamps, and error details if failed.

Get Job Logs

curl -sk "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/logs" \
  -H "Authorization: Bearer $TOKEN"

Returns Ray stdout/stderr for the job.

Extract Job Result

curl -sk "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/result" \
  -H "Authorization: Bearer $TOKEN"

Extracts structured JSON from the job's stdout. The job code must print a result marker:

import json
result = {"accuracy": 0.95, "model": "v2"}
print(f"KZ_MESH_RUN_ON_JSON::{json.dumps(result)}")

Status Code	Meaning
200	Result extracted successfully
404	No such job, or not a job you submitted
409	Job has not succeeded yet
410	Ray logs expired (result no longer available)

Cancel a Job

curl -sk -X POST "https://kamiwaza.example.com/api/cluster/jobs/{job_id}/cancel" \
  -H "Authorization: Bearer $TOKEN"

Requests cancellation of a running job. The job transitions to STOPPED status.

Job Lifecycle

Status	Description
PENDING	Submitted to Ray, not yet running
RUNNING	Executing on Ray cluster
SUCCEEDED	Completed successfully
FAILED	Exited with error
STOPPED	Canceled by user or timed out

Timeout and Auto-Cancel

Set timeout_seconds on submission. If the job exceeds this duration during a /run call, it is automatically canceled and marked with timed_out: true.

API Reference

You can list, check, read and cancel only the jobs you submitted. Any other job ID returns 404.

Method	Path	Description
GET	/cluster/jobs/	Your jobs, newest first (limit, offset)
POST	/cluster/jobs/submit	Async submission
POST	/cluster/jobs/run	Sync submit + poll + result
GET	/cluster/jobs/{id}/status	Job status with Ray refresh
GET	/cluster/jobs/{id}/result	Extract structured result
GET	/cluster/jobs/{id}/logs	Ray stdout/stderr
POST	/cluster/jobs/{id}/cancel	Cancel running job