Job Submission

Submit and manage Ray jobs on local or remote clusters. Jobs run Python entrypoints on the cluster's Ray runtime and can return structured results via a log marker protocol.

Prerequisites

• Admin authentication (JWT token with admin role)
• Ray cluster running on the target cluster

Submit a Job (Async)

curl -sk -X POST "https://kamiwaza.test/api/cluster/jobs/submit" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python train.py --epochs 10",
    "runtime_env": {"pip": ["numpy", "pandas"]},
    "timeout_seconds": 300
  }'

Response:

{"job_id": "uuid", "ray_job_id": "raysubmit_...", "status": "PENDING"}

Run a Job (Synchronous)

Submits and blocks until completion, returning the extracted result:

curl -sk -X POST "https://kamiwaza.test/api/cluster/jobs/run" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "entrypoint": "python inference.py",
    "timeout_seconds": 60
  }'

Response:

{
  "job_id": "uuid",
  "ray_job_id": "raysubmit_...",
  "status": "SUCCEEDED",
  "result": {"accuracy": 0.95},
  "duration_seconds": 12.3
}

Check Job Status

curl -sk "https://kamiwaza.test/api/cluster/jobs/{job_id}/status" \
  -H "Authorization: Bearer $TOKEN"

Returns the full job record including status, timestamps, and error details if failed.

Get Job Logs

curl -sk "https://kamiwaza.test/api/cluster/jobs/{job_id}/logs" \
  -H "Authorization: Bearer $TOKEN"

Returns Ray stdout/stderr for the job.

Extract Job Result

curl -sk "https://kamiwaza.test/api/cluster/jobs/{job_id}/result" \
  -H "Authorization: Bearer $TOKEN"

Extracts structured JSON from the job's stdout. The job code must print a result marker:

import json
result = {"accuracy": 0.95, "model": "v2"}
print(f"KZ_MESH_RUN_ON_JSON::{json.dumps(result)}")

Status Code	Meaning
200	Result extracted successfully
409	Job has not succeeded yet
410	Ray logs expired (result no longer available)

Cancel a Job

curl -sk -X POST "https://kamiwaza.test/api/cluster/jobs/{job_id}/cancel" \
  -H "Authorization: Bearer $TOKEN"

Requests cancellation of a running job. The job transitions to STOPPED status.

Job Lifecycle

Status	Description
PENDING	Submitted to Ray, not yet running
RUNNING	Executing on Ray cluster
SUCCEEDED	Completed successfully
FAILED	Exited with error
STOPPED	Cancelled by user or timed out

Timeout and Auto-Cancel

Set timeout_seconds on submission. If the job exceeds this duration during a /run call, it is automatically cancelled and marked with timed_out: true.

API Reference

Method	Path	Description
POST	/cluster/jobs/submit	Async submission
POST	/cluster/jobs/run	Sync submit + poll + result
GET	/cluster/jobs/{id}/status	Job status with Ray refresh
GET	/cluster/jobs/{id}/result	Extract structured result
GET	/cluster/jobs/{id}/logs	Ray stdout/stderr
POST	/cluster/jobs/{id}/cancel	Cancel running job