Execution Gates

The federated job service on the receiving cluster consults a pluggable execution gate before dispatching any job to Ray. The gate decides whether this user is allowed to run jobs on this cluster, independent of what data the job will read. Record-level filtering — "what columns and rows can this user see once the job runs?" — is handled separately by per-dataset attribute gates (see the retrieval docs).

This page covers the operator-facing surface: what the default behavior is, how to configure a policy, and how to diagnose 403 responses on federated job submissions.

Default behavior

An Execution Gate is a cluster-wide resource: one gate is in force at a time. When no gate is configured, the platform applies source-scoped default policy:

Submission source	Default when unconfigured	Reason code
`local` (same-cluster authenticated user)	ALLOW	`no_execution_gate_configured_local_allowed`
`mesh` (arrived via federation peer)	DENY (403)	`no_execution_gate_configured_for_mesh`

Local submissions preserve the pre-feature behavior of single-cluster clusters. Mesh submissions — the new cross-cluster surface — are denied until an operator explicitly declares the cluster's policy. This prevents a forgotten configuration from silently accepting federated work.

A gate value that is set but does not parse is not treated as unconfigured. Every submission, local or mesh, is refused with gate_config_invalid until the value is fixed; see Troubleshooting.

Configuring a gate

A gate can be set in three places. The cluster enforces the first one that holds a value:

The API. PUT /api/cluster/execution-gate binds a gate at runtime. It applies to the next submission, with no restart.
The AUTHZ_EXECUTION_GATE environment variable. Use it only for ad hoc testing.
The Helm value core.authz.executionGate.{type, config}, which the chart renders into the core-runtime-config ConfigMap.

Use one mechanism per cluster, so the active gate is not in doubt. GET /api/cluster/execution-gate reports the active gate and, in source, which of the three it came from: api, env, or helm. See Verifying the active gate.

Quick start: permit all mesh submissions

warning

AllowAllExecutionGate admits every authenticated federated submission. A mesh job submitted without delegated_access runs on the receiving cluster's shared Ray head, next to the platform's own services. Use it only for evaluation clusters and peers you trust. For any other peer, keep the default mesh-deny, or write a site-specific gate that admits only the peers and users you trust. A gate sees the job's source, source_cluster_id, user_id, entrypoint and target_datasets, but not whether it carries delegated_access. See Federated Job Submission for delegated jobs.

If your cluster's security posture is "accept all authenticated federated submissions" (for example, an evaluation cluster or a single-tenant deployment), set the classpath to the shipped allow-all gate. Add this to the cluster's values file:

core:
  authz:
    executionGate:
      type: kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate
      config: {}

Then rerun the helm upgrade --install command from Install on Kubernetes with Istio with the same --version and the updated values file. The chart rolls the pods when the rendered configuration changes, and the explicit allow-all decision replaces the default mesh-deny policy.

An administrator can also bind a gate at runtime through the API, without a Helm upgrade. A gate bound through the API takes precedence over the Helm value until it is cleared; see Clearing an API binding. ADMIN_TOKEN is an admin token for the cluster, obtained as in Get an admin token:

curl --fail --silent --show-error \
  --request PUT --header "Authorization: Bearer $ADMIN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"type":"kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate","config":{}}' \
  "https://kamiwaza.example.com/api/cluster/execution-gate" | jq .

Installing a site-specific policy

For clusters with a real policy — clearance gating, tenant isolation, program compartments, etc. — point at your own gate class. The platform does not ship site-specific gates. Ship your class in one of two ways:

As a gate package. An administrator installs it through the API, and Kamiwaza loads its gate classes with no further configuration. Gate-package storage is off by default; that page says how to turn it on.
In your own build of the Core image. Also list the class's package in AUTH_GATE_CLASSPATH_PREFIXES, as described in Allow your gate's classpath. From 1.3.4, Kamiwaza refuses to load a class that is not listed.

Then set the classpath, for example in your values file:

core:
  authz:
    executionGate:
      type: acme_sec.policy.ClearanceExecutionGate
      config:
        min_clearance: C
        require_authenticated: true

Requirements for the class:

Its module must be importable on the Ray head, core-raycluster-head. The Kamiwaza API runs there: it evaluates the gate on every job submission and serves the binding API. A gate package is mounted there.
Its classpath must be allowed; see Allow your gate's classpath.
It must subclass kamiwaza.services.authz.gates.ExecutionGate and implement name (property), required_attributes(), and authorize().
The gate is created with the config dict. A changed config takes effect like any other gate change; see When a gate change takes effect.

Gate authors' guide: see the Kamiwaza Agent SDK docs for the AttributeGate / ExecutionGate plugin authoring interface.

Allow your gate's classpath

Kamiwaza imports a gate class only when its classpath:

starts with kamiwaza.services.authz.gates. or kamiwaza_extensions.,
belongs to an installed gate package, or
starts with a prefix listed in AUTH_GATE_CLASSPATH_PREFIXES.

This applies to the execution gate and to dataset gates, whether the gate is set through the API, the AUTHZ_EXECUTION_GATE environment variable, or the Helm value. Kamiwaza checks the classpath before it imports anything. Before 1.3.4, only the API checked it, so a class set through the environment variable or the Helm value loaded without the check. From 1.3.4, a class that fails it is refused:

Every job submission, local and mesh, returns 403 with reason gate_load_error.
GET /api/cluster/execution-gate returns 403 with reason classpath_not_allowed. Earlier 1.3 releases return the same, so you can check before you upgrade.
A dataset gated by such a class is not served: retrieval returns 503, and a job result that reads the dataset returns 403.

Classes from an installed gate package need nothing more. For a class in your own build of the Core image:

Choose the prefix: the name of the package that holds your gates, ending with a dot. For acme_sec.policy.ClearanceExecutionGate, use acme_sec.. A prefix allows every module whose name starts with it, so keep it as narrow as your package. Separate several prefixes with commas.
Add it to your values file:
```
core:
  scheduler:
    extraEnv:
      - name: AUTH_GATE_CLASSPATH_PREFIXES
        value: "acme_sec."
```
core.scheduler.extraEnv sets the variable on core-scheduler, on the Ray head and workers, and on core-api if you enabled it. It is a list, and Helm replaces a list as a whole: if any of your values files already sets core.scheduler.extraEnv, add this entry to that list and keep the entries already in it. If you enabled the retrieval streamer (core.retrievalStreamer.enabled: true), add the same entry to core.retrievalStreamer.extraEnv.
Rerun the helm upgrade --install command from your environment's install page, with the same --version and your updated values files. The Core pods restart, which interrupts the API and any job running on the Ray head. Before an upgrade to 1.3.4 or later, you can instead add the entry first and run the upgrade: it applies the entry.
Verify. As an administrator, GET /api/cluster/execution-gate returns 200, with your class in type:
```
curl --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/cluster/execution-gate" | jq .
```
For each gated dataset, GET /api/catalog/datasets/<URL-encoded URN>/gate returns 200.

Env var override (ad-hoc testing)

For one-off testing or debugging without a Helm rollout, you can set the env var on the Ray head, core-raycluster-head. The Kamiwaza API runs there: it enforces the gate on job submissions and reports it through GET. Setting it on core-scheduler has no effect. Setting or removing it restarts the Ray head, which interrupts the API and any job running there, so don't do it on a cluster that is serving work:

kubectl set env deployment/core-raycluster-head -n kamiwaza \
  AUTHZ_EXECUTION_GATE='{"type":"kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate","config":{}}'

The value is a JSON string. It takes precedence over the Helm value, but not over a gate bound through the API, and GET /api/cluster/execution-gate reports it with source: env.

The variable is stored in the Deployment's pod template, so restarts and rollouts keep it, and it keeps overriding the Helm value. Do not rely on a helm upgrade to remove it. Remove it explicitly when you are done:

kubectl set env deployment/core-raycluster-head -n kamiwaza \
  AUTHZ_EXECUTION_GATE-

Then check that GET /api/cluster/execution-gate no longer reports source: env; see Verifying the active gate.

Verifying the active gate

Ask the cluster which gate is active:

curl --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/cluster/execution-gate" | jq .

Expected when a gate is configured:

{
  "type": "kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate",
  "config": {},
  "gate_name": "allow_all_execution_gate",
  "kind": "execution",
  "source": "helm"
}

source names where the gate came from: api, env, or helm. When no gate is configured anywhere, the call returns 404 with reason not_configured. That is the correct state for the "default mesh-deny" posture.

When the highest layer that holds a value holds one that does not parse, the call returns 409 with reason execution_gate_invalid, the source that holds it, and a remediation. A lower layer's gate does not apply in its place.

Clearing an API binding

DELETE /api/cluster/execution-gate removes a gate bound through the API. A gate from the env var or the Helm value, if there is one, then applies again, and the response reports it in active_after:

{
  "deleted": true,
  "previous_type": "kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate",
  "active_after": {
    "type": "acme_sec.policy.ClearanceExecutionGate",
    "source": "helm"
  }
}

active_after is null when no gate remains. The API cannot remove a gate that comes only from the env var or the Helm value: DELETE then returns 409 with reason execution_gate_not_api_managed. Change the Helm value or remove the env var instead.

The Kamiwaza API logs the active gate's decision on every job submit. It writes to Ray's session logs on the Ray head, which kubectl logs does not show. To read them:

kubectl -n kamiwaza exec deploy/core-raycluster-head -- \
  grep -rh "Execution gate" /app/tmp/ray/session_latest/logs

Each decision line contains, for example, Execution gate allow_all_execution_gate decision: allow (reason=default_allow_all).

Troubleshooting

403 on a federated job submit

The API returns a structured body with the gate's reason code:

{
  "detail": {
    "reason": "no_execution_gate_configured_for_mesh",
    "gate": "(unconfigured)"
  }
}

Common reason codes:

`reason`	Meaning	Fix
`no_execution_gate_configured_for_mesh`	Receiving cluster has no execution gate set and the default policy denies mesh submissions.	Configure a gate on the receiving cluster; see Configuring a gate. If you set the Helm value, `GET /api/cluster/execution-gate` should report it with `source: helm`.
`gate_config_invalid`	A gate value is set but does not parse, so every submission is refused, local ones included.	`GET /api/cluster/execution-gate` returns `409` with reason `execution_gate_invalid` and the `source` that holds the value. Fix that value, or clear an API binding with `DELETE`.
`gate_load_error`	The configured class cannot be loaded: its classpath is not allowed, or its module cannot be imported.	If `GET /api/cluster/execution-gate` returns `403` with reason `classpath_not_allowed`, see Allow your gate's classpath. Otherwise find the import error in the Kamiwaza API's log on the Ray head, which `kubectl logs` does not show: `kubectl -n kamiwaza exec deploy/core-raycluster-head -- grep -rh "Failed to load execution gate" /app/tmp/ray/session_latest/logs`. Make sure the module is importable on the Ray head.
`gate_exception`	The gate's `authorize()` method raised.	Find the traceback in the Kamiwaza API's log on the Ray head: `kubectl -n kamiwaza exec deploy/core-raycluster-head -- grep -rh -A 20 "raised during authorize" /app/tmp/ray/session_latest/logs`. Fix the gate, or unset the config to revert to defaults.
(site-specific)	Reason strings beyond the ones above come from the configured gate itself (e.g., `clearance_insufficient`, `tenant_mismatch`).	Look at the gate's own docs for the reason vocabulary.

When a gate change takes effect

The gate is resolved on every job submission, so a gate bound or cleared through the API applies to the next one. A Helm value change takes effect when helm upgrade rolls the pods, which the chart does whenever the rendered configuration changes. An env var change takes effect when the pods restart.

Ray worker pods do not evaluate the gate — it runs once per submit at the control plane, before Ray sees the job.

Gate configured but local submissions still bypass

That is correct behavior. Local submissions (from same-cluster authenticated users) also run through the configured gate — if your gate authorize() method returns ALLOW for them, they pass. If you want the gate to be the sole gatekeeper for both local and mesh, your gate's authorize() should inspect job_spec["source"] and apply identical rules to both. The platform's default policy is source-scoped only when no gate is configured.

Default behavior​

Configuring a gate​

Quick start: permit all mesh submissions​

Installing a site-specific policy​

Allow your gate's classpath​

Env var override (ad-hoc testing)​

Verifying the active gate​

Clearing an API binding​

Troubleshooting​

403 on a federated job submit​

When a gate change takes effect​

Gate configured but local submissions still bypass​

See also​