Execution Gates

The federated job service on the receiving cluster consults a pluggable execution gate before dispatching any job to Ray. The gate decides whether this user is allowed to run jobs on this cluster, independent of what data the job will read. Record-level filtering — "what columns and rows can this user see once the job runs?" — is handled separately by per-dataset attribute gates (see the retrieval docs).

This page covers the operator-facing surface: what the default behavior is, how to configure a policy, and how to diagnose 403 responses on federated job submissions.

Default behavior

An Execution Gate is a cluster-wide resource: one gate is in force at a time. When no gate is configured, the platform applies source-scoped default policy:

Submission source	Default when unconfigured	Reason code
local (same-cluster authenticated user)	ALLOW	no_execution_gate_configured_local_allowed
mesh (arrived via federation peer)	DENY (403)	no_execution_gate_configured_for_mesh

Local submissions preserve the pre-feature behavior of single-cluster clusters. Mesh submissions — the new cross-cluster surface — are denied until an operator explicitly declares the cluster's policy. This prevents a forgotten configuration from silently accepting federated work.

A gate value that is set but does not parse is not treated as unconfigured. Every submission, local or mesh, is refused with gate_config_invalid until the value is fixed; see Troubleshooting.

Configuring a gate

A gate can be set in three places. The cluster enforces the first one that holds a value:

1. The API. PUT /api/cluster/execution-gate binds a gate at runtime. It applies to the next submission, with no restart.
2. The AUTHZ_EXECUTION_GATE environment variable. Use it only for ad hoc testing.
3. The Helm value core.authz.executionGate.{type, config}, which the chart renders into the core-runtime-config ConfigMap.

Use one mechanism per cluster, so the active gate is not in doubt. GET /api/cluster/execution-gate reports the active gate and, in source, which of the three it came from: api, env, or helm. See Verifying the active gate.

Quick start: permit all mesh submissions

warning

AllowAllExecutionGate admits every authenticated federated submission. A mesh job submitted without delegated_access runs on the receiving cluster's shared Ray head, next to the platform's own services. Use it only for evaluation clusters and peers you trust. For any other peer, keep the default mesh-deny, or write a site-specific gate that admits only the peers and users you trust. A gate sees the job's source, source_cluster_id, user_id, entrypoint and target_datasets, but not whether it carries delegated_access. See Federated Job Submission for delegated jobs.

If your cluster's security posture is "accept all authenticated federated submissions" (for example, an evaluation cluster or a single-tenant deployment), set the classpath to the shipped allow-all gate. Add this to the cluster's values file:

core:
  authz:
    executionGate:
      type: kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate
      config: {}

Then rerun the helm upgrade --install command from Install on Kubernetes with Istio with the same --version and the updated values file. The chart rolls the pods when the rendered configuration changes, and the explicit allow-all decision replaces the default mesh-deny policy.

An administrator can also bind a gate at runtime through the API, without a Helm upgrade. A gate bound through the API takes precedence over the Helm value until it is cleared; see Clearing an API binding. ADMIN_TOKEN is an admin token for the cluster, obtained as in Get an admin token:

curl --fail --silent --show-error \
  --request PUT --header "Authorization: Bearer $ADMIN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"type":"kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate","config":{}}' \
  "https://kamiwaza.example.com/api/cluster/execution-gate" | jq .

Installing a site-specific policy

For clusters with a real policy — clearance gating, tenant isolation, program compartments, etc. — point at your own gate classpath:

core:
  authz:
    executionGate:
      type: acme_sec.policy.ClearanceExecutionGate
      config:
        min_clearance: C
        require_authenticated: true

Requirements for the classpath:

• The module must be importable on core-scheduler, which dispatches jobs and evaluates the gate, and on the Ray head, which serves the binding API. Bake it into the core image or install it as a gate package, which is mounted on both. The platform does not ship site-specific gates.
• The class must subclass kamiwaza.services.authz.gates.ExecutionGate and implement name (property), required_attributes(), and authorize().
• The gate is created with the config dict. A changed config takes effect like any other gate change; see When a gate change takes effect.

Gate authors' guide: see the Kamiwaza Agent SDK docs for the AttributeGate / ExecutionGate plugin authoring interface.

Env var override (ad-hoc testing)

For one-off testing or debugging without a Helm rollout, you can set the env var on the two Deployments that read the gate: core-scheduler, which dispatches jobs and enforces the gate, and core-raycluster-head, which serves the binding API. Set it on both. On the Ray head alone, GET reports a gate that dispatch does not enforce; on the scheduler alone, dispatch enforces a gate that GET does not report. Setting or removing it restarts both pods, which interrupts the API and any job running there, so don't do it on a cluster that is serving work:

kubectl set env deployment/core-scheduler deployment/core-raycluster-head -n kamiwaza \
  AUTHZ_EXECUTION_GATE='{"type":"kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate","config":{}}'

The value is a JSON string. It takes precedence over the Helm value, but not over a gate bound through the API, and GET /api/cluster/execution-gate reports it with source: env.

The variable is stored in the Deployment's pod template, so restarts and rollouts keep it, and it keeps overriding the Helm value. Do not rely on a helm upgrade to remove it. Remove it explicitly when you are done:

kubectl set env deployment/core-scheduler deployment/core-raycluster-head -n kamiwaza \
  AUTHZ_EXECUTION_GATE-

Then check that GET /api/cluster/execution-gate no longer reports source: env; see Verifying the active gate.

Verifying the active gate

Ask the cluster which gate is active:

curl --silent --show-error --header "Authorization: Bearer $ADMIN_TOKEN" \
  "https://kamiwaza.example.com/api/cluster/execution-gate" | jq .

Expected when a gate is configured:

{
  "type": "kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate",
  "config": {},
  "gate_name": "allow_all_execution_gate",
  "kind": "execution",
  "source": "helm"
}

source names where the gate came from: api, env, or helm. When no gate is configured anywhere, the call returns 404 with reason not_configured. That is the correct state for the "default mesh-deny" posture.

When the highest layer that holds a value holds one that does not parse, the call returns 409 with reason execution_gate_invalid, the source that holds it, and a remediation. A lower layer's gate does not apply in its place.

Clearing an API binding

DELETE /api/cluster/execution-gate removes a gate bound through the API. A gate from the env var or the Helm value, if there is one, then applies again, and the response reports it in active_after:

{
  "deleted": true,
  "previous_type": "kamiwaza.services.authz.gates.default_gates.AllowAllExecutionGate",
  "active_after": {
    "type": "acme_sec.policy.ClearanceExecutionGate",
    "source": "helm"
  }
}

active_after is null when no gate remains. The API cannot remove a gate that comes only from the env var or the Helm value: DELETE then returns 409 with reason execution_gate_not_api_managed. Change the Helm value or remove the env var instead.

core-scheduler logs the active gate's decision on every job submit (kubectl logs deployment/core-scheduler -n kamiwaza):

INFO  kamiwaza.cluster.jobs.gate_runner  Execution gate allow_all_execution_gate
      decision: allow (reason=default_allow_all)

Troubleshooting

403 on a federated job submit

The API returns a structured body with the gate's reason code:

{
  "detail": {
    "reason": "no_execution_gate_configured_for_mesh",
    "gate": "(unconfigured)"
  }
}

Common reason codes:

reason	Meaning	Fix
no_execution_gate_configured_for_mesh	Receiving cluster has no execution gate set and the default policy denies mesh submissions.	Configure a gate on the receiving cluster; see Configuring a gate. If you set the Helm value, GET /api/cluster/execution-gate should report it with source: helm.
gate_config_invalid	A gate value is set but does not parse, so every submission is refused, local ones included.	GET /api/cluster/execution-gate returns 409 with reason execution_gate_invalid and the source that holds the value. Fix that value, or clear an API binding with DELETE.
gate_load_error	Configured classpath cannot be imported.	Check the core-scheduler logs for the ImportError; make sure the module is importable there.
gate_exception	The gate's authorize() method raised.	Check the core-scheduler logs for the traceback; fix the gate or unset the config to revert to defaults.
(site-specific)	Reason strings beyond the ones above come from the configured gate itself (e.g., clearance_insufficient, tenant_mismatch).	Look at the gate's own docs for the reason vocabulary.

When a gate change takes effect

The gate is resolved on every job submission, so a gate bound or cleared through the API applies to the next one. A Helm value change takes effect when helm upgrade rolls the pods, which the chart does whenever the rendered configuration changes. An env var change takes effect when the pods restart.

Ray worker pods do not evaluate the gate — it runs once per submit at the control plane, before Ray sees the job.

Gate configured but local submissions still bypass

That is correct behavior. Local submissions (from same-cluster authenticated users) also run through the configured gate — if your gate authorize() method returns ALLOW for them, they pass. If you want the gate to be the sole gatekeeper for both local and mesh, your gate's authorize() should inspect job_spec["source"] and apply identical rules to both. The platform's default policy is source-scoped only when no gate is configured.

See also

• Operations & Troubleshooting — general federation ops runbook, including federation-pair setup and diagnostic queries.
• Federated Job Submission — the client side of the /cluster/jobs/ API.
• Federation API Reference — full endpoint reference.