Federated Retrieval

Federated retrieval lets you discover and query data on remote clusters without the data leaving its origin. The mesh proxy routes your requests transparently — the remote cluster's catalog and retrieval services process them as if they were local.

Prerequisites

• A paired federation between the clusters
• A dataset registered in the receiver's Data Catalog that needs no stored credential. The S3 example on that page uses one, so federated retrieval can't read it. See Credential-backed datasets.
• An active receiver-side allowlist entry for the shared subject
• Explicit per-dataset ReBAC grants on the target cluster. Mesh-originated requests do not receive the native-cluster admin bypass.

Discover Remote Datasets

# List datasets on the remote cluster
curl -sk "https://kamiwaza.example.com/api/mesh/<remote-cluster>/api/catalog/datasets/" \
  -H "Authorization: Bearer $TOKEN"

Response: array of dataset metadata (URNs, names, schemas) from the remote cluster's catalog.

Retrieve Remote Data

# Create a retrieval job on the remote cluster
curl -sk -X POST "https://kamiwaza.example.com/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_urn": "urn:li:dataset:(urn:li:dataPlatform:file,sales_summary.csv,PROD)",
    "columns": ["region", "quarter", "revenue"],
    "limit_rows": 100,
    "transport": "inline"
  }'

The retrieval service on the remote cluster reads the dataset through its local data adapters and returns the results through the mesh proxy. Adapters that read a stored credential, such as S3 or Postgres with a credential_secret, don't work for federated callers yet; see Credential-backed datasets. Only the query results cross the network — raw data stays on the remote cluster.

Streaming Retrieval (SSE)

For large result sets, use the streaming transport:

# Submit job, then stream results
JOB_ID=$(curl -sk -X POST "https://kamiwaza.example.com/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataset_urn": "...", "transport": "sse"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["job_id"])')

# Stream results via SSE
curl -sk "https://kamiwaza.example.com/api/mesh/<remote-cluster>/api/retrieval/jobs/$JOB_ID/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

SSE events stream through the mesh proxy transparently.

Credential-backed datasets

A federated caller can't retrieve a dataset whose connector reads a stored credential, such as an S3 bucket or a database registered with a credential_secret. The receiver reads the credential as the caller, and a dataset grant doesn't extend to the credential, so the retrieval fails. The same applies to a local user who can view the dataset but doesn't own the credential. Use datasets that need no stored credential for federated retrieval.

List Remote Models

curl -sk "https://kamiwaza.example.com/api/mesh/<remote-cluster>/api/serving/deployments" \
  -H "Authorization: Bearer $TOKEN"

Returns deployment metadata: model names, engine types, status, and access paths.

Run Remote Inference

# Chat completion on a remote model
curl -sk -X POST \
  "https://kamiwaza.example.com/api/mesh/<remote-cluster>/runtime/models/<deployment-id>/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-4B",
    "messages": [{"role": "user", "content": "Summarize the NPS themes"}]
  }'

The inference request routes through the mesh proxy to the remote cluster's Ray Serve deployment. The response streams back.

Access Control

Receiver-owned grants

All mesh users require explicit per-dataset ReBAC grants on the target cluster. The native-cluster admin bypass is suppressed for mesh-originated requests. The catalog listing endpoint filters results so federated users see only datasets they have receiver-local grants for.

Grant access at federation-pairing time through the brokered-user allowlist's initial_tuples field — not through /api/auth/tuples.

Why not /api/auth/tuples?

A federated caller has no local user account on the target cluster until their first mesh request, when the brokering service auto-provisions a local Keycloak user with a freshly-minted UUID. The per-dataset check authorizes against that local UUID, which is not known ahead of time — so a tuple written against the source cluster's user UUID via /api/auth/tuples never matches, and the call returns 204 while access stays denied (404 Dataset not found). The allowlist's initial_tuples solves this with a {{user_id}} placeholder that renders to the local UUID at provision time.

# On the TARGET cluster, add the source user to the federation allowlist
# WITH the dataset grant. `{{user_id}}` renders to the local brokered KC
# UUID when the user is auto-provisioned on first mesh request, so the
# viewer relation lands on the subject the per-dataset check looks up.
curl -sk -X POST "https://<TARGET_IP>/api/cluster/federations/<FEDERATION_ID>/users" \
  -H "Authorization: Bearer $REMOTE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "external_id": "<source-user-uuid>@<source-cluster-uuid>",
    "initial_tuples": [
      {
        "subject": "user:{{user_id}}",
        "relation": "viewer",
        "object": "dataset:<dataset-urn>"
      }
    ]
  }'

initial_tuples entries use namespaced "<namespace>:<id>" strings; {{user_id}} (local brokered UUID) and {{external_id}} (<source-user-uuid>@<source-cluster-uuid>) are substituted at provision time.

initial_tuples are fixed when the entry is created: a second POST for the same external_id returns 409, and restore cannot replace them. Include every grant the user needs when you add them. To grant more after the user has made their first request, write the tuple with POST /api/auth/tuples against the user's receiver-local ID, found with the authorization_relations query in Operations: ReBAC Denied.

Mesh Proxy Endpoint Reference

All federated operations use the mesh proxy at /api/mesh/{cluster_selector}/:

Parameter	Description
cluster_selector	An exact federation UUID, an exact remote cluster UUID, or the remote cluster name as an exact or prefix match. Use the federation UUID in scripts. A name or prefix that matches more than one pair returns 409 mesh_target_ambiguous.
{path}	The API path on the remote cluster. Paths starting with runtime/ are routed as-is; all others get an /api/ prefix.

Timeouts

Path Type	Default Timeout	Env Override
API paths	60 seconds	MESH_PROXY_READ_TIMEOUT_SECONDS
Runtime paths (inference)	600 seconds	MESH_PROXY_RUNTIME_READ_TIMEOUT_SECONDS

Headers

The mesh proxy adds these headers to cross-cluster requests:

Header	Purpose
X-KZ-Mesh-Source-Cluster-Id	Originating cluster UUID
X-KZ-Mesh-User-Id	Authenticated user ID
X-KZ-Mesh-User-Roles	User roles (comma-separated)
X-KZ-Mesh-Signature	HMAC-SHA256 signature
X-KZ-Mesh-Correlation-Id	Request correlation ID for cross-cluster tracing
X-KZ-Mesh-Route	Hop trace for loop detection

Identity-mode-dependent trust

X-KZ-Mesh-User-Id/-Roles are source-asserted. In shared_idp (receiver-controlled) mode the receiver establishes identity from the caller's own validated shared-realm token and strips source-asserted cluster roles; the forwarded values are the identity only in source-trusted peer_kc mode. See Identity Trust Modes.