Federated Retrieval

Federated retrieval lets you discover and query data on remote clusters without the data leaving its origin. The mesh proxy routes your requests transparently — the remote cluster's catalog and retrieval services process them as if they were local.

Prerequisites

• A paired federation between the clusters
• Remote CA cert stored in the federation record
• The mesh user must have per-dataset ReBAC grants on the target cluster (admin users with extauthz trust are exempt)

Discover Remote Datasets

# List datasets on the remote cluster
curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/catalog/datasets/" \
  -H "Authorization: Bearer $TOKEN"

Response: array of dataset metadata (URNs, names, schemas) from the remote cluster's catalog.

Retrieve Remote Data

# Create a retrieval job on the remote cluster
curl -sk -X POST "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_urn": "urn:li:dataset:(urn:li:dataPlatform:postgres,nps_verbatims,PROD)",
    "columns": ["customer_id", "verbatim", "score"],
    "filters": {"quarter": "Q1-2026"},
    "transport": "inline"
  }'

The retrieval service on the remote cluster accesses its local data adapters (S3, Postgres, Hive, etc.) and returns the results through the mesh proxy. Only the query results cross the network — raw data stays on the remote cluster.

Streaming Retrieval (SSE)

For large result sets, use the streaming transport:

# Submit job, then stream results
JOB_ID=$(curl -sk -X POST "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"dataset_urn": "...", "transport": "sse"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["job_id"])')

# Stream results via SSE
curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/retrieval/jobs/$JOB_ID/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: text/event-stream"

SSE events stream through the mesh proxy transparently.

List Remote Models

curl -sk "https://kamiwaza.test/api/mesh/<remote-cluster>/api/serving/deployments" \
  -H "Authorization: Bearer $TOKEN"

Returns deployment metadata: model names, engine types, status, and access paths.

Run Remote Inference

# Chat completion on a remote model
curl -sk -X POST \
  "https://kamiwaza.test/api/mesh/<remote-cluster>/runtime/models/<deployment-id>/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-4B",
    "messages": [{"role": "user", "content": "Summarize the NPS themes"}]
  }'

The inference request routes through the mesh proxy to the remote cluster's Ray Serve deployment. The response streams back.

Access Control

Admin Users

On Istio clusters with the extauthz trust model, admin users bypass per-resource ReBAC checks on the target cluster (standard admin bypass). No additional seeding is needed.

Non-Admin Users

Non-admin users require explicit per-dataset ReBAC grants on the target cluster. The admin bypass is suppressed for mesh-originated requests — federated callers must have explicit viewer access to each dataset they query or browse. The catalog listing endpoint filters results so federated users only see datasets they have grants for.

Grant access at federation-pairing time through the brokered-user allowlist's initial_tuples field — not through /api/auth/tuples.

Why not /api/auth/tuples?

A federated caller has no local user account on the target cluster until their first mesh request, when the brokering service auto-provisions a local Keycloak user with a freshly-minted UUID. The per-dataset check authorizes against that local UUID, which isn't known ahead of time — so a tuple written against the source cluster's user UUID via /api/auth/tuples never matches, and the call returns 204 while access stays denied (404 Dataset not found). The allowlist's initial_tuples solves this with a {{user_id}} placeholder that renders to the local UUID at provision time.

# On the TARGET cluster, add the source user to the federation allowlist
# WITH the dataset grant. `{{user_id}}` renders to the local brokered KC
# UUID when the user is auto-provisioned on first mesh request, so the
# viewer relation lands on the subject the per-dataset check looks up.
curl -sk -X POST "https://<TARGET_IP>/api/cluster/federations/<FEDERATION_ID>/users" \
  -H "Authorization: Bearer $REMOTE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "external_id": "<source-user-uuid>@<source-cluster-uuid>",
    "initial_tuples": [
      {
        "subject": "user:{{user_id}}",
        "relation": "viewer",
        "object": "dataset:<dataset-urn>"
      }
    ]
  }'

initial_tuples entries use namespaced "<namespace>:<id>" strings; {{user_id}} (local brokered UUID) and {{external_id}} (<source-user-uuid>@<source-cluster-uuid>) are substituted at provision time. To add a grant for an already-provisioned brokered user, re-POST the allowlist entry — seeding is idempotent.

Mesh Proxy Endpoint Reference

All federated operations use the mesh proxy at /api/mesh/{cluster_selector}/:

Parameter	Description
cluster_selector	Remote cluster name, UUID, or federation ID. Prefix matching supported (e.g., studio matches studio-1).
{path}	The API path on the remote cluster. Paths starting with runtime/ are routed as-is; all others get an /api/ prefix.

Timeouts

Path Type	Default Timeout	Env Override
API paths	60 seconds	MESH_PROXY_READ_TIMEOUT_SECONDS
Runtime paths (inference)	600 seconds	MESH_PROXY_RUNTIME_READ_TIMEOUT_SECONDS

Headers

The mesh proxy adds these headers to cross-cluster requests:

Header	Purpose
X-KZ-Mesh-Source-Cluster-Id	Originating cluster UUID
X-KZ-Mesh-User-Id	Authenticated user ID
X-KZ-Mesh-User-Roles	User roles (comma-separated)
X-KZ-Mesh-Signature	HMAC-SHA256 signature
X-KZ-Mesh-Correlation-Id	Request correlation ID for cross-cluster tracing
X-KZ-Mesh-Route	Hop trace for loop detection