Federation setup

This guide pairs two Kamiwaza clusters using the receiver-controlled shared_idp identity mode. The receiver validates the caller's shared-realm token, then applies its own brokered-user allowlist, ReBAC, and policy gates.

It applies to Kamiwaza 1.3.3 clusters.

receiver_realm is reserved for the future receiver-owned guest-identity workflow and is rejected by Kamiwaza 1.3.3. peer_kc remains available for compatibility, but new peer_kc pairs are refused unless the operator explicitly enables ALLOW_UNTRUSTED_FEDERATION.

Prerequisites

• Two Kamiwaza 1.3.3 clusters, each installed with Install on Kubernetes with Istio and each serving its own domain. This guide uses fed-a.example.com and fed-b.example.com; substitute your own. Keep each cluster's values file; step 0 adds to it.
• HTTPS reachability in both directions: each cluster reaches the other's ingress gateway on port 443.
• Hostname peer addresses, or an ingress gateway that preserves the caller's source address. See Use hostnames for the peer address.
• Peer name resolution inside each cluster. See Resolve the peer's domain.
• Edge certificates each cluster trusts. See Trust the peer's certificate.
• One independently owned shared OIDC realm, its issuer URL, and a way for shared-realm users to obtain access tokens from it. Both clusters must reach the issuer over HTTPS and trust its certificate. The receiver fetches the realm's signing keys from <issuer>/protocol/openid-connect/certs, the Keycloak path, so the issuer must serve them there.
• The admin user on each cluster.

Resolve the peer's domain

Kamiwaza calls the peer and the shared realm's issuer from inside the cluster, so pods on each cluster must resolve the other cluster's domain to an address that reaches the peer's gateway, and the issuer's host to the issuer. If your DNS already answers for both domains and pods use it, nothing more is needed. Otherwise add a cluster DNS entry on each cluster. With CoreDNS, add a hosts block to the Corefile, above the kubernetes plugin. On fed-a:

hosts {
  10.0.0.12 fed-b.example.com
  10.0.0.20 idp.example.com
  fallthrough
}

10.0.0.12 is an address of fed-b's ingress gateway that fed-a's pods can reach, and 10.0.0.20 is the issuer's. Make the reciprocal entry, and the issuer entry, on fed-b. Distributions that manage the CoreDNS configuration, k0s among them, need the change made through their own configuration, as described in DNS. Allow a minute or two for CoreDNS to reload before testing.

Use hostnames for the peer address

Give each remote_ips entry the peer's domain as its ip, for example {"ip": "fed-a.example.com", "primary": true}. The peer's domain must resolve inside the cluster; see Resolve the peer's domain.

A receiver accepts a pairing request only from its peer. For a hostname entry, the request's signature is that check, so the pre-shared key is the only proof that a request comes from the peer: handle it like a password. For a literal IP entry, the receiver also requires the request to arrive from that address, as its ingress gateway sees it. Most ingress paths rewrite the address: a gateway Service with the default externalTrafficPolicy: Cluster, and host-level forwarding such as a host that forwards ports 80 and 443 to a NodePort, or a Lima VM.

If you must use literal IP addresses, see Connect by IP address, and preserve the caller's address:

• For a gateway Service exposed directly as a LoadBalancer or NodePort, set externalTrafficPolicy: Local. This does not help when host-level forwarding sits in front of the Service.
• For a load balancer in front of the gateway, enable the PROXY protocol on both the load balancer and the Istio ingress gateway, for example with the gateway annotation proxy.istio.io/config: '{"gatewayTopology": {"proxyProtocol": {}}}'. See Istio's PROXY protocol guide. Enabled on one side only, it breaks every connection through that load balancer, not only pairing.

Otherwise preflight passes but pairing fails with source_ip_denied. The error reports the address the receiver saw, in observed_source, with a remediation.

Trust the peer's certificate

Each cluster verifies the peer's edge certificate when it tests, pairs, and pings. The peer's domain must appear in the subject alternative names of the peer's edge certificate, and its issuer must be trusted:

• Publicly trusted CA: nothing to do.
• Private CA: publish every CA that signs either cluster's edge certificate, concatenated into one file, in the kamiwaza-edge-trust ConfigMap on both clusters. The simplest setup issues both edge certificates, and the shared realm's certificate, from one private CA. Create the ConfigMap before installing, as described in Edge certificate. If you change it on an installed cluster, restart the scheduler: kubectl -n kamiwaza rollout restart deployment/core-scheduler.
• Self-signed: federation with self-signed edge certificates is for evaluation only. Set core.scheduler.tlsInsecure: true in step 0 on both clusters. This turns off certificate verification on federation control calls, so never use it in production.

Pairing then exchanges and pins each cluster's platform CA for mesh traffic.

0. Add federation values to each cluster

Add these values to each cluster's values file:

Value	Set it to
core.initialClusterName	A name unique across the federation, for example fed-a and fed-b. Every install that leaves it unset is named Default Cluster.
core.scheduler.trustedSharedIssuers	A list holding the shared realm's issuer URL. Empty, the default, rejects every shared-realm token.
core.delegatedJobs.enabled, core.delegatedJobs.admission.enabled	true, on a cluster that runs federated jobs. See Job Submission.
core.authz.executionGate	An execution gate, on a cluster that runs federated jobs. Without one, the cluster refuses every job submitted from a peer. See Execution Gates.

On fed-a, for example:

core:
  initialClusterName: fed-a
  scheduler:
    trustedSharedIssuers:
      - https://idp.example.com/realms/federation

Then rerun the helm upgrade --install command from Install on Kubernetes with Istio with the same --version and the updated values file. The scheduler restarts with the new values.

A later name change takes effect after the scheduler restarts, but an already-paired peer keeps its cached name until the federation is re-paired. The display name labels and selects a federation; the hostname in remote_ips controls the HTTP Host header and TLS SNI.

Connect by IP address

For an IP-based connection, provide both the connect address and the hostname:

{"ip": "10.0.0.12", "hostname": "fed-b.example.com", "primary": true}

Kamiwaza connects to ip but sends hostname as the HTTP Host and TLS SNI. The hostname must be the peer's domain, which its edge certificate and Gateway already cover. On the receiver, a literal IP entry is also checked against the address the pairing request arrives from; see Use hostnames for the peer address.

Get an admin token

The API examples use these variables. The admin password is the one from Verify:

A_API=https://fed-a.example.com/api
B_API=https://fed-b.example.com/api

A_ADMIN_TOKEN=$(curl --fail --silent --show-error \
  --data-urlencode username=admin --data-urlencode "password=$A_ADMIN_PASSWORD" \
  "$A_API/auth/token" | jq -er .access_token)

Get B_ADMIN_TOKEN the same way against $B_API. SHARED_USER_TOKEN is an access token for a shared-realm user, issued by the shared realm. Shared-realm tokens are often short-lived (five minutes by default in Keycloak); fetch a new one when a call returns 401.

1. Verify both routes before pairing

In the console, select Cluster > Federations > Add Request, enter the peer address, and use Test Connection. Resolve DNS, TCP, TLS, or L7 failures before creating either record.

The API equivalent is admin-only:

curl --fail --silent --show-error \
  --request POST \
  --header "Authorization: Bearer $A_ADMIN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"remote_ips":[{"ip":"10.0.0.12","hostname":"fed-b.example.com","primary":true}]}' \
  "$A_API/cluster/federations/preflight" | jq .

Run the same check in the opposite direction. Pairing needs bidirectional reachability.

2. Create the receiver record

On the receiving cluster:

1. Select Cluster > Federations > Add Request.
2. Choose Wait for remote cluster.
3. Enter the initiator's unique cluster name, and its domain as the connect address. See Use hostnames for the peer address.
4. Enter a strong PSK from your approved secret manager.
5. Choose shared_idp.
6. Enter the shared issuer URL. If the issuer uses a private CA, supply the complete CA chain. Kamiwaza derives the JWKS URL from the issuer and ignores any JWKS URL you enter.
7. Create the request.

The receiver should show a WAITING record.

3. Create and pair the initiator

On the initiating cluster:

1. Select Cluster > Federations > Add Request.
2. Choose Initiate connection.
3. Enter the receiver's unique name, connect address, and hostname.
4. Use Test Connection.
5. Enter the same PSK and shared-realm values.
6. Create the request and approve the confirmation.

Both cards should reach PAIRED and show Receiver-controlled with the same shared issuer. Use Ping and verify that Last Ping updates.

Pairing automatically exchanges the platform CA certificates. Do not update cluster_federations.remote_ca_cert with SQL and do not copy a raw PSK from the database.

Raw API equivalent

The console uses a two-step API:

1. POST /api/cluster/federations creates the receiver and initiator records.
2. POST /api/cluster/federations/{id}/pair on the initiator drives the handshake.

Keep the PSK in mode-0600 request files so it does not appear in process arguments or shell history. The initiator request shape is:

{
  "remote_cluster_name": "fed-b",
  "remote_ips": [
    {
      "ip": "10.0.0.12",
      "hostname": "fed-b.example.com",
      "primary": true
    }
  ],
  "preshared_key": "read-from-a-private-file",
  "callback_hostname": "fed-a.example.com",
  "role": "initiator",
  "shared_issuer_url": "https://idp.example.com/realms/federation",
  "shared_ca_pem": "optional-private-ca-chain"
}

The receiver request uses role: "receiver", the initiator's name and route, and the receiver's own domain as callback_hostname:

{
  "remote_cluster_name": "fed-a",
  "remote_ips": [
    {
      "ip": "fed-a.example.com",
      "primary": true
    }
  ],
  "preshared_key": "read-from-a-private-file",
  "callback_hostname": "fed-b.example.com",
  "role": "receiver",
  "shared_issuer_url": "https://idp.example.com/realms/federation",
  "shared_ca_pem": "optional-private-ca-chain"
}

Create the receiver first, then the initiator, then pair the returned initiator ID:

curl --fail --silent --show-error \
  --request POST --header "Authorization: Bearer $B_ADMIN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data-binary @receiver-create-private.json \
  "$B_API/cluster/federations" > receiver.json

curl --fail --silent --show-error \
  --request POST --header "Authorization: Bearer $A_ADMIN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data-binary @initiator-create-private.json \
  "$A_API/cluster/federations" > initiator.json

FEDERATION_ID="$(jq -er .id initiator.json)"
B_FEDERATION_ID="$(jq -er .id receiver.json)"
curl --fail --silent --show-error \
  --request POST --header "Authorization: Bearer $A_ADMIN_TOKEN" \
  "$A_API/cluster/federations/$FEDERATION_ID/pair" | jq .

Delete the private request files after pairing.

4. Grant receiver-local access

Pairing establishes trust; it does not grant access to datasets, models, or jobs. On the receiver, open the federation's Access panel and add each shared-realm subject that may use the pair. Assign only the required receiver-local grants, for example:

• dataset:<dataset-urn>#viewer for discovery and retrieval;
• the exact model relation required for chat; and
• cluster_jobs:__all__#executor for job submission.

The API equivalent, on the receiver, uses the receiver's federation ID:

curl --fail --silent --show-error \
  --request POST --header "Authorization: Bearer $B_ADMIN_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{
    "external_id": "<subject>@<initiator-cluster-id>",
    "initial_tuples": [
      {"subject": "user:{{user_id}}", "relation": "viewer", "object": "dataset:<dataset-urn>"},
      {"subject": "user:{{user_id}}", "relation": "executor", "object": "cluster_jobs:__all__"}
    ]
  }' \
  "$B_API/cluster/federations/$B_FEDERATION_ID/users" | jq .

For shared_idp, <subject> is the sub claim of the user's shared-realm token, and <initiator-cluster-id> is the initiating cluster's local_cluster_id, shown on the initiator's federation record (GET $A_API/cluster/federations/$FEDERATION_ID). A call from a subject that is not yet allowed fails with 403 and unauthorized_brokered_user; the response names the exact external_id to add. {{user_id}} is replaced with the receiver-local user ID when the receiver provisions the user on its first request. See Receiver-owned grants.

The receiver creates and controls the local brokered identity. Source-cluster administrator roles do not cross the federation boundary.

5. Verify the pair

Use the exact federation UUID when scripting. Name prefixes are rejected when they are ambiguous.

curl --fail --silent --show-error \
  --request POST --header "Authorization: Bearer $A_ADMIN_TOKEN" \
  "$A_API/cluster/federations/$FEDERATION_ID/ping" | jq -e '.reachable == true'

curl --fail --silent --show-error \
  --header "Authorization: Bearer $SHARED_USER_TOKEN" \
  "$A_API/mesh/$FEDERATION_ID/api/catalog/datasets/" | jq .

The catalog response must contain only datasets authorized on the receiver. For up to a minute after you grant access, calls can return 503 with authorization_unavailable while the receiver applies the grants; retry. Continue with Federated Retrieval and Job Submission.

Diagnose a failure

Run the stored-route diagnostic before changing records:

curl --fail --silent --show-error \
  --request POST --header "Authorization: Bearer $A_ADMIN_TOKEN" \
  "$A_API/cluster/federations/$FEDERATION_ID/diagnose" | jq .

Common stable reasons include dns_unresolvable, connection_failed, connection_timeout, tls_untrusted_ca, tls_hostname_mismatch, peer_identity_aliases_local, psk_propagation_timeout, peer_ca_projection_forbidden, and source_ip_denied. source_ip_denied appears only when pairing, after preflight has passed, and reports the address the receiver saw in observed_source; see Use hostnames for the peer address.

If CA inspection is required, support both deployed certificate layouts and never assume root-ca lives in the kamiwaza namespace:

if kubectl get secret core-internal-ca-tls -n kamiwaza >/dev/null 2>&1; then
  kubectl get secret core-internal-ca-tls -n kamiwaza \
    -o jsonpath='{.data.ca\.crt}' | base64 -d
else
  kubectl get secret root-ca -n kamiwaza-ca \
    -o jsonpath='{.data.ca\.crt}' | base64 -d
fi

This command is diagnostic only. Normal pairing exchanges and stores the CA.

Disconnect

Use Disconnect in the federation card or call the disconnect endpoint. Disconnect blocks new mesh admission and revokes delegated authority. Running work is reconciled through the job lifecycle; confirm terminal status before deleting evidence or receiver-local grants.