Skip to main content

Connector Destinations

A connector registered by endpoint is a service that Kamiwaza calls at a URL an administrator supplies. Kamiwaza calls it from inside the cluster, so an unchecked endpoint could point Kamiwaza at its own internal services or at a cloud metadata service. From Kamiwaza 1.4, Kamiwaza checks every connector endpoint before it connects:

  • At registration and subscription (POST /api/connectors/register and POST /api/connectors/subscriptions), before Kamiwaza contacts the endpoint or stores it.
  • On every later call to a stored endpoint: execute, verify, whoami and the manifest refresh after a settings change. Kamiwaza resolves the host again on each call, checks every address it gets, and connects only to an address it checked. A hostname that later resolves to a private address (DNS rebinding) is refused.

Connectors that Kamiwaza deployed itself, such as those in Run Your Own Connector, keep working: Kamiwaza admits the exact Service it deployed for that connector.

What a connector may reach​

A connector endpoint is admitted when it resolves only to:

  • public internet addresses;
  • a host, domain or address range the operator lists site-wide (see Site-wide allowlist);
  • a host:port an administrator approved for that connector (see Per-connector approved destinations); or
  • for calls to a stored endpoint, the Service Kamiwaza deployed for that connector.

Everything else is refused. That includes private (RFC 1918), CGNAT, site-local and other non-public addresses, which you can admit with a listing or an approval.

The floor​

Some addresses are refused even when they are listed or approved:

  • loopback (127.0.0.0/8, ::1) and 0.0.0.0/8;
  • link-local addresses;
  • cloud metadata services: 169.254.169.254, AWS fd00:ec2::254 and fd00:ec2::23, GCP fd20:ce::254, OCI fd00:c1::a9fe:a9fe, Azure WireServer 168.63.129.16 and Alibaba 100.100.100.200;
  • unspecified, multicast and reserved addresses, Teredo (2001::/32), and NAT64 addresses in 64:ff9b::/32 other than the well-known 64:ff9b::/96.

The 0.0.0.0/8 and cloud metadata refusals also apply to Kamiwaza's other outbound destinations, so an external model endpoint or a custom S3 endpoint cannot target them either.

The only way past the floor is the development-only override, which an install refuses under the production environment and the stig security profile.

Endpoint URL rules​

A connector endpoint must:

  • use http or https;
  • carry no credentials (user:password@), query string or fragment;
  • name a host, as an ASCII hostname (letters, digits, -, _ and .). Give an internationalized name in its punycode (xn--) form;
  • contain no control characters or IPv6 zone (%25...);
  • have a port from 1 to 65535, if it gives one.

An IP address is accepted only when it falls inside a CIDR that you list site-wide. Legacy IPv4 spellings such as 2130706433 or 127.1 count as IP addresses.

A refused registration returns HTTP 400 with a message that says exactly what is not allowed and names the setting that would admit it, for example:

Connector endpoint host 'crm.internal.example' is not allowed: it must resolve,
from Core, only to public addresses, or the host or its address must be listed
in KAMIWAZA_CONNECTOR_ENDPOINT_ALLOWLIST. Private and CGNAT addresses are
refused unless listed. ...

That setting is the Helm value core.connectors.endpointAllowlist (see Site-wide allowlist).

Manifest fetch at registration​

When you register a connector, Kamiwaza fetches <endpoint>/manifest. That fetch:

  • connects only to the address Kamiwaza checked;
  • accepts only HTTP 200 and does not follow redirects;
  • times out after 3 seconds to connect and 5 seconds per read, and gives up 10 seconds after it connects, whatever is still arriving;
  • refuses compressed manifests and manifests over 1 MiB.

Every fetch failure returns the same message, so the response does not reveal what is listening at an address. Kamiwaza logs the cause as connector_manifest_fetch_failed, with the endpoint's scheme, host and port only.

These connections ignore HTTP_PROXY and HTTPS_PROXY, like Kamiwaza's other address-checked clients. A site that reaches connectors only through a forward proxy cannot register them by endpoint; the failure log says proxy=ignored.

Site-wide allowlist​

List internal hosts that any connector may reach with the Helm value core.connectors.endpointAllowlist. It is empty by default. Each entry is one of:

  • a hostname, such as connector.internal.example.com;
  • a domain suffix with a leading dot, such as .kamiwaza.svc.cluster.local;
  • a CIDR, such as 10.20.0.0/16.

An entry has no scheme and no port. An exact host:port for one connector is a per-connector approval instead.

core:
  connectors:
    endpointAllowlist:
      - connector.internal.example.com
      - .kamiwaza.svc.cluster.local

Apply it with helm upgrade --install and the same values files you installed with. The chart renders the list as KAMIWAZA_CONNECTOR_ENDPOINT_ALLOWLIST on core-api, the scheduler and the Ray head and workers.

The render fails when:

  • an entry is a URL, a host:port, a CIDR with a prefix out of range or characters an address cannot contain, or contains a comma or a space. Kamiwaza would reject the whole list and then fail every connector call;
  • you also set KAMIWAZA_CONNECTOR_ENDPOINT_ALLOWLIST as a core.scheduler.extraEnv (or core.scheduler.linuxNvidia.extraEnv) entry. Move the list into the value, one list item per comma-separated entry, and remove the extraEnv entry. An extraEnv entry on its own still works while the value is empty.

The chart catches common mistakes only; Kamiwaza validates every entry in full. An entry the chart lets through but Kamiwaza cannot parse, such as the CIDR 12345::/64, renders and installs. Kamiwaza then rejects the whole list: registration returns 500 naming the entry, connector calls fail with 500 naming the setting, and Kamiwaza logs connector_endpoint_allowlist_invalid with the entry. Fix the entry and run helm upgrade again.

List as narrowly as you can. Each entry admits every connector to that destination:

  • Prefer one host, or .<namespace>.svc.cluster.local for the namespace your connectors run in.
  • A cluster-wide .svc.cluster.local also admits the Kubernetes API, the platform's databases and every other Service.
  • A Service or pod CIDR admits every hostname that resolves into it.
  • The two entries 0.0.0.0/0 and ::/0, each its own list item (an entry with a comma fails the render), restore the private-network reach connectors had before 1.4, for new registrations too, while still refusing the floor. They also admit the Kubernetes API, databases and every other internal host. Use them only as a last resort during an upgrade, and replace them with narrow entries. While any /0 entry is listed, the scheduler logs connector_endpoint_allowlist_wide_open when it starts.

Use the site-wide list for address ranges, domain suffixes and for registering a new connector at an internal host: a per-connector approval needs the connector to exist first. For one existing connector, an approval is narrower.

Per-connector approved destinations​

An administrator can approve one exact host:port for one existing connector. An approval:

  • records who approved it, when, and why (a reason of 1 to 500 characters is required), and emits a kamiwaza.audit.sensitive_data.change audit event;
  • admits whatever non-floor address the host resolves to on each call, like a site-wide host entry. Approve only names whose DNS you control;
  • relaxes the policy only down to the floor. A floor address, localhost, or a host that includes a port is refused.

To review and revoke approvals in the console, open Settings > Connectors. Under Approved destinations by connector, each approval shows its destination, who approved it and when, and its reason. Approvals made by the upgrade show Grandfathered on upgrade. Select Revoke and give a reason.

A revoked approval admits nothing. It is kept as history, and the upgrade never restores it. To admit the destination again, approve it again; that adds a new approval.

Through the API, as an administrator:

# Approve one destination for one connector
curl -sS -X POST "$KAMIWAZA_URL/api/connectors/<connector id>/approved-destinations" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"host": "crm.internal.example", "port": 8443, "reason": "CRM connector, change 1234"}'

# List the connector's active approvals
curl -sS "$KAMIWAZA_URL/api/connectors/<connector id>/approved-destinations" \
  -H "Authorization: Bearer $TOKEN"

# Revoke one (a reason is required)
curl -sS -X POST \
  "$KAMIWAZA_URL/api/connectors/<connector id>/approved-destinations/<approval id>/revoke" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"reason": "CRM moved to a public endpoint"}'

Approving a floor IP address or localhost returns 409. A hostname is not resolved when you approve it; the floor still applies to its addresses on every call. A non-administrator gets 403 on every route on this page.

Existing connectors on upgrade​

When you upgrade to 1.4, existing connectors keep working without a site-wide entry. On its first start with 1.4, core-api records the upgrade time before it serves any request. Then core-api and the scheduler each run a one-time pass, so an install without the scheduler is covered too. For every connector that existed before the upgrade and whose stored endpoint now resolves to a private, non-floor address, the pass approves that exact host:port for that connector only.

These approvals:

  • show as Grandfathered on upgrade under Settings > Connectors;
  • are logged as connector_destination_approved and audited as a system change;
  • can be revoked like any other approval. A revoked approval stays revoked.

The pass ignores the site-wide allowlist, so if you listed a broad range for the upgrade, you can narrow it later without breaking a grandfathered connector.

The pass does not grandfather:

  • Floor addresses or invalid stored URLs, for example with credentials, a query string or fragment, trailing whitespace, or a bad host or port. Re-register these connectors with a valid endpoint.
  • Hosts that do not resolve during the pass, or whose Kubernetes ownership lookup fails. Kamiwaza retries them at later starts for up to 7 days, then logs connector_destination_grandfather_expired and leaves them for an administrator to approve.
  • Connectors registered after the upgrade. Admit their hosts site-wide or with an approval.

Until the first pass finishes, a few seconds after the first start, calls to those connectors are refused and recorded in the blocked destinations view. They succeed once their approvals exist.

At startup, the scheduler logs connector_endpoint_outside_policy for each enabled connector whose endpoint is still refused, and connector_endpoint_invalid for each one whose stored URL is invalid, with advice to re-register it.

Blocked connector destinations​

Every destination Kamiwaza refuses for a connector is recorded: the connector, the host:port, the address class, the reason, when it was first and last seen, and how many times. Repeat refusals update one row, and the list keeps at most 1,000 rows. Each refusal is also logged as connector_destination_refusal and counted in the kamiwaza.connectors.destination.refusals metric, by error.type.

Administrators see the list in the console under Settings > Connectors

Blocked connector destinations, or with GET /api/connectors/destination-refusals.

To admit a refused destination for its connector, select Allow this destination for this connector and give a reason. That creates a per-connector approval for exactly that host:port. Through the API:

curl -sS -X POST "$KAMIWAZA_URL/api/connectors/destination-refusals/<refusal id>/approve" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"reason": "Approved after review, change 1234"}'

Some rows cannot be allowed. The console shows why, and the API returns 409:

ShownWhyWhat to do
Never allowed: <address class>The host resolves to a floor address.Point the connector at another address.
Invalid endpoint: re-register the connectorThe stored URL breaks the URL rules.Re-register the connector with a valid endpoint.
Refused before resolution: see the Core logKamiwaza refused the call before it resolved an address, so there is no destination to approve.Read the core-api log for the rule.

Allowing a destination here does not change what the connector's caller saw: callers always get the same refusal error, whatever the cause.

Development-only floor override​

For local development against a service on the developer's own machine, the Helm value core.connectors.unsafeDevAllowLoopbackAndMetadata: true lets a listed or approved destination also be loopback, link-local, cloud metadata or 0.0.0.0/8. Never use it on a shared install.

It works only on a development install:

  • The render fails unless the effective KAMIWAZA_ENV (global.environment, or a core.scheduler.extraEnv entry) is dev, development, local, test or testing. A blank value counts as production, and community and default are not development.
  • The render fails under the stig security profile (global.securityProfile or core.security.profile set to stig, or an extraEnv AUTH_SECURITY_PROFILE or KAMIWAZA_SECURITY_PROFILE of stig).
  • core-api and the scheduler refuse to start outside a development KAMIWAZA_ENV or when AUTH_SECURITY_PROFILE is stig, with ConnectorUnsafeDevOverrideRefused, so setting KAMIWAZA_CONNECTOR_UNSAFE_DEV_ALLOW_LOOPBACK_AND_METADATA by another route does not get around it.

While the override is on, the notes Helm prints after an install or upgrade show WARNING: UNSAFE DEVELOPMENT SETTING, Kamiwaza logs a warning at startup, and it logs UNSAFE connector_destination_floor_bypassed for every call it admits below the floor. A development KAMIWAZA_ENV also relaxes Kamiwaza's other production-only hardening.

Troubleshooting​

Symptom or log eventMeaningFix
Registration returns 400 naming KAMIWAZA_CONNECTOR_ENDPOINT_ALLOWLISTThe endpoint resolves to a non-public address that is not listed.List the host in core.connectors.endpointAllowlist, narrowly.
Registration returns 400 about the URLThe endpoint breaks a URL rule.Fix the URL.
Registration fails with a generic manifest errorThe manifest fetch failed: refused, timed out, not 200, a redirect, compressed, or too large.Read connector_manifest_fetch_failed in the core-api log.
connector_destination_refusedA call to a stored endpoint was refused.Check Blocked connector destinations, then allow it or list it.
A connector call fails as a retryable transient_failure (503)Kamiwaza could not confirm with Kubernetes that it deployed the Service the connector points at, so it sent nothing.Retry. If it persists, list the host site-wide, or redeploy the connector.
connector_endpoint_outside_policy at scheduler startupAn existing connector is still refused after the upgrade.Approve it for that connector, or list it.
connector_endpoint_invalidThe stored URL is invalid and is refused on every call.Re-register the connector.
connector_endpoint_allowlist_wide_openA /0 entry is listed.Replace it with narrow entries.
connector_endpoint_allowlist_invalidAn allowlist entry is invalid; every connector call fails with 500.Fix the entry named in the log.
core-api or the scheduler does not start: ConnectorUnsafeDevOverrideRefusedThe development override is on outside a development install or under stig.Turn the override off.