Routing Modes and URLs

Current Kamiwaza deployments use HTTPS ingress with path-based runtime URLs as the standard public access pattern.

In Kubernetes deployments, the charts seed path-based routing by default for:

models
apps
tools

Canonical Runtime Paths

The standard public URL shape is:

Models: /runtime/models/<deployment-id>
Apps: /runtime/apps/<deployment-id>
Tools: /runtime/tools/<deployment-id>

Examples:

https://kamiwaza.example.com/runtime/models/<deployment-id>
https://kamiwaza.example.com/runtime/apps/<deployment-id>
https://kamiwaza.example.com/runtime/tools/<deployment-id>

For model runtimes that expose OpenAI-compatible APIs, the model-scoped base URL follows the same pattern:

https://kamiwaza.example.com/runtime/models/<deployment-id>/v1

Best Practice

For customer-facing documentation, integrations, bookmarks, and extension guidance:

use path-based URLs as the canonical form
treat the platform HTTPS origin as the stable entrypoint
avoid documenting port-specific runtime URLs as the preferred access pattern

This aligns with the current Kubernetes deployment model. Istio is the default routing provider. Greymatter is the supported alternative on deployments where Greymatter Core is installed and managed outside the Kamiwaza chart.

(Note: In local development deployments using the k0s-podman or k0s-lima Mesh M1 configuration, TLS is handled via a self-signed placeholder certificate. For production, see Production Certificate Swaps.)

Production Certificate Swaps

When deploying Kamiwaza in a production environment with Mesh M1, you must replace the default placeholder certificate with a valid TLS certificate for your domain.

The typical procedure involves:

Configuring DNS: Ensure your target domain (e.g., kamiwaza.example.com) resolves to the external IP of the Istio Ingress Gateway.
Using cert-manager: Deploy cert-manager to automatically provision and rotate certificates from Let's Encrypt or your internal CA.
Updating the Istio Gateway secret: Configure your Istio Gateway resource to reference the TLS secret generated by cert-manager (or manually provisioned) so that the ingress terminates TLS correctly.

Routing Configuration API

Routing is managed through the platform configuration API:

GET /api/config/routing
PATCH /api/config/routing

The API supports:

base_host
service_prefixes
enabled_services

In the current implementation, the API reports and persists routing in path mode for the public runtime paths.

Default Kubernetes Routing Behavior

The deployment charts seed routing with path-based defaults similar to:

{
  "routing": {
    "mode": "path",
    "model_routing_method": "path",
    "base_host": "https://<platform-origin>",
    "service_prefixes": {
      "api": "/api",
      "frontend": "/",
      "models": "/runtime/models",
      "apps": "/runtime/apps",
      "tools": "/runtime/tools",
      "notebooks": "/notebooks",
      "admin_ray": "/admin/ray"
    }
  }
}

The customer-facing hostname is typically derived from the deployment domain, and the scheduler config maps that into:

KAMIWAZA_EXTERNAL_URL
KAMIWAZA_ORIGIN

Best practice:

keep your canonical public hostname aligned with global.domain
keep KAMIWAZA_ORIGIN aligned to the same HTTPS origin
only change routing prefixes when you have a clear platform-wide reason to do so

Platform Gateway for Extensions

Starting in 1.4, Kamiwaza gives each extension one in-cluster address for the platform, KAMIWAZA_PLATFORM_GATEWAY_URL, on every routing provider. Extensions call the platform through it instead of the public origin. The variables an extension reads are listed in the Extension Developer Guide.

The value is an origin only: scheme://host[:port], with no path. Extensions append /api/... themselves.

Routing provider	`KAMIWAZA_PLATFORM_GATEWAY_URL`
Envoy	`https://kamiwaza-envoy.<namespace>.svc.<cluster-domain>`, the Envoy edge Service. A `KAMIWAZA_INTERNAL_GATEWAY_URL` override on Core picks another edge origin.
GreyMatter	`https://<edge-service>.<namespace>.svc.<cluster-domain>:10908`, with `KAMIWAZA_GATEWAY_CA_FILE` and `KAMIWAZA_GATEWAY_TLS_SERVER_NAME`. Set only for extensions that declare `KAMIWAZA_PLATFORM_GATEWAY_CONTRACT=1`; other extensions get the deprecated `KAMIWAZA_TRAEFIK_GATEWAY_HOST` instead of the URL, with the same CA file and server name.
Istio	`https://<gateway Service host>`, taken from `KAMIWAZA_INGRESS_GATEWAY_SERVICE_HOST` on Core. Extensions reach the gateway through the chart's `core-direct-runtime-istio-gateway-egress` NetworkPolicy, on by default, and its DestinationRule. With the extension operator (`core.extensionRuntime.mode: operator`), the operator sets the same value once it has confirmed the gateway Service; see Extension operator on Istio.

On GreyMatter, Kamiwaza withholds the variable, and says why on the deployment status, when the edge Service is missing, the edge CA Secret is missing, the host is not an in-cluster Service, or KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME is set blank.

Before 1.4, Istio installs gave extensions no in-cluster gateway, so extensions called the platform through its public origin.

Extension operator on Istio

When the extension operator runs App Garden extensions (extensionOperator.enabled: true and core.extensionRuntime.mode: operator; public 1.4.0 installs do not support it, see Extension operator layout), it gives every extension the same KAMIWAZA_PLATFORM_GATEWAY_URL that Core gives on Istio: https://<gateway Service host>, with KAMIWAZA_GATEWAY_CA_FILE where the platform trust store is mounted, and no KAMIWAZA_GATEWAY_TLS_SERVER_NAME. These values of the extension operator chart control it:

Value	Default	Effect
`extensionOperator.istioGatewayEgress.enabled`	`true`	Passes `global.ingress.gateway.serviceHostname` to the operator and renders a DestinationRule in the extensions namespace that lets extension pods reach the gateway's TLS listener. Once the operator has read the gateway Service and found the port, it sets the variable on every extension in the extensions namespace and allows traffic to the gateway in each extension's own NetworkPolicy. `false` keeps operator-managed extensions on the public origin.
`extensionOperator.istioGatewayEgress.port`	`443`	The gateway Service port extensions connect to.
`extensionOperator.istioGatewayEgress.serviceReadGrant`	`true`	With the operator's default `rbacMode: namespaced`, creates the Role and RoleBinding that let the operator read the gateway Service. Set `false` when a cluster administrator manages them.

Installing the extension-operator chart on its own, the values are istioGatewayEgress.enabled, istioGatewayEgress.port and istioGatewayEgress.serviceReadGrant. global.ingress.gateway.serviceHostname is the gateway Service host, for example istio-ingressgateway.istio-system.svc.cluster.local. When it is unset, the host is istio-ingressgateway.<global.ingress.gateway.namespace>.svc.<cluster-domain>, with the namespace defaulting to istio-system. The operator chart reads the value this way:

It is lowercased, and any https:// or other scheme, a trailing dot on the host and a trailing slash are dropped.
A :port sets the gateway port. It replaces istioGatewayEgress.port when that is 443, and must match it otherwise.
A short name such as <service>.<namespace> or <service>.<namespace>.svc is expanded to <service>.<namespace>.svc.<cluster-domain>.

The install stops with an error if the value:

is a single name or not a valid host name, or carries a path, credentials or whitespace;
has a port outside 1-65535 or with a leading zero;
has a :port that differs from an istioGatewayEgress.port other than 443;
names a cluster domain other than global.clusterDomain, when that is set.

The error names the setting and removes any user:password@. Where Core's gateway checks render, the install can also stop if the host names a namespace other than global.ingress.gateway.namespace. Core's direct runtime uses the value as given, so with core.extensionRuntime.mode: direct, write the Service host with no scheme, and in full.

enabled: true needs the networking.istio.io/v1 DestinationRule CRD. Set enabled: false where that CRD or the gateway namespace is missing. The chart cannot see Core's settings, so also set it to false if the operator chart is enabled while core.extensionRuntime.mode is direct: otherwise it still renders the DestinationRule and the Role described below. On a non-chart operator install, set KAMIWAZA_INGRESS_GATEWAY_SERVICE_HOST and KAMIWAZA_INGRESS_GATEWAY_EXTENSIONS_NAMESPACE on the operator, and render that DestinationRule yourself in the extensions namespace. With the namespace variable unset, the operator gives the gateway to every namespace it reconciles. Without the rule, the condition can read True while the extension's sidecar wraps its TLS in mesh mTLS.

Only extensions in the extensions namespace get the gateway, because the DestinationRule applies there. Before setting the variable, the operator reads the gateway Service and checks that it exposes the port. If it cannot confirm the Service, it sets nothing and adds no egress, and extensions keep using the public origin. The one exception is a temporary API error after an earlier confirmation, described after the conditions below.

The operator needs get on that one Service. With the operator's default rbacMode: namespaced, the chart grants it with a Role and RoleBinding in the gateway Service's namespace, limited to that Service by name. With rbacMode: cluster, the operator's ClusterRole already allows it. A namespace-admin install (global.managedK8s.namespaceAdmin: true, or a platform profile that defaults to it) cannot create that Role, so under rbacMode: namespaced the chart skips it, as it does with serviceReadGrant: false, and its install notes describe the Role a cluster administrator has to add. To create the Role itself, the install identity needs rights to create a Role and RoleBinding in that namespace, and must itself be allowed to get that Service there. Without those rights, helm upgrade stops at the Role; set serviceReadGrant: false and have an administrator add it. A serviceHostname that is not an in-cluster Service name gets no Role and no DestinationRule, and the install notes say so. The Role and RoleBinding look like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <operator-namespace>-extension-operator-istio-gateway-reader
  namespace: istio-system            # the gateway Service's namespace
rules:
  - apiGroups: [""]
    resources: [services]
    resourceNames: [istio-ingressgateway]   # the gateway Service's name
    verbs: [get]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <operator-namespace>-extension-operator-istio-gateway-reader
  namespace: istio-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <operator-namespace>-extension-operator-istio-gateway-reader
subjects:
  - kind: ServiceAccount
    name: extension-operator         # the chart's serviceAccount.name
    namespace: <operator-namespace>

The extension operator reports the result on each extension's PlatformGateway condition:

True: the condition names the value.
False with PlatformGatewayUndiscovered: the gateway host is not usable. For example, it is the public host, or its Service was not found or has no such port. Nothing is set.
False with PlatformGatewayUnconfirmed: the operator could not confirm the gateway. It has not read the gateway Service yet, it may not read it, a temporary API error stopped the read and there is no earlier confirmation to keep, or the host is not an in-cluster Service name (<service>.<namespace>.svc[.<cluster-domain>]) and is not the public host. For a refused read, the message names the Service and the exact permission that is missing. Nothing is set.
False with PlatformGatewayNotDerived: the extension runs outside the extensions namespace. Extensions that declare KAMIWAZA_PLATFORM_GATEWAY_CONTRACT=1 also carry this condition when the operator was given no gateway host, for example because istioGatewayEgress.enabled is false.

When no gateway is set, extensions use the public origin, unless a service sets its own KAMIWAZA_PLATFORM_GATEWAY_URL. A service that does so and runs a released sandbox-controller image also needs spec.sandbox, because the operator then sets no SANDBOX_PROVIDER. An extension with a networkPolicy reaches the public origin only when that policy allows external access.

After a temporary API error, the operator keeps a gateway it confirmed within the last 5 minutes, and reads the Service again on its next pass, at least 10 seconds later or when those 5 minutes end. It holds that confirmation in memory, so an operator that restarts during an API outage has nothing to keep. While a gateway is confirmed, the operator reads the Service at most once a minute. When a read is refused or finds the Service missing or without the port, the gateway is withdrawn without that wait, after at most a brief retry of a refused read, as it is when a temporary error outlasts the 5 minutes: the variable and the egress are removed, and the running services of every extension that had the gateway restart. The operator reads the Service again at least 10 seconds later, and the gateway returns, with another restart, once the read succeeds.

Each operator-managed extension that gets the gateway restarts once after the upgrade, because its environment gains the new variables. Where an administrator adds the Role, that restart comes when the read first succeeds.

Core settings

Set these in core.scheduler.extraEnv, or in Core's environment on a non-chart install.

Setting	Default	Use
`KAMIWAZA_INGRESS_GATEWAY_SERVICE_HOST`	Set by the chart on Istio installs, from `global.ingress.gateway.serviceHostname`	The Istio gateway Service host, for example `istio-ingressgateway.istio-system.svc.cluster.local`, with an optional `:port`. Set it by hand only on a non-chart install or for a non-default gateway Service. Core never guesses this host: without it, Istio extensions get no `KAMIWAZA_PLATFORM_GATEWAY_URL`, and the deployment status says why.
`KAMIWAZA_CLUSTER_DOMAIN`	Set by the chart from `global.clusterDomain` when that is set; otherwise the `svc.<domain>` entry in Core's DNS search list, else `cluster.local`	The cluster DNS domain. Used for the default Envoy edge, the GreyMatter edge host and, when `KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME` is unset, the GreyMatter TLS server name. On chart installs, set `global.clusterDomain` instead, so the chart's certificates use the same domain.
`KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME`	Set by the chart from `global.greymatter.edge.tlsServerName`, else `global.domain`. Unset without the chart, where Core uses `<edge-service>.<namespace>.svc.<cluster-domain>`	The TLS server name Core gives extensions for the GreyMatter edge, as `KAMIWAZA_GATEWAY_TLS_SERVER_NAME`. On chart installs, change it through `global.greymatter.edge.tlsServerName`. Do not set it blank: a blank value turns the server name off, and Core then withholds `KAMIWAZA_PLATFORM_GATEWAY_URL` and says why on the deployment status.

Deprecated variables

Kamiwaza still sets these through 1.4.x and logs once per Core process that they are deprecated. They are removed after 1.4.x.

Variable	Replacement
`KAMIWAZA_INTERNAL_GATEWAY_URL` (Envoy)	`KAMIWAZA_PLATFORM_GATEWAY_URL`
`KAMIWAZA_TRAEFIK_GATEWAY_HOST` (GreyMatter, bare `host:port`)	`KAMIWAZA_PLATFORM_GATEWAY_URL` with `KAMIWAZA_GATEWAY_CA_FILE` and `KAMIWAZA_GATEWAY_TLS_SERVER_NAME`. On Core, it is still a valid override to pick the GreyMatter edge.
`KAMIWAZA_INTERNAL_ORIGIN`	`KAMIWAZA_PLATFORM_GATEWAY_URL`. It is still set, but it was never a gateway; do not use it as one.

A KAMIWAZA_INTERNAL_GATEWAY_URL override on Core must be an origin, with no path. Core appends /api itself.

TLS names for the platform gateway

An extension connects to KAMIWAZA_PLATFORM_GATEWAY_URL, but keeps the platform's public host as the HTTP Host header. It verifies the gateway's certificate against this name:

Routing provider	Name the extension verifies	Certificate
Envoy	The public host, `global.domain`. Calls through `KAMIWAZA_API_URL`, which Core points at the same edge, verify `kamiwaza-envoy.<namespace>.svc.<cluster-domain>`	`kamiwaza-tls` must carry both names. The generated certificate does.
Istio	The public host, `global.domain`	The gateway certificate, which already carries it
GreyMatter	`KAMIWAZA_GATEWAY_TLS_SERVER_NAME`: the value of `KAMIWAZA_GREYMATTER_EDGE_TLS_SERVER_NAME`, which the chart sets to `global.greymatter.edge.tlsServerName` or `global.domain`. When it is unset, `<edge-service>.<namespace>.svc.<cluster-domain>`	GreyMatter's edge certificate, managed outside the chart, must carry it

<cluster-domain> is the cluster's DNS domain. On Istio, an extension that follows this contract needs no extra certificate name.

Some clients verify the host in the URL instead. For them, the certificates the chart issues also carry the gateway Service's names: the short name, <service>.<namespace>, <service>.<namespace>.svc and <service>.<namespace>.svc.<cluster-domain>. That is the Envoy kamiwaza-envoy Service, or the Istio Service in global.ingress.gateway.serviceHostname. An Istio generated certificate issued before 1.4 is kept on upgrade without these names; reissue it by hand to add them. Set global.clusterDomain when the cluster's domain is not cluster.local. A generated certificate lists its names in the kamiwaza.ai/tls-dns-names annotation on its Secret. With a certificate you provide (network.tls.mode: byo), add the same names. On Envoy the full name kamiwaza-envoy.<namespace>.svc.<cluster-domain> is required, not optional: Core's own calls and extension calls through KAMIWAZA_API_URL dial it and verify that name. See Envoy with network.tls.mode: byo.

If the certificate lacks the name a client verifies, its calls through the gateway fail TLS hostname verification. On Envoy, Core checks the certificate in the edge Secret: it logs an error at startup and adds an entry to each extension deployment's platform_transport_warnings that names the missing host. On Istio and GreyMatter the deployment status does not report a missing certificate name.

Check which gateway extensions use

The deployment's status lists gateway problems: platform_transport_warnings on the app deployment in the API, and the extensions.kamiwaza.io/platform-transport-warnings annotation on the deployment's runtime ConfigMap.
GET /api/kamiwaza/install-modes reports the platform transport as one of internal_gateway, greymatter_gateway, public_origin, invalid, or extension_operator.
Each Kaizen service logs platform_transport_resolved source=<variable> at startup, naming the variable it used.

For the error texts and fixes, see Extension platform gateway issues.

Legacy Port-Based Routing

Some code paths still retain compatibility for legacy port-based or dual-mode routing behavior. That exists for backward compatibility, internal transitions, or older integrations.

For public docs and new integrations, path-based routing should be considered the supported standard unless your deployment team has explicitly documented a different compatibility requirement.

Migrating from Traefik

Traefik routing support was removed after the platform moved to Istio and Envoy ext_authz. An installation that explicitly sets KAMIWAZA_ROUTING_PROVIDER=traefik now fails during startup with migration guidance instead of silently selecting another provider.

To migrate an older installation:

Remove the traefik provider override and any network.traefik values.
For chart installs, set global.mesh.provider=istio and global.ingress.provider=istio. Select greymatter in both keys only when the cluster has a supported Greymatter Core installation. The chart renders KAMIWAZA_ROUTING_PROVIDER; set that environment variable directly only for a non-chart process launch.
Replace custom Traefik IngressRoute and Middleware resources with the route mechanism owned by the selected provider.
Render the deployment values and verify the canonical runtime paths before upgrading production traffic.

Do not translate Traefik ForwardAuth headers into application configuration. Both surviving providers carry the authoritative request path in :path.

App and Tool URL Behavior

App and tool deployments use the same runtime path model:

apps open under /runtime/apps/<deployment-id>
tools open under /runtime/tools/<deployment-id>

The built-in Workrooms page and App Garden experiences rely on these path-based URLs when launching runtime apps.

Administrative and Platform Paths

Not every internal or administrative surface is meant to be documented as a public runtime path.

The main customer-facing patterns to document are:

/api
/runtime/models/...
/runtime/apps/...
/runtime/tools/...

Additional administrative paths such as /admin/ray are platform-specific and should be documented only when they are intentionally exposed in a given environment.

When To Update Routing

Use the routing API when you need to:

move the canonical base host to a different customer-facing domain
change a service prefix intentionally across the platform
disable or hide a specific routed service prefix

Do not treat pod-local config files such as runtime_config.json as the primary manual control point for customer operations. In current deployments, the shared runtime configuration and Helm-rendered defaults are the source of truth.

Validation Steps

After a routing change, validate:

the platform home page loads at the intended HTTPS origin
/api/ping or another known API endpoint resolves correctly
at least one model URL works under /runtime/models/<deployment-id>/v1
at least one App Garden app opens under /runtime/apps/<deployment-id>
auth redirects and logout flows still return users to the correct origin

Use the Quickstart and Observability guides if a routing change breaks app launches or browser flows.

Canonical Runtime Paths​

Best Practice​

Production Certificate Swaps​

Routing Configuration API​

Default Kubernetes Routing Behavior​

Platform Gateway for Extensions​

Extension operator on Istio​

Core settings​

Deprecated variables​

TLS names for the platform gateway​

Check which gateway extensions use​

Legacy Port-Based Routing​

Migrating from Traefik​

App and Tool URL Behavior​

Administrative and Platform Paths​

When To Update Routing​

Validation Steps​