Skip to main content

Configuration Reference

Current Kamiwaza deployments are Kubernetes-based. In customer environments, configuration should be managed through your Helm values and cluster release workflow, not through ad hoc edits to source-managed env.sh files.

Configuration Model​

In the current deployment model, configuration comes from three main layers:

  • Helm values and override files in the deployment workflow
  • Kubernetes ConfigMaps and Secrets rendered from those values
  • a small set of runtime-managed settings stored through platform APIs

For most teams, the best practice is:

  1. set customer-specific defaults in Helm values
  2. keep secrets in Kubernetes Secrets or an external secret manager
  3. use the platform API only for runtime settings that are explicitly designed to be changed after deployment

Start With These Docs​

Use these guides together:

Core Platform Settings​

Domain and Origin​

The most important public-access setting is the deployment domain.

In the Kubernetes deployment charts, this is typically set through:

global:
  domain: kamiwaza.example.com

That domain is then used to derive platform-facing values such as:

  • KAMIWAZA_EXTERNAL_URL
  • KAMIWAZA_ORIGIN
  • allowed browser origins for authenticated platform traffic

Best practice:

  • set global.domain to the canonical customer-facing hostname
  • keep KAMIWAZA_ORIGIN aligned to that same HTTPS origin. Starting in 1.4, Core on Kubernetes refuses to start unless KAMIWAZA_ORIGIN is an http(s) URL with a host; the chart always sets it from global.domain
  • avoid mixing multiple “primary” hostnames unless your ingress and auth setup explicitly require it

Authentication and Security Configuration​

Authentication, identity provider integration, consent, banners, ReBAC, and secrets management are covered in the Administrator Guide.

Best practice:

  • manage auth and banner settings through your Helm values and platform admin workflow
  • keep secrets out of plain-text values files whenever possible
  • use Kubernetes Secrets, sealed secrets, or your cluster’s secret-management pattern for credentials and keys
Never put credentials in a Keycloak or OIDC URL

Starting in 1.4.0, when authentication is enabled, Core refuses to start when a Keycloak, Keycloak admin, JWKS, issuer, or external OIDC endpoint URL setting carries embedded credentials (user:password@). The error contains Refusing to start: and names the setting. Remove the user:password@ part. Set client credentials in AUTH_GATEWAY_KEYCLOAK_CLIENT_ID and AUTH_GATEWAY_KEYCLOAK_CLIENT_SECRET for the Keycloak client, or AUTH_GATEWAY_OIDC_CLIENT_ID and AUTH_GATEWAY_OIDC_CLIENT_SECRET for an external OIDC identity provider. The chart already sets the Keycloak administrator password from the keycloak-admin Secret. See Keycloak and OIDC URL credentials for the settings checked, and Remove credentials from Keycloak and OIDC URLs for the pre-upgrade check.

TLS Trust for Private Certificate Authorities​

By default, Kamiwaza verifies every TLS certificate it is presented. When a certificate is issued by a private CA, add that CA to Kamiwaza's trust anchors instead of turning verification off.

External model endpoints behind a private CA, such as a private Amazon Bedrock endpoint, or a chat, embedding, or reranking endpoint behind a corporate CA or a TLS-inspecting proxy: add the root CA, and any issuing CA, to global.trust.extraAnchors:

global:
  trust:
    extraAnchors:
      - name: corporate-root
        pem: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----

Each entry is a PEM string or {name, pem}. The chart checks every anchor when it renders and fails on one with no certificate in it, or with only a CA-issued leaf certificate: include the root CA, not the server's own certificate. A self-signed certificate is its own root and is accepted. Kamiwaza adds the anchors to the trust store of Core, Ray, Keycloak, and App Garden extension pods, beside the public roots. Extensions get it through SSL_CERT_FILE and the related CA bundle variables, so Kaizen verifies external model hosts against it. Extensions pick up changed anchors the next time they are deployed. Core's pods build their trust store when they start, and changing only the anchors does not restart them, so restart them after the upgrade, for example with kubectl -n kamiwaza rollout restart deployment/core-api deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker. Also restart core-retrieval-streamer if you enabled it. Keycloak restarts on its own when the anchors change.

The platform's own domain, when its edge certificate is not issued by a publicly trusted CA (always on OpenShift with GreyMatter): set global.trust.edgeAnchor.pem to the CA chain that signs the domain, root included, not the leaf. For a self-signed edge certificate, use the certificate itself. extraAnchors does not stand in for it.

global:
  trust:
    edgeAnchor:
      pem: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----

The chart publishes edgeAnchor.pem and extraAnchors to Kamiwaza's workloads in the kamiwaza-edge-trust ConfigMap, and it is the only writer of that ConfigMap. Do not create or edit it yourself. If an earlier installation had you create kamiwaza-edge-trust by hand, move the CA that signs the domain into global.trust.edgeAnchor.pem and any other CA it holds into extraAnchors, then, right before you rerun helm upgrade, delete it:

kubectl -n kamiwaza delete configmap kamiwaza-edge-trust

While a ConfigMap of that name that Helm does not manage exists, an install or upgrade that sets either value, or that generates the edge certificate (network.tls.mode: generated), stops with ConfigMap kamiwaza-edge-trust already exists in namespace kamiwaza and is not managed by Helm, before it changes anything. Without either value, a hand-made ConfigMap still reaches Kamiwaza's workloads, and the install notes print a warning that asks you to move it into values. The check reads the cluster, so helm template and client-side dry runs skip it. The global.trust.edgeAnchor.publishedBySubstrate, .configMapName, and .key values are deprecated and are removed in 1.5.0; the install notes warn while any of them is set.

Extension operator layout​

The extension-operator runtime is not supported on any install of the public Kamiwaza 1.4.0 release: the chart published at oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza, whether it pulls its images from ghcr.io/kamiwaza-ai/releases/ or from your own registry, as an air-gapped install or another mirror does. The public release publishes no extension-operator image, so a mirror of it has none either.

When global.imageRegistryPrefix is under ghcr.io/kamiwaza-ai/releases/, the chart refuses the operator when it renders and stops before anything is installed. It refuses any extensionOperator.enabled value other than the YAML boolean false, including the string "false" and an empty value:

Extension operator: the extension-operator runtime is not supported on public Kamiwaza 1.4.0 installs (global.imageRegistryPrefix=ghcr.io/kamiwaza-ai/releases/kamiwaza), and the release publishes no extension-operator image, so extensionOperator.enabled=true would leave the operator unable to pull its image. Set extensionOperator.enabled: false and use the direct runtime (core.extensionRuntime.mode: direct, the chart default), and rerun helm.
warning

The chart cannot tell that a mirrored prefix, such as <your registry host>/kamiwaza-ai/releases/kamiwaza, holds the public release, so it does not refuse the operator there. On a mirrored or air-gapped install, remove the operator values yourself, as described below; otherwise the install succeeds and the operator pod cannot pull its image.

Public installs run App Garden extensions through the direct runtime, which is the chart default: Core applies each extension's workloads itself. Leave extensionOperator.enabled unset or false, and leave core.extensionRuntime.mode at direct:

core:
  extensionRuntime:
    mode: direct
extensionOperator:
  enabled: false

If your values file sets extensionOperator.enabled: true or core.extensionRuntime.mode: operator, remove whichever it sets before you install or upgrade to 1.4.0. No public release has published an extension-operator image, so on a public install with extensionOperator.enabled: true the operator pod could not pull its image, and core.extensionRuntime.mode: operator handed extensions to an operator that never ran.

The direct runtime also needs Kamiwaza's extensions and sandboxes in the release namespace, and no cluster-scoped extension RBAC. The chart defaults meet both, but an operator values file often overrides them. Also remove any global.namespaces.extensions or global.namespaces.sandboxes that names another namespace, and any core.clusterScopedRbac.create: true or core.clusterScopedRbac.extensionManagerMode override. If one remains, the render stops with an error that starts extensionRuntime.mode=direct requires and names the value to change. The direct runtime also needs core.extensionRuntime.kubernetesApi.egressCidrs; see Required values.

Wherever the operator runs (extensionOperator.enabled: true and core.extensionRuntime.mode: operator), for example on an install that Kamiwaza packages with the operator outside the public release, the chart also checks its layout when it renders. A mirrored public install is checked the same way, but is still not supported. The chart stops, naming the value to set, if Core and the operator disagree:

RequirementValues
The operator's extensions namespace is Core's.extensionOperator.extensionsNamespace equals global.namespaces.extensions.
The operator watches Core's extensions namespace.If extensionOperator.watchNamespaces is set (with the default extensionOperator.rbacMode: namespaced), it includes global.namespaces.extensions. Leave it unset to use the default watch set.
Core's sandbox namespace is the operator's.global.namespaces.sandboxes equals extensionOperator.sandboxNamespace (default kamiwaza-sandboxes). Change whichever one names a namespace that does not exist.
On Istio, an https workload registrar URL names the platform domain.extensionOperator.workloadRegistrar.apiUrl is https://<global.domain>/api/v1/delegated-workloads. The gateway serves TLS only for the platform domain, so another host resets the TLS handshake.

Extension trust bundle​

This applies when the extension operator runs App Garden extensions (extensionOperator.enabled: true and core.extensionRuntime.mode: operator), which public 1.4.0 installs do not support (see Extension operator layout). The umbrella chart defaults to core.extensionRuntime.mode: direct, where Core applies extension workloads itself and the setting below has no effect. The operator mounts the kamiwaza-trust-bundle ConfigMap (key ca-certificates.crt) into extension pods as SSL_CERT_FILE, together with the kamiwaza-edge-trust anchors. That file replaces the image's own trust store, so it must hold the public roots as well as the platform CA. Otherwise, outbound HTTPS from the extension to public hosts, for example pypi.org, fails certificate verification, while HTTPS to platform services still verifies.

With core.certs.mode: generated, the chart publishes a complete bundle: the generated platform CA plus public roots, read from the pinned core.utilityImages.kubectl image. core.trustManager.generatedBundle.publicRoots.enabled: false publishes the platform CA alone, an anchors-only bundle. If you publish kamiwaza-trust-bundle with trust-manager, include useDefaultCAs: true in the Bundle sources.

The operator checks each extension's trust sources and sets the extension's TrustBundleReady condition to False with one of these reasons:

ReasonTrust sources
TrustBundleIncompletekamiwaza-trust-bundle holds no public roots (an anchors-only bundle).
TrustBundleMissingkamiwaza-edge-trust is published, but kamiwaza-trust-bundle is not.
TrustBundleInvalidThe ca-certificates.crt key of kamiwaza-trust-bundle is missing or not valid PEM.

What happens next depends on one value of the extension operator chart:

ValueDefaultEffect
extensionOperator.extensions.trust.requireCompleteBundlefalsefalse (warn): the condition carries a warning, and phase and Ready follow the extension's workloads; Kamiwaza does not report it as failed for trust. true (strict): the extension goes to phase Failed with Ready=False, and the operator creates no workloads for it.

Installing the extension-operator chart on its own, the value is extensions.trust.requireCompleteBundle. The chart sets KAMIWAZA_REQUIRE_COMPLETE_TRUST_BUNDLE on the operator from it, and the install notes name the setting when strict mode is on. Unset or empty means false. Accepted values are Go booleans: true, false, 1, 0, t, f (any case); yes and on are refused. The operator refuses to start if the variable holds any other value. On an operator run outside the chart, the --require-complete-trust-bundle flag overrides the variable, including a malformed one, which the operator then logs and ignores. The operator logs the effective mode at startup (extension trust mode, requireCompleteTrustBundle).

Warn is the default because installations without internet access legitimately run an anchors-only bundle: with no route to public hosts, the missing public roots cost nothing. Turn strict mode on only where extensions are expected to reach public hosts and you want a bundle without public roots to stop them rather than warn:

extensionOperator:
  extensions:
    trust:
      requireCompleteBundle: true

Turning strict mode on does not tear an extension down: one already running with such a bundle goes Failed at its next reconcile and the operator stops reconciling it, but its existing Deployments are not deleted.

A complete bundle is unaffected by the mode. A damaged kamiwaza-edge-trust next to a complete bundle fails the extension in both modes.

The condition message says what breaks and how to fix it. For an anchors-only bundle, the warning reads:

TrustBundleIncomplete: kamiwaza-trust-bundle in <ns> holds <n> certificate(s) and no public roots. It is mounted as SSL_CERT_FILE in <workload-ns> and replaces the image trust store, so outbound HTTPS from this extension to public hosts (for example pypi.org) fails certificate verification; HTTPS to platform services still verifies. To fix, publish public roots in kamiwaza-trust-bundle (trust-manager useDefaultCAs) or use the chart-generated bundle. This warning does not fail the extension; set extensions.trust.requireCompleteBundle (KAMIWAZA_REQUIRE_COMPLETE_TRUST_BUNDLE, or the --require-complete-trust-bundle flag) to true to fail it instead.

In strict mode the message ends: The extension is Failed because extensions.trust.requireCompleteBundle (KAMIWAZA_REQUIRE_COMPLETE_TRUST_BUNDLE, or the --require-complete-trust-bundle flag) is true; set it to false to warn instead.

To fix any of the three reasons, publish a complete kamiwaza-trust-bundle: keep core.trustManager.generatedBundle.publicRoots.enabled at its default true with core.certs.mode: generated, or add useDefaultCAs: true to your trust-manager Bundle. On an installation without internet access, you can instead leave strict mode off and accept the warning. To see the condition:

kubectl get kamiwazaextension <name> -n <namespace> \
  -o jsonpath='{.status.conditions[?(@.type=="TrustBundleReady")]}'

Context Service TLS verification​

From 1.4.0, the Context Service verifies the TLS certificate on its runtime model calls (embeddings and LLM calls) to the platform's own origins. This applies to new installations and to upgraded ones. Calls to external model hosts are verified too, also from 1.4.0; see externalTlsInsecureWithDnsRetargetingRisk. Core verifies against its trust store: the public roots plus the anchors in global.trust. If the edge certificate is not publicly trusted, its CA must be in global.trust. Put the CA that signs the domain in global.trust.edgeAnchor.pem: a domain CA only in global.trust.extraAnchors verifies at runtime, but the install check below does not accept it.

ValueDefaultEffect
core.context.embedding.verifySsltruefalse turns verification off for these calls, which was the behavior before 1.4.0. The chart renders it as CONTEXT_SERVICE_EMBEDDING_SSL_VERIFY. A value you set always wins over a setting stored by an earlier release. A CONTEXT_SERVICE_EMBEDDING_SSL_VERIFY entry in core.scheduler.extraEnv or core.scheduler.linuxNvidia.extraEnv overrides this value; if several entries set it, the last one wins.
global.trust.edgeAnchor.publiclyTrustedunsettrue declares that the edge certificate chains to a public CA, so no edge anchor is needed. Verification stays on. If the declaration is wrong, the calls fail; it never turns verification off. It also silences the install-notes warning about a missing edge anchor.

core.scheduler.tlsInsecure: true also turns verification off for these calls, because it covers every platform origin.

While verification is off by any setting, the install notes print WARNING: TLS verification is DISABLED for: .... The warning names each value that turns it off, for example Context Service runtime model calls (core.context.embedding.verifySsl=false). At every start, Core logs the settings in its own environment that turn verification off, in its own wording.

Edge certificates minted outside the chart​

The chart cannot read the CA of an edge certificate it did not create, or one issued from a CA it does not hold. That is the case on:

  • OpenShift with GreyMatter;
  • Istio or Envoy with network.tls.mode: byo, the default. This includes EZUA, whose gateway certificate EZUA issues;
  • Istio or Envoy with network.tls.mode: customerCA. A customer CA you added to the ca chart's trust bundle is not visible to this chart, so put the same CA in global.trust.edgeAnchor.pem;
  • Istio or Envoy with network.tls.mode: cert-manager when global.certificates.issuers.app.name is not the platform's app-ca;
  • Istio or Envoy with network.tls.mode: generated when the chart does not generate the certificate itself.

cert-manager with the platform app-ca issuer is not checked: Core's trust store already carries that CA.

On these installations, an install or upgrade fails before it changes anything if all of the following are true:

  • the release publishes no edge anchor: global.trust.edgeAnchor.pem is empty, or network.enabled is false;
  • publiclyTrusted is not true;
  • verification is on: neither core.context.embedding.verifySsl: false, a CONTEXT_SERVICE_EMBEDDING_SSL_VERIFY (off) or AUTH_GATEWAY_TLS_INSECURE (on) entry in core.scheduler.extraEnv, nor core.scheduler.tlsInsecure: true is set. The last entry of a name wins. An entry set through valueFrom cannot be read by the chart and counts as verifying;
  • no kamiwaza-edge-trust ConfigMap that Helm does not manage exists. The check reads the cluster, so helm template and client-side dry runs never see that ConfigMap, and they fail where a real upgrade passes;
  • the deprecated global.trust.edgeAnchor.publishedBySubstrate is not true. When it is true, the check passes unless the cluster shows no substrate-created kamiwaza-edge-trust ConfigMap (missing, or owned by this release). helm template cannot see the cluster and accepts it. configMapName or key alone never count.

The error contains preflight: Context runtime model calls verify TLS by default since 1.4.0. In parentheses after edgeAnchor.pem it names any setting that looks like a fix but does not count: a pem with network.enabled=false, a missing substrate ConfigMap, or a valueFrom entry. It then lists three fixes, with a hint for the CA to use on GreyMatter, customerCA and a custom cert-manager issuer. Their risks differ:

  1. Publish the edge CA (recommended). Set global.trust.edgeAnchor.pem to the CA chain that signs the domain, root included. On GreyMatter, this is the ca.crt of Secret greymatter-edge-ingress:

    oc -n kamiwaza get secret greymatter-edge-ingress \
      -o jsonpath='{.data.ca\.crt}' | base64 -d > edge-ca.crt

    With customerCA, use the customer CA; with a custom cert-manager issuer, that issuer's CA. Verification stays on, and every Kamiwaza workload can verify the platform. network.enabled must not be false, because the network chart publishes the anchor.

  2. Declare a publicly trusted edge certificate. Set global.trust.edgeAnchor.publiclyTrusted: true only if the edge certificate chains to a public CA. It never applies on GreyMatter, whose edge certificate is always issued by your mesh CA. Verification stays on. If the certificate is in fact private, Context calls fail with a certificate error.

  3. Turn verification off. core.context.embedding.verifySsl: false restores the behavior before 1.4.0. The Context Service then accepts any certificate for the platform's own origins, so anyone who can redirect the platform's domain, through site DNS or from inside the cluster, can read the calls and their tokens. The install notes and Core's startup log name the setting until you remove it. Without an anchor, App Garden extensions still cannot verify the platform either. Use it only until you can publish the CA.

global:
  trust:
    edgeAnchor:
      pem: |                       # fix 1: the CA chain that signs the domain
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      # publiclyTrusted: true      # fix 2: only for a publicly trusted edge certificate
# core:
#   context:
#     embedding:
#       verifySsl: false           # fix 3: verification off

After you set or change an anchor, restart Core's pods as described in TLS Trust for Private Certificate Authorities. If verification fails at runtime, Core logs TLS certificate verification failed for platform origin <host> once for each host, with the trust store it used and the fix.

Envoy with network.tls.mode: byo: the certificate also needs the gateway Service names. Publishing the CA is not enough on Envoy. Core and App Garden extensions call the platform inside the cluster at https://kamiwaza-envoy.<namespace>.svc.<cluster-domain> and check that the certificate names that host (extensions do so for calls through KAMIWAZA_API_URL). Besides the platform domain, the certificate in network.tls.secretName must list kamiwaza-envoy.<namespace>.svc.<cluster-domain> in its subjectAltName (cluster.local unless you set global.clusterDomain). The certificate the chart generates also lists kamiwaza-envoy, kamiwaza-envoy.<namespace> and kamiwaza-envoy.<namespace>.svc; add them too if your CA allows names without a domain.

Without the full name, extension calls through KAMIWAZA_API_URL, embedding API requests and App Garden readiness probes fail with Hostname mismatch, even though the CA is trusted. The install notes name it. Core checks the certificate at startup and logs an error that names the missing host; it also reports the problem in each extension deployment's platform_transport_warnings. On a hostname mismatch at runtime, Core logs TLS hostname verification failed for platform origin <host> instead of the trust-store message. After you reissue the certificate, update the Secret, restart Envoy (or change network.envoy.rolloutKey) and redeploy running extensions. The certificate the chart generates (network.tls.mode: generated) already carries these names.

core.scheduler.tlsInsecure​

core.scheduler.tlsInsecure: true turns off certificate verification for the platform's own origins: the platform's domain, in-cluster services, and federation control calls to paired clusters. One exception: when every model endpoint of an OmniParse instance may skip verification, each under its own setting (a platform origin under tlsInsecure, an external host only when externalTlsInsecureWithDnsRetargetingRisk names it), tlsInsecure turns off OmniParse's pod-wide VERIFY_SSL, which also covers document downloads from any host. OmniParse's force insecure model SSL setting, and an instance's own VERIFY_SSL: false, have the same pod-wide effect under the same condition. Core also passes tlsInsecure to App Garden extensions (KAMIWAZA_VERIFY_SSL=false). Kaizen already did not apply it to an external model host on 1.3, except in its reranker. From 1.4, neither Core (for example, Context Service model calls) nor any part of Kaizen applies it to an external model host, so this setting cannot work around a private CA on one in Core or Kaizen. Other extensions may still apply KAMIWAZA_VERIFY_SSL=false to any host. Use global.trust.extraAnchors for that, or, for a trial only, core.scheduler.externalTlsInsecureWithDnsRetargetingRisk.

Do not use it outside evaluation. Without verification, anyone who can make the platform's domain resolve somewhere else, for example through site or corporate DNS or from inside the cluster, can intercept the calls, including the members' tokens that Kaizen forwards to the platform. When Kaizen reaches the platform through the in-cluster gateway, its platform calls do not depend on site DNS, but other calls still do. Wherever you can, publish the CA with global.trust.edgeAnchor.pem or global.trust.extraAnchors instead.

core.scheduler.externalTlsInsecureWithDnsRetargetingRisk​

A trial or proof of concept sometimes has to reach an external model endpoint whose certificate cannot be verified yet, before anyone has set up the CA. core.scheduler.externalTlsInsecureWithDnsRetargetingRisk turns off certificate verification for external model hosts only. It is off by default and separate from tlsInsecure, which from 1.4 covers an external host only through the OmniParse VERIFY_SSL exception described above.

core:
  scheduler:
    # Every external model host:
    externalTlsInsecureWithDnsRetargetingRisk: true
    # Or only these exact, fully qualified hostnames (no wildcards,
    # no subdomains, no scheme or port):
    # externalTlsInsecureWithDnsRetargetingRisk:
    #   - models.example.com

It applies to the Context Service's and OmniParse's calls to external model hosts and to Kaizen. Core's External Models engines (such as Bedrock and OpenAI-compatible endpoints) always verify and ignore this setting and tlsInsecure; use global.trust.extraAnchors for them. From 1.4 it is the only setting that relaxes an external model host's model calls. Core verifies every external host it does not name, even with tlsInsecure: true. These settings still relax platform model services, but no longer an external host:

  • the Context Service TLS settings,
  • OmniParse's force insecure model SSL setting,
  • a request's vision model verify_ssl: false.

For Core, a host is external unless it is the platform's domain, a URL Core itself is configured with (such as its API, gateway, or identity provider URL), a loopback address, localhost, host.docker.internal, host.lima.internal, or an in-cluster *.svc or *.cluster.local name. A model server addressed by a private IP address (for example https://10.0.0.5) or a short hostname is external.

Kaizen is stricter. It never relaxes a short hostname, an IPv6 address, or an in-cluster *.svc or *.cluster.local name through this setting, even with true, and its KAMIWAZA_VERIFY_SSL=false relaxes only the exact platform URLs (scheme, host, and port) it is configured with. For a model server that Kaizen calls by a short name, use a fully qualified DNS name outside *.svc and *.cluster.local if one resolves. For a Service name or an IPv6 address, add the server's CA to global.trust.extraAnchors.

A list entry matches only that hostname: models.example.com does not cover eu.models.example.com. Each entry must be a fully qualified hostname (with at least one dot) or an IPv4 address such as 10.0.0.5. true relaxes every external model host, present and future, so prefer the list. An entry with a wildcard, scheme, port, or path, a single-label name, or an IPv6 address fails the install with a message that says why. In Core, true covers an IPv6 address or a short hostname; Kaizen never relaxes either.

The setting does not cover:

  • An OmniParse translation endpoint. Translation calls follow OmniParse's pod-wide VERIFY_SSL, which also governs document downloads. Until you open and save an instance on 1.4, it keeps its 1.3 VERIFY_SSL, which may leave verification off for an external host. Once you save it, Core turns VERIFY_SSL on whenever the instance has an external translation endpoint, or any external endpoint that must verify. Add an external translation host's CA to global.trust.extraAnchors before you save. In a mixed instance a self-signed platform translation endpoint that tlsInsecure used to cover is verified too, so trust its CA through global.trust.edgeAnchor.pem or global.trust.extraAnchors.
  • A token endpoint that receives a client secret, such as the OmniParse service-identity token endpoint. Core always verifies it unless it is a platform origin.

The name states the risk. Without verification, Kamiwaza sends the API keys it holds for that provider to whoever answers DNS for the host or sits on the network path. The install notes print a warning while it is on, Core and Kaizen log a warning that names the hosts, and Kaizen's diagnostics show it. Use it only for a trial. For anything you keep, add the provider's CA to global.trust.extraAnchors and remove this setting.

Core reads a change to this setting, on or off, when you rerun helm upgrade. App Garden extensions such as Kaizen receive it only when they are deployed, so a running extension keeps the old value until you redeploy it. After a trial, turning the setting off is not complete until you redeploy every extension that calls an external model. Plain http:// endpoints have no TLS and are not affected by this setting or by tlsInsecure.

Object Storage​

If users will upload files, preserve workroom context, or use the Skills Library, object storage must be available before rollout.

Default self-managed installs use namespace-local SeaweedFS for workroom object storage; no object-store configuration is needed. Its volume uses the block storage class you name at install; see Block storage in your environment's prerequisites: Kubernetes with Istio or OpenShift with GreyMatter. When workroom content must live in external AWS S3, use:

Best practice:

  • set bucket, region, and secret references in your Helm values file, under core.context.objectStorage
  • prefer secret references or ambient cloud identity over inline credentials

Logging and Observability​

Logging and telemetry configuration should be treated as platform configuration, not per-user customization.

Use:

Best practice:

  • verify the UI log viewer works for deployment troubleshooting
  • configure any OpenTelemetry or external log forwarding through your platform release values
  • validate log access and retention as part of rollout readiness

CORS and Browser Origins​

Kamiwaza supports additional browser origins through serving configuration, including KAMIWAZA_CORS_ORIGINS.

Best practice:

  • keep browser origins specific
  • do not use wildcard CORS unless you are in a controlled development environment
  • make sure the configured allowed origins match the actual HTTPS origins users and extensions will access

If credentials are in use, broad or mismatched CORS settings are a common source of extension and browser failures.

Routing Configuration​

Routing is now managed as a runtime control-plane setting, backed by the shared runtime store rather than per-pod file edits.

The routing API is:

  • GET /api/config/routing
  • PATCH /api/config/routing

Current best practice is to treat path-based routing as the standard customer-facing mode.

Canonical runtime prefixes are:

  • models: /runtime/models/<deployment-id>
  • apps: /runtime/apps/<deployment-id>
  • tools: /runtime/tools/<deployment-id>

The Kubernetes charts already seed path-based routing defaults. Use the routing API only when you intentionally need to adjust the base host or service prefixes after deployment.

Extensions reach the platform in-cluster through KAMIWAZA_PLATFORM_GATEWAY_URL. On Istio, the chart sets KAMIWAZA_INGRESS_GATEWAY_SERVICE_HOST on Core from global.ingress.gateway.serviceHostname; override it in core.scheduler.extraEnv only for a gateway Service the chart does not describe. Extensions run by the extension operator get the same gateway through extensionOperator.istioGatewayEgress, on by default; see Extension operator on Istio. On a cluster whose DNS domain is not cluster.local, set global.clusterDomain. See Platform Gateway for Extensions.

For more detail, see:

Where To Change Settings​

Use this table as a quick guide:

Setting AreaBest Place to Change It
Customer hostname / ingress domainHelm values
Auth and IdP integrationHelm values + admin workflow
Secrets and credentialsKubernetes Secrets or external secret manager
Object storageHelm values + secret references
Observability endpoints and log forwardingHelm values
Runtime routing prefixes / base hostRouting API

Anti-Patterns to Avoid​

  • editing source-repo install scripts as a customer configuration mechanism
  • treating old package-install env.sh flows as the primary deployment model
  • storing cloud credentials directly in user-editable metadata
  • documenting port-specific runtime URLs as the preferred public access pattern

Validation Checklist​

After a configuration change, validate:

  • the platform is reachable at the expected HTTPS origin
  • administrator and standard-user sign-in still work
  • app and tool launches resolve under the expected runtime paths
  • storage-backed workflows still function
  • logs remain visible to the right administrators

When in doubt, use the Quickstart as the post-change validation path.