Skip to main content

Administrator Guide

Use this guide to understand the security-related responsibilities of a Kamiwaza administrator in a customer deployment. It focuses on the current Kubernetes-based platform and avoids internal helper scripts, source-repo workflows, and host-local bootstrap procedures.

Authentication Model​

Kamiwaza 1.3 runs with authentication enabled. It uses an external or bundled identity provider, validates tokens, and enforces policy. Running Kamiwaza with authentication turned off, sometimes called Lite mode, is not supported in 1.3.

For customer environments, use a customer-facing HTTPS domain, managed ingress, and centralized identity.

Identity Provider Responsibilities​

Administrators are responsible for confirming that the identity provider integration is correct for the deployment.

At a minimum, verify:

  • the deployment domain is correct
  • redirect and callback URLs match the deployed hostname
  • the audience or client ID expected by Kamiwaza is configured correctly
  • issuer and JWKS metadata are reachable by the platform
  • user claims required by policy are present

For more detail on tenant-aware authorization, see:

User and Role Management​

User lifecycle management should be handled through the deployment's identity provider and normal administrative workflow.

Best practice:

  • create administrator and standard-user accounts through the identity provider or supported admin workflow
  • avoid relying on node-local or package-local CLI tools as the default customer user-management path
  • review group and role mappings as part of onboarding
  • disable or remove temporary test identities before production rollout
  • once individual administrators exist, disable the shared bootstrap administrator; for stig installs, see Retire the bootstrap administrator

Kamiwaza commonly distinguishes between:

Role TypeTypical Purpose
administratorPlatform setup, access policy, troubleshooting, and environment validation
standard userDaily use of models, apps, tools, and workrooms
service identityAutomation, extensions, or system-to-system integration

Secrets Management​

Treat authentication, signing, and integration credentials as deployment-managed secrets.

Best practice:

  • store secrets in Kubernetes Secrets or your approved external secret manager
  • keep long-lived secrets out of plain-text values files whenever possible
  • rotate administrative and client secrets through your normal release workflow
  • avoid documenting or depending on file-system secret fallbacks inside running containers

This applies to credentials such as:

  • identity provider client secrets
  • administrative passwords
  • signing keys and shared secrets
  • object storage credentials

Kamiwaza can enforce a pre-login consent gate and show classification banners for the deployment.

Use these controls when your environment requires:

  • an explicit user acknowledgment before access
  • persistent classification or handling caveats across the UI
  • consistent banner behavior for embedded experiences

For configuration details, see:

Content Security Policy​

The console and the Kamiwaza API send an enforced Content-Security-Policy header in both security profiles (default and stig). The browser refuses anything the policy does not allow, which limits what an injected script or a malicious page fragment could do.

  • The console allows scripts and styles only from its own origin, and fonts from its own origin or data: URLs. Inline styles are allowed, inline scripts are not. Connections and images may also use the console's own host name on any HTTPS port, which port-routed model deployments use; images may also come from data: and blob: URLs. Plugins are refused, and the console cannot be framed.
  • API responses carry default-src 'none'; frame-ancestors 'none', which lets them load nothing and refuses framing. This is always enforced.
  • The API docs pages (/api/docs, /api/redoc) carry a policy that allows only their own bundled scripts.
  • App Garden extensions under /runtime/ send their own policy. On Istio and standalone Envoy installs, and on GreyMatter installs when the chart renders its GSL bundle, the platform adds a baseline policy when an extension sends none, and a floor policy (object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'self') beside the extension's own when it sends one. The Helm value global.extensionRuntimeCsp.mode sets whether these are enforced; when they are, a policy the extension sends only as Content-Security-Policy-Report-Only is enforced too. See Extension runtime policy under stig.

What this means for content you configure:

  • The News widget reads its feed through Kamiwaza, so the Kamiwaza API pods (not each browser) need network access to the feed host set in global.newsApiUrl. Feed images are shown as links. When Kamiwaza cannot reach the feed, the widget keeps showing the last feed it loaded, with a notice, and Kamiwaza tries the feed again at most once a minute. The Refresh button fetches the feed again at most once a minute per Kamiwaza API pod. The X-Kamiwaza-News-Status header on GET /api/news/quadrants says whether an answer is live, stale (an older copy), fallback (the placeholder) or disabled (no feed is configured).
  • An App Garden template whose preview image is an external URL shows no preview image. Use an images/ path or upload the image with the template.
  • Consent banner HTML cannot load external images or run inline event handlers.

Browsers report each refused resource to /api/security/public/csp-report, and Kamiwaza logs them as CSP violation, rate-limited per user or client.

To diagnose a problem, an operator can switch a component to report-only mode, where the browser reports violations but blocks nothing. Set core.security.csp.mode: report-only or frontend.security.csp.mode: report-only, together with security.csp.acknowledgeReportOnly: true on the same component. In the default profile, the same applies to a KAMIWAZA_CSP_MODE entry in the component's extraEnv: any value other than enforce, including a valueFrom or a $(VAR) reference, needs the acknowledgment, because the install cannot check what it resolves to. In the stig profile, KAMIWAZA_CSP_MODE cannot be set in extraEnv at all; use security.csp.mode with the acknowledgment. Without the acknowledgment the install refuses the setting and names the setting it refused. An existing release that already runs report-only this way fails helm upgrade until it adds the acknowledgment. The older key security.csp.acknowledgeReportOnlyUnderStig still works until 1.5.0. With it, the install output warns that STIG V-204760 is not met, and the component logs the same warning when it starts. Switch back to enforcement once the problem is fixed.

Extension runtime policy under stig​

global.extensionRuntimeCsp.mode sets the mode of the policy the platform applies to App Garden extension responses under /runtime/. When it is unset, it follows the security profile:

  • default profile: report-only. Set enforce to enforce it.
  • stig profile (global.securityProfile: stig): enforced, wherever the platform applies the policy (below). A stig install that left the mode unset starts enforcing when it upgrades.

The policy is applied by the extension pods' Istio sidecars, by the standalone Envoy edge (global.ingress.provider: envoy), and by the GreyMatter edge when the chart renders its GSL bundle. The standalone Envoy edge enforces the policy under stig in the same way as the Istio sidecars. On a stig install where none of these applies the policy, the install output warns V-204760 not met for App Garden extensions instead of claiming the control. A GreyMatter install that publishes the GSL bundle from a separate render must render it with global.securityProfile: stig too. Extension pods in Istio ambient mode have no sidecar, so on an Istio install they get no platform policy, and the chart cannot detect that: record such extensions against V-204760 yourself. The mode reads global.securityProfile only: a Core-only core.security.profile: stig does not change it.

To run the extension policy report-only under stig, for example while you diagnose a problem, set both:

global:
  extensionRuntimeCsp:
    mode: report-only
    acknowledgeReportOnly: true

Without the acknowledgment the install refuses the setting and names it. The acknowledgment must be a boolean (true, not "true"). With it, the install output warns V-204760 not met, and STIG checks report V-204760 as failed. Remove both settings once the problem is fixed.

Before you install or upgrade with stig, upgrade Kaizen to the Kaizen release in the 1.4 App Garden catalog, or later, which supports an enforced extension policy. Earlier Kaizen releases cannot show artifact previews under it. Check your other extensions too: sample their /runtime/ response headers. A policy an extension sends only as Content-Security-Policy-Report-Only will be enforced as written, including directives browsers ignore in report-only mode (sandbox, upgrade-insecure-requests). The floor applies even to an extension that sends its own policy, so an extension that is framed by another origin, or whose forms post or redirect to another origin, is blocked. See Extension runtime policy under stig in the upgrade guide.

Relationship-Based Access Control​

ReBAC adds tenant-aware and relationship-aware authorization on top of authentication.

Administrators should treat ReBAC as part of deployment configuration, not as an ad hoc runtime toggle.

Recommended responsibilities:

  • confirm ReBAC is enabled only in auth-enabled environments
  • validate tenant and relationship data before rollout
  • verify representative allow and deny flows during release testing
  • preserve evidence of policy and access validation for accreditation or audit needs

Use these docs together:

Personal Access Tokens and Service Access​

If your deployment uses automation, extensions, or system integrations, review how those workflows authenticate.

Best practice:

  • issue only the minimum access required for the integration
  • prefer managed service identities and approved token workflows
  • rotate and revoke credentials when an integration changes ownership or scope
  • keep tenant scoping aligned with the intended workload

Logging, Audit, and Troubleshooting​

Security troubleshooting should use the environment's approved observability path.

That may include:

  • the Kamiwaza UI log viewer
  • Kubernetes-native logging
  • OpenTelemetry or external logging systems configured for the deployment

Administrators should be able to verify:

  • successful authentication events
  • denied access events
  • session expiration or revocation behavior
  • enough request context to support incident response or accreditation evidence

Audit coverage​

  • Matched API routes emit kamiwaza.audit.api_action for authenticated POST, PUT, PATCH and DELETE requests and privileged requests, including reads by verified platform administrators and admin-gated routes. Records identify the method, route, endpoint target and HTTP outcome. Each service also emits records for user and role management, permission changes, personal access token creation and revocation, and access to, change, disposal or export of sensitive data. The Observability Guide lists the event names and what is audited today.
  • A denied-request audit middleware records 401 and 403 responses so failed access attempts leave a trail even when the downstream handler is never invoked. Core-owned SAML, certificate and Lite sign-ins emit catalog authentication outcomes, and DDE classification enforcement emits document-denial records.
  • Audit records are surfaced through the same observability path as other security logs, and are also written to each service's stdout.
  • In Kamiwaza 1.4.0, catalog audit records emitted in authenticated bearer, personal access token, and signed platform-gateway request contexts carry the acting user (user.id). Token creation and self-revocation records identify the acting owner. Administrative PAT revocation identifies the administrator in user.id, the affected owner in enduser.id and the token identifier in kamiwaza.audit.target.id. Unverified credentials do not establish a user; native sign-in failures can identify a previously resolved account.
  • Credential issuance and known session ends have their own outcomes and correlation fields. Session expiry is recorded when Core observes it, so an unobserved expiry has no guaranteed end record. Core-owned session identifiers are keyed with a stable per-installation fingerprint key of at least 32 bytes. The Core chart creates and keeps that key (Secret kamiwaza-audit-fingerprint) and mounts it into every Core audit producer, unless the site supplies its own key through Core extraEnv; under stig a key is required. A diagnostic reports a missing key. See What is audited today for Secret injection, session attribution and identifier limits.

Coverage limits. Successful non-admin reads are not generically audited; explicitly instrumented sensitive-data reads are covered. An API action record describes the HTTP outcome: 202 means accepted, not that asynchronous work completed. Interrupted response delivery is recorded as a failure even when success headers were sent. Request bodies, query strings, response contents and document contents are excluded from API-action records; service-specific records describe individual object changes. Calls that authenticate their own non-platform credential, including delegated-workload capability, effect and credential-use calls, are outside the generic API-action hook and rely on service-specific events.

Sensitive metadata and volume. Endpoint target identifiers can contain usernames, file paths and URNs derived from filenames. Treat audit storage as containing personal and sensitive metadata even when bodies are excluded. Every verified global administrator's request is audited, including UI polling and admin-role automation. A view polling every three to five seconds produces roughly 720–1,200 records per hour while open; concurrent views and administrators increase that load.

Retention and storage. Audit records are written to container stdout and exported over OTLP when configured. Collect them into your log backend or SIEM, restrict access and set retention to match your accreditation requirements. The UI log viewer serves deployment logs rather than a complete audit archive. The recent-decisions API does not provide durable storage.

Authorization decision export​

GET /api/auth/audit/decisions/export returns recent ReBAC authorization decisions to administrators through the standard /api entry point. It accepts tenant, result, updated_after, limit (default 200, up to 1,000) and cursor query parameters.

It is a diagnostic view, not an audit store:

  • each API replica keeps only its own most recent decisions in memory (2,000 by default)
  • the history is lost when the replica restarts
  • a request reaches one replica, so with more than one replica the result is incomplete

For durable coverage, collect both catalog kamiwaza.audit.authorization.decision events and legacy audit records marked event_category: "audit". The ReBAC decision buffer writes legacy rebac_decision records; it does not turn every buffered decision into a catalog event. See How audit events arrive.

Authenticated-by-default API surface​

The platform now applies the AuthenticatedUser dependency to all non-exempt endpoints, including admin and destructive routes. The effective contract:

  • Every API endpoint requires an authenticated caller unless explicitly exempt (health, login, public metadata).
  • Admin and destructive endpoints additionally require the appropriate role or scope.
  • The OpenAPI spec now documents auth requirements per endpoint — use it as the source of truth when integrating. The spec is served by the platform at /api/openapi.json (with an interactive browser at /api/docs) on the deployment's own hostname.

Public paths at the edge. The Helm list global.authz.extAuthz.exemptPaths names the paths that skip the external authorization check. It covers what a signed-out browser needs before login (public security configuration, identity provider discovery, sign-in and token refresh, the consent banner, and the CSP report sink) and a few machine-to-machine paths that check their own credentials. The Kamiwaza API's authorization check, the Istio gateway and the standalone Envoy edge all read it. It is a subset of the paths the Kamiwaza API serves without a session: SAML login (/api/auth/saml/metadata, /api/auth/saml/login, /api/auth/saml/acs, /api/auth/saml/sls) and certificate login (/api/auth/certificate/login) are not in the shipped list. Add those exact paths to the list, and run helm upgrade, before you turn on SAML or certificate login; otherwise the identity provider's POST to /api/auth/saml/acs is denied. Helm replaces a list value rather than merging it, so your values must carry the whole shipped list plus the new paths: copy global.authz.extAuthz.exemptPaths from helm show values for your chart version, then append them. A list with only the new paths removes every shipped exemption, and signed-out login stops working on every install type.

global:
  authz:
    extAuthz:
      exemptPaths:
        # ...every entry of the shipped list, unchanged...
        - /api/auth/saml/metadata
        - /api/auth/saml/login
        - /api/auth/saml/acs
        - /api/auth/saml/sls
        - /api/auth/certificate/login

An entry ending in / is a prefix; any other entry is exact. /api/auth/ is exact, so /api/auth/check, /api/auth/users/... and /api/auth/pats still require a session.

From 1.4, the standalone Envoy edge (global.ingress.provider: envoy) sends each /api/... entry of that list straight to the Kamiwaza API without its external authorization check, ahead of the protected /api route. Before, every signed-out login call on that edge went through the check, so login depended on it. No path becomes newly public: the edge skips the check for exactly the listed paths, which the authorization check already admitted without credentials. Every other /api path still goes through the check. Notes for Envoy-edge installs:

  • The edge reads the list when the chart renders. To change it, run helm upgrade; restarting the Kamiwaza API is not enough.
  • An /api/... entry the edge cannot serve fails the render with the reason: an entry with *, ?, #, %, ,, spaces, or empty or dot segments; /api or /api/ itself; the same path with and without its trailing slash; or two paths that map to one route. The shipped list passes. Istio and GreyMatter installs are unchanged.
  • Before any check, the edge removes the internal identity and admission headers a client could forge, now including X-User-MCP-Admission. A client cannot send one through the edge to the Kamiwaza API, on public paths or protected ones.

Cluster trust and machine-to-machine federation​

For multi-cluster and federated deployments, identity headers from peer services are only honored when signed by a trusted cluster identity. Unsigned requester-identity headers are now distrusted and dropped. Operators federating two Kamiwaza clusters must exchange cluster trust material as part of deployment; contact Kamiwaza Support for the per-environment procedure.

Rotation, revocation, and compromise. Cluster trust material is long-lived but not permanent. Rotation, revocation on decommission, and response to suspected key compromise are deployment-specific procedures and are not exposed as a self-service UI in the platform. If you need to rotate or revoke a peer cluster's trust material, or if you suspect a cluster identity has been compromised, contact Kamiwaza Support for the per-environment procedure — do not attempt to edit signing material in running pods. Until a compromised identity is revoked through the supported path, isolate the affected cluster at the network layer.

Local user password policy​

By default, every password a user or administrator sets must meet the platform password policy. The policy is the same in both security profiles (default and stig):

  • Length: at least 15 characters.
  • Character classes: at least one uppercase letter, one lowercase letter, one digit and one special character. Any character that is not a letter or a digit counts as special, including a space.
  • Not the username: the password must not be the username.

Where the policy is enforced depends on where the password is stored:

  • Installs with Keycloak (the full platform): the Keycloak realm password policy applies. The deploy chart sets it on install and upgrade through auth.keycloak.realm.passwordPolicy in the kamiwaza chart's values (keycloak.realm.passwordPolicy if you install the auth chart on its own). It also keeps a history of the last 5 passwords and stores them as PBKDF2-SHA512 with 210,000 iterations. A password set on 1.3 was stored with argon2, and Keycloak rehashes it with the policy's algorithm when its user signs in after the realm's policy names PBKDF2. With FIPS enabled (the default), the upgrade to 1.4.0 stops while any argon2 password remains (see Move local users' passwords off argon2). You can tune these values, and in the default profile turn the managed policy off. At install and on every upgrade, the stig profile refuses turning the managed policy off, a shorter length, fewer character classes, less history, a non-PBKDF2 algorithm or fewer iterations. It does not stop you from turning off the not-the-username rule; leave it on (the chart preset enables it). Whatever the realm allows is what the platform allows.
  • Lite, SAML and certificate installs: user passwords are stored by the platform itself, and the platform applies the policy above when you create a user, reset a user's password, or a user changes their own password. These installs do not keep password history. Passwords are stored as PBKDF2-HMAC-SHA256 with a per-user random salt and 200,000 iterations.

In the console, the password fields for adding a user, resetting a password and changing your own password show this policy as a hint. In redirect login mode (always on under stig), users change their own password on the Keycloak account page instead, and Keycloak shows its own policy error there. Because a site can tune its realm policy, the hint does not block you from submitting; the server decides. A password that does not meet the policy is refused with HTTP 400. On Lite, SAML and certificate installs the error names every rule the password failed, for example Password does not meet the password policy: it must be at least 15 characters long. With Keycloak the error is Keycloak's own and names the first rule it failed, for example Invalid password: minimum length 15. The error never repeats the password. No usable account or password is saved when a password is refused. Existing passwords keep working; the policy applies the next time a password is set.

The bootstrap administrator password (KAMIWAZA_ADMIN_PASSWORD) is install configuration rather than a password set through the platform, so it keeps its own startup check: at least 12 characters (the community edition also accepts its documented default). Choose one that meets the policy above; the startup check only enforces 12 characters.

The local user store does not lock an account after repeated failed sign-ins. If you need lockout or password history for these users, run the full platform with Keycloak, whose realm is created with brute-force protection on and keeps a 5-password history. For lockout alone you can also rate-limit sign-in at your ingress. Forced password expiry is not configured by the chart on either kind of install.

For broader deployment logging guidance, see:

Ephemeral Sessions for App Garden​

Some deployments use ephemeral app sessions so launched applications are automatically cleaned up when the user logs out or the session expires.

Use this behavior when you want:

  • reduced persistence for temporary app launches
  • automatic cleanup for demos or evaluation environments
  • stronger alignment between user session state and app lifetime

If your deployment enforces ephemeral extension behavior globally, document that choice as part of rollout readiness so administrators and users understand the lifecycle impact.

Administrator Validation Checklist​

Before handing the environment to users, confirm:

  • users can sign in at the expected HTTPS hostname
  • role mappings behave as expected
  • required banners or consent messaging appear correctly
  • protected actions allow the right users and deny the wrong ones
  • logs and audit evidence are visible through the approved operations path
  • secrets and callback URLs are managed through the deployment workflow, not manual pod edits

Next Steps​