Hardened-profile install and secure configuration guide

Use this guide to install Kamiwaza 1.4 with the stig security profile and to configure it securely. It walks through each step in order: prepare the host and cluster, prepare identity, write the values file, install, configure, and verify. Each step gives the command or values to use and the result you should see.

The Security profiles and STIG alignment page explains what the stig profile is and how it differs from default. This guide is the procedure. Your assessor can use it, with the verification results, as evidence of a documented secure configuration.

What this guide covers, and what it doesn't

Benchmark. The stig profile's controls are aligned with the DISA Application Server Security Requirements Guide (SRG) V4R5.

Scope: the application layer. Kamiwaza's scope is the application it ships and operates. You bring the hardened host operating system, the Kubernetes or OpenShift cluster, and, if you use one, the external database host. Those layers are yours: harden them to their own STIGs and record them as inherited controls in your Customer Responsibility Matrix (CRM). This guide lists what Kamiwaza expects from them, so the application-layer controls hold, but it doesn't make any claim about them.

Kamiwaza is qualifying the stig profile on a RHEL 9 host with the DISA STIG profile applied and FIPS mode on. That qualification shows compatibility only; it is not a claim about your host.

Claim wording. When you describe this configuration in your system security plan, use the same wording Kamiwaza uses:

• Say: "a FIPS 140-3-enforced configuration using NIST-validated modules within a declared boundary".
• Don't say "FIPS-validated product" or "STIG-certified". FIPS 140-3 validation applies to cryptographic modules, not to products, and only your assessor and authorizing official (AO) decide whether your system meets its requirements.

Who this is for. Operators installing Kamiwaza for a deployment that will be assessed against DISA STIGs or SRGs, for example for an ATO or IATT. For every other install, use the default profile and the standard installation guides.

Before you start

You need:

Item	Requirement
Kamiwaza release	1.4.0 or later, from the published OCI chart
Helm	4.2 or a later 4.x release. The supported install path requires it; the chart itself doesn't check the client version.
kubectl	Matching your cluster's minor version
Cluster	An existing Kubernetes or OpenShift cluster, hardened by you (Step 2)
Hosts	RHEL 9 with FIPS mode and the DISA STIG profile (Step 1), or the equivalent your AO accepts
Identity	A Keycloak-backed full install. Your enterprise identity provider (IdP) for CAC/PIV, brokered through Keycloak (Step 3).
PKI	A server certificate for the platform hostname from your approved PKI, and the trust anchors for your IdP
SIEM	An OTLP collector that feeds your SIEM, reachable from the cluster over TLS (Step 4.8)
Storage	A StorageClass that encrypts volumes at rest (Step 2)
License	A valid Kamiwaza license

Keep one site values file, for example stig-values.yaml, and pass it on every install and every upgrade. The profile and every setting in this guide live in that file.

In the commands and values in this guide, replace every value in angle brackets, such as <encrypted-class>, with your site's value before you run or apply it. The shell commands use variables such as $VERSION and $HOST; set them first.

Step 1: Prepare the hosts (your responsibility)

The host operating system is outside Kamiwaza's boundary. Harden it to the RHEL 9 STIG yourself. Kamiwaza depends on the four host properties below. Check each one on every node, control plane and worker, including GPU nodes.

1.1 Turn on FIPS mode

Install RHEL 9 in FIPS mode where you can. Red Hat recommends enabling FIPS mode at install time, so that no keys are generated before FIPS mode is on. To install in FIPS mode, add fips=1 to the kernel command line at installation.

If you must enable FIPS mode on an installed host, first make sure you can still sign in after the reboot. In FIPS mode, sshd refuses Ed25519 user keys and accepts only RSA and ECDSA keys. An administrator who signs in only with an Ed25519 key is locked out, and agent-based cloud recovery, such as Azure Run Command and VMAccess, also fails on a FIPS host (Step 1.2). Give every administrator an RSA (3072 bits or more) or ECDSA public key in ~/.ssh/authorized_keys (or in your directory service, if sshd reads keys from it). Keep that plain key authorized even if you sign in with SSH certificates: FIPS mode also refuses Ed25519 certificates and certificates an Ed25519 CA signed, and the check below proves only the plain key. Prove the key works on its own, over a fresh connection with public key authentication only:

# On your workstation. ECDSA shown; for RSA use KEY=~/.ssh/id_rsa_fips and -t rsa -b 3072.
KEY=~/.ssh/id_ecdsa_fips
[ -f "$KEY" ] || ssh-keygen -t ecdsa -b 384 -f "$KEY"
ssh-copy-id -i "$KEY.pub" <admin-user>@<host>     # add -p <port> if sshd is not on 22
# Must print "key OK". It ignores your ssh config, agent, any open shared connection and
# any certificate next to the key ($KEY-cert.pub), so only the plain key can sign in. It
# never falls back to a password (it may ask for the key's passphrase). Use the host's
# real address (add -p <port> if sshd is not on 22), since your ssh config is not read:
ssh -F /dev/null -S none -o IdentityAgent=none -o IdentitiesOnly=yes -o CertificateFile=/dev/null -i "$KEY" \
  -o PreferredAuthentications=publickey -o PasswordAuthentication=no \
  -o KbdInteractiveAuthentication=no <admin-user>@<host> echo key OK

The key name is not one ssh offers by default, so add it (your $KEY path) to ~/.ssh/config, under the exact name or address you type for the host. Otherwise, after the reboot, ssh without -i still offers only your Ed25519 key, and with several agent keys it can hit sshd's MaxAuthTries before it reaches this one:

Host <host>
  IdentityFile ~/.ssh/id_ecdsa_fips
  IdentitiesOnly yes

Put this block above any Host * block. IdentitiesOnly yes keeps agent keys out, but IdentityFile lines from every matching block add up, so a Host * entry for an Ed25519 key is still tried. ssh -G <host> | grep -i identityfile lists the keys ssh will offer. RSA keys sign with SHA-2 (rsa-sha2-256 or rsa-sha2-512), the default for current OpenSSH clients; FIPS mode refuses SHA-1 ssh-rsa signatures from older tools.

sshd itself also needs a host key that FIPS mode can load. Check which ones it uses; the list must include an RSA or ECDSA key file that exists:

sudo sshd -T | grep -i '^hostkey'

If it lists only ssh_host_ed25519_key, add HostKey /etc/ssh/ssh_host_ecdsa_key (create the key with sudo ssh-keygen -q -t ecdsa -b 384 -N '' -f /etc/ssh/ssh_host_ecdsa_key if it is missing) and restart sshd before you go on; you replace that key after the reboot. Have console access ready in case a step fails.

Then run:

sudo fips-mode-setup --enable
sudo reboot

Then regenerate keys created before FIPS mode was on, such as the SSH host keys, and record the deviation for your assessor. Keep your current session open until a new one works: new connections use the new host keys as soon as they are written. Generate the new host keys yourself and check them before the restart, rather than relying on sshd-keygen, which cloud-init images can disable:

sudo rm -f /etc/ssh/ssh_host_*_key /etc/ssh/ssh_host_*_key.pub   # keeps any host certificates
sudo ssh-keygen -q -t ecdsa -b 384 -N '' -f /etc/ssh/ssh_host_ecdsa_key
sudo ssh-keygen -q -t rsa -b 3072 -N '' -f /etc/ssh/ssh_host_rsa_key
sudo restorecon -v /etc/ssh/ssh_host_*
# SSH host CA sites: sign the two new .pub files and install the certificates now
sudo sshd -t && sudo systemctl restart sshd && systemctl is-active sshd

sshd -t must print nothing and is-active must print active; if not, fix it from the session you still have open. If sshd -t names ssh_host_ed25519_key, remove that HostKey line from /etc/ssh/sshd_config or /etc/ssh/sshd_config.d/: FIPS mode cannot load it. Host CA sites: check a fresh certificate-trusting connection too before you close the session. After that restart, the host's Ed25519 host key is no longer offered and its other host keys are new. A client that knew the old key refuses to connect (REMOTE HOST IDENTIFICATION HAS CHANGED), and one that didn't asks you to confirm an unknown key. In the session you kept open (or on the host's console), print every host key fingerprint:

for k in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$k"; done

On your workstation, remove the old entry with ssh-keygen -R <host> (for sshd on another port: ssh-keygen -R '[<host>]:<port>'), connect again, and type yes only if the fingerprint ssh shows matches the console line for the same key type. Never turn off StrictHostKeyChecking to get past the warning.

Verify:

cat /proc/sys/crypto/fips_enabled
grep -o 'fips=1' /proc/cmdline
fips-mode-setup --check
update-crypto-policies --show

Expected result: 1, fips=1, FIPS mode is enabled., and FIPS.

Kamiwaza doesn't check FIPS mode on your hosts when it installs with Helm. Keep the output of these commands as evidence.

If you install the cluster with KKS under stig, KKS checks the node it runs on before it installs anything, then checks the cluster's nodes again at the handoff, and refuses the run unless each node reports 1. KKS reads FIPS mode only on the node it runs on: every other node the cluster lists is recorded as not checked, and under stig that refuses the handoff, so use KKS under stig for single-node clusters only. An unreadable value counts as not confirmed. The preflight refusal names the file, the value it found and the remedy (at the handoff, the check lists every node that isn't confirmed):

global.securityProfile=stig requires kernel FIPS mode, but
/proc/sys/crypto/fips_enabled on this node is 0, not 1. RHEL 9 STIG
RHEL-09-671010 requires the operating system to run in FIPS mode. Enable FIPS
mode on the host (RHEL 9: sudo fips-mode-setup --enable, then reboot), confirm
cat /proc/sys/crypto/fips_enabled prints 1, and retry. k0s and the platform
were not installed, and any earlier hand-off values file was removed. To build
without host FIPS mode, use global.securityProfile: default, which records and
warns instead.

1.2 Apply the DISA STIG profile

Apply the RHEL 9 STIG with the SCAP Security Guide content. Use the scap-security-guide version that matches the benchmark your assessor names.

Before you apply it, keep a way to become root. The STIG removes every NOPASSWD sudo grant, including the ones cloud images create for their first user (for example /etc/sudoers.d/waagent on Azure). An administrator without a password and wheel membership loses root for good when the playbook runs. Give your administration account a password and add it to wheel (or give it an equivalent sudoers entry that asks for a password) first, then check it from a new session:

sudo passwd <admin-user>
sudo usermod -aG wheel <admin-user>
# In a new SSH session as <admin-user>:
sudo -k; sudo -v      # must ask for the password and succeed

Don't count on your cloud's out-of-band recovery. On Azure, for example, Run Command and the VMAccess extension fail on a FIPS-mode host, so only the serial console still works.

sudo dnf install -y scap-security-guide openscap-scanner ansible-core
sudo ansible-playbook -i localhost, -c local \
  /usr/share/scap-security-guide/ansible/rhel9-playbook-stig.yml
sudo reboot

Scan the host before you install Kamiwaza and keep the results. You scan again after the install (Step 7) and explain every difference.

sudo oscap xccdf eval \
  --profile xccdf_org.ssgproject.content_profile_stig \
  --results-arf host-pre-arf.xml --report host-pre-report.html \
  /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml

Expected result: an ARF file and an HTML report. Disposition every failed rule before you continue.

Some host STIG settings interact with Kubernetes and container runtimes. Check these with your cluster vendor before you scan, and record any exception in your CRM:

• fapolicyd in deny-by-default mode can block cluster binaries and container shims. It also blocks client tools you install later, such as Helm (Step 5.1) and etcdctl (Step 2.1): add each one to its trust, for example sudo fapolicyd-cli --file add /usr/local/bin/helm then sudo fapolicyd-cli --update. It also stops interpreters from reading untrusted script files, even under your home directory: add a helper script to the trust the same way before you run it.
• Under sudo, secure_path leaves out /usr/local/bin on RHEL 9. Call tools installed there by their full path under sudo, for example sudo /usr/local/bin/etcdctl.
• noexec on /tmp, /var/tmp and /dev/shm. Kamiwaza's distributed runtime (Ray) uses /dev/shm for its object store.
• SELinux must stay enforcing. Your cluster distribution must support it.
• firewalld deny-by-default must still allow your CNI's traffic.
• A separate /var partition must be large enough for container images and model files.

1.3 Synchronize time

Audit records and session timeouts rely on the node clock. Point every node at an authoritative time source.

sudo systemctl enable --now chronyd
chronyc sources
chronyc tracking

Expected result: chronyc sources lists at least one server marked * (selected), and chronyc tracking shows Leap status : Normal.

1.4 Prepare GPU nodes

On GPU nodes, install the NVIDIA driver from Red Hat's precompiled kernel module streams where possible, so the host needs no compiler. If UEFI Secure Boot is on, the kernel modules must be signed. nvidia-container-toolkit must run with SELinux enforcing. See GPU nodes.

Step 2: Prepare the cluster (your responsibility)

The Kubernetes or OpenShift cluster is outside Kamiwaza's boundary. Harden it to the Kubernetes STIG or the platform STIG your assessor names, and record it as inherited. Kamiwaza depends on the cluster properties below.

2.1 Encrypt Kubernetes secrets at rest (etcd)

Kamiwaza stores credentials as Kubernetes Secrets. Configure the API server to encrypt Secrets in etcd, preferably with a KMS v2 provider backed by your key management service. On an upstream API server this is an EncryptionConfiguration passed with --encryption-provider-config:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:
          apiVersion: v2
          name: site-kms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - identity: {}

On OpenShift, set spec.encryption.type on the APIServer resource instead. Follow your distribution's documentation; the provider and endpoint are site-specific.

On k0s, which KKS installs, the API server runs as the kube-apiserver user, so a root-only file stops it from starting (open /etc/k0s/encryption.yaml: permission denied). Add the setting under the existing spec.api.extraArgs in /etc/k0s/k0s.yaml (don't replace the file; KKS wrote the rest), give that user read access, restart, then rewrite the existing Secrets so they are stored encrypted:

# /etc/k0s/k0s.yaml
spec:
  api:
    extraArgs:
      encryption-provider-config: /etc/k0s/encryption.yaml

sudo chown kube-apiserver /etc/k0s/encryption.yaml
sudo chmod 0400 /etc/k0s/encryption.yaml
sudo systemctl restart k0scontroller
for i in $(seq 1 60); do sudo /usr/local/bin/k0s kubectl get --raw /readyz >/dev/null 2>&1 && break; sleep 5; done
sudo /usr/local/bin/k0s kubectl get --raw /readyz; echo   # prints ok
sudo /usr/local/bin/k0s kubectl get secrets -A -o json | sudo /usr/local/bin/k0s kubectl replace -f -

With a KMS provider, the kube-apiserver user also needs access to the KMS plugin's socket. On k0s, replace the certificate flags in the verify command below with --cacert=/var/lib/k0s/pki/etcd/ca.crt --cert=/var/lib/k0s/pki/apiserver-etcd-client.crt --key=/var/lib/k0s/pki/apiserver-etcd-client.key. KKS keeps your settings in /etc/k0s/k0s.yaml when it runs again. Still, check that encryption-provider-config is set after any KKS run, because a k0s restart without it can't read the Secrets already stored encrypted. etcdctl doesn't ship with k0s: install the etcdctl from the etcd release that matches your cluster's etcd, and add it to the fapolicyd trust (Step 1.2).

Verify: after you create a test Secret, read it directly from etcd. The value must start with k8s:enc:kms:v2: and not show the plaintext.

kubectl create secret generic enc-check -n default --from-literal=probe=plaintext-probe
# On a control-plane node; certificate paths shown are kubeadm's, use your distribution's.
# Under the STIG, call etcdctl by its full path (Step 1.2), wherever you installed it.
sudo /usr/local/bin/etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/enc-check | hexdump -C | head
kubectl delete secret enc-check -n default

On OpenShift, check the Encrypted condition on the openshiftapiserver and kubeapiserver resources instead.

Expected result: the stored value begins with k8s:enc:kms:v2:site-kms (or k8s:enc:aescbc:v1: if your AO accepts a local key), and plaintext-probe does not appear.

2.2 Provide an encrypted StorageClass

Every Kamiwaza persistent volume, including the PostgreSQL volumes and the pre-upgrade snapshot volume, must sit on storage that is encrypted at rest. Use a StorageClass whose backing storage encrypts volumes, such as an encrypted cloud volume type, LUKS-backed local storage, or an encrypted storage array.

kubectl get storageclass
kubectl get storageclass <encrypted-class> -o yaml

Expected result: the class you will use shows its encryption parameter (for example encrypted: "true" on AWS EBS), or your storage vendor's documentation says it encrypts every volume. Record which it is. Step 4 asks you to name the class and attest to it.

2.3 Expose only HTTPS

Only the HTTPS entry point (TCP 443) may be reachable from outside the cluster; under stig, port 80 must be closed, with no HTTP redirect. Also allow SSH (TCP 22) to the hosts for administration if your site allows it. The Kubernetes API (for example TCP 6443) should be reachable only from your administration network. Internal services, such as the Ray dashboard, metrics endpoints and database ports, must not be reachable from outside.

Whether anything listens on port 80 depends on who owns the ingress gateway:

• The chart owns the gateway (Istio ingress with the default VirtualService route, and global.ca.enabled: true; the chart's default is false, so this case applies only where your values set it): under stig, the chart's Istio Gateway has only the HTTPS server on 443, with no port 80 server and no redirect. (Under default, it also renders an HTTP server on port 80, plus a VirtualService that answers it with a 301 redirect to HTTPS.)
• KKS or your platform team owns the gateway (global.ca.enabled: false, the chart default, with observedTls recorded): the chart renders no Gateway, so port 80 exists only if the gateway owner configures it; if it does, the chart's 301 redirect still attaches to it. Under stig, KKS's gateway has no port 80 server; see Step 2.5 for the NodePorts it still exposes.
• The standalone Envoy edge (global.ingress.provider: envoy): the edge listens for HTTPS only.

In every case, block port 80 at your firewall and load balancer and confirm it with the scan in Step 7.3.

Set this with your host firewall and your network boundary, not only with Kubernetes objects. After the install, Step 7 checks it with a port scan.

2.4 Apply pod security

Kamiwaza workloads run as non-root. Some components don't meet Pod Security restricted yet, so the environment prerequisites (Step 2.6) enforce baseline on the kamiwaza and kamiwaza-extensions namespaces through an admission policy with a narrow exception for Istio's init container, and label both namespaces to warn and audit at restricted. Don't add a Pod Security enforce label to those namespaces, restricted or baseline: Istio's init container needs network capabilities that either label rejects, so every injected pod would fail. If your cluster STIG requires more, record the difference in your CRM.

2.5 If you use Kamiwaza Kubernetes Setup (KKS)

STIG deployments normally install onto the site's own hardened cluster, and KKS is not the supported route for them. Kamiwaza uses KKS for its own qualification install. If you use KKS anyway, for example in a lab, put the profile and the TLS policy in a shared site-security.yaml and pass it to KKS and, later, to Helm:

global:
  securityProfile: stig
  ingress:
    tls:
      minProtocolVersion: TLSV1_2

ansible-playbook ansible/playbooks/local-k0s-podman.yml -e @site-security.yaml

Expected result:

• KKS refuses an edge TLS floor below TLS 1.2 under stig.
• Its handoff reads the live gateway and writes global.ingress.tls and global.ingress.gateway.observedTls for the Helm install (Step 4.5). It stops if the live gateway's TLS settings differ from the global.ingress.tls you passed.
• A run that reaches the handoff records each node's FIPS mode in ~/.kamiwaza/evidence/<cluster>/fips-host-mode.yaml. Under stig it refuses the run on any node not confirmed in FIPS mode (Step 1.1), and refuses global.fips.enabled: false. A refused run leaves no handoff file behind.
• With FIPS requested, the k0s control plane, etcd and kubelet set a TLS 1.2 minimum and allow only FIPS-approved TLS 1.2 suites. TLS 1.3 stays negotiable on them, including TLS_CHACHA20_POLY1305_SHA256, and the konnectivity server (TCP 8132) and the k0s join API (TCP 9443) keep Go's default TLS suites; record these in your CRM. Istio runs with its fips-140-2 compliance policy, which restricts its TLS settings; it is not a FIPS 140-3 claim. The stock Istio proxy is not itself a validated module; record it in your CRM or POA&M. The other cluster components KKS runs aren't FIPS images either: k0s's CoreDNS, kube-proxy, kube-router, metrics-server, konnectivity agent, pause and CNI node images, the local-path provisioner and its helper (busybox) image, and the k0s binary itself, which is the stock upstream build. List them in your CRM as cluster-layer components.
• KKS doesn't configure time synchronization, etcd encryption or an encrypted StorageClass. Do Steps 1.3, 2.1 and 2.2 yourself.
• Under stig, KKS's gateway has no port 80 server and its host bridge for port 80 is stopped. In Kamiwaza's qualification, its Istio NodePorts were still reachable from outside: 31443 (the HTTPS edge, with the same certificate and TLS policy) and 31021 (the Istio status port, plain HTTP). The cluster's NodePort forwarding bypasses firewalld. Block them at your network boundary, and confirm with the scan in Step 7.3.

The handoff does not carry securityProfile into the Helm values; set it in stig-values.yaml (Step 4.1). Leave global.ca.enabled at its default, false, on the KKS route: KKS applies its own ingress gateway, and true would make the chart render a second one with the same name.

2.6 Complete your environment's prerequisites

Complete the Prerequisites page for your environment (see Environments) before Step 4, because Step 4 creates Secrets and a volume claim in the kamiwaza namespace. For Kubernetes with Istio, that page has you:

• create the kamiwaza namespace, labeled for Istio sidecar injection;
• create the namespace-scoped installer identity you run Helm and kubectl as (never cluster-admin);
• install the pod security admission guard for the kamiwaza and kamiwaza-extensions namespaces;
• set up the ingress gateway, the edge certificate from your PKI, and DNS.

Expected result: kubectl get namespace kamiwaza --show-labels shows the namespace with its injection label, and every command from Step 4 on runs as the installer identity.

Step 3: Prepare identity

3.1 Use a Keycloak-backed full install

The stig profile requires the bundled Keycloak. The password policy, session limits, MFA and inactive-account controls are Keycloak and Core controls, so installs without Keycloak can't meet them. Under stig, Kamiwaza Lite, direct-SAML and certificate-only installs refuse to start and name the reason.

3.2 Decide how users sign in

Under stig, every user signs in through the Keycloak browser flow. The Kamiwaza UI redirects there, and Core refuses the password grant with a named reason. MFA is required for every person.

Choose one of these for each population of users:

Users	How they sign in	MFA
Your workforce, with CAC/PIV	Brokered from your enterprise IdP into Keycloak over OpenID Connect, as a chart-managed broker (Step 6.2). Your IdP performs the CAC/PIV authentication.	Your IdP asserts MFA in a claim, by default amr containing mfa. A brokered sign-in without it is refused.
Local accounts, such as break-glass administrators	Keycloak username and password	TOTP (authenticator app), enrolled at first sign-in
The Core service account (svc-core)	A confidential client, kamiwaza-service-grant, whose secret only Core holds	Exempt as a non-person entity. Every use is audited.

Collect from your IdP team before you install:

• the IdP's OIDC issuer URL, and a client registered there for Kamiwaza with the redirect URI https://<platform-host>/realms/kamiwaza/broker/<alias>/endpoint, its client ID and its client secret;
• the claim that carries the MFA assertion and the values that mean MFA, for example amr containing mfa, or acr matching an AAL2 or AAL3 value;
• the CA that issued the IdP's TLS certificate, if it isn't publicly trusted;
• the claim (for example groups) and values that map users to Kamiwaza roles.

Under stig, only chart-managed OIDC brokers are allowed, because only they carry the MFA claim check. Kamiwaza refuses to add or enable a broker any other way, and the realm Job fails the release if the realm has an enabled broker the chart doesn't manage. SAML brokers and Core's built-in SAML sign-in aren't supported under stig; if your IdP offers only SAML, connect it through an OpenID Connect interface.

3.3 Name your break-glass accounts

Keep at least one local administrator account for when your IdP is unavailable. Break-glass accounts use a password and TOTP like every local account, are exempt from the inactive-account disable only if you list them (Step 4), and are used only through an audited procedure. Write the list down now; Step 4 puts it in the values file.

Step 4: Write the stig values file

Create stig-values.yaml with the blocks in this step, and add your site's usual values (hostname, certificates, license, model settings) from your environment's install page. Leave every per-component security value unset (~) unless this step tells you to set it: unset values follow the profile.

4.1 Select the profile

global:
  securityProfile: stig
auth:
  keycloak:
    realm:
      securityProfile: stig

global.securityProfile is the switch. It sets every per-component security setting, and it turns on the install-wide checks: edge TLS, encrypted storage, the pre-upgrade backup and Dashboards sign-in. The realm key follows it when unset; setting it to stig as well is harmless, and setting it to default under a stig umbrella is refused.

Under stig, the chart refuses to render an override that weakens a control it checks. Helm's error contains the setting, the rule and the fix, for example:

global.securityProfile=stig refuses core.auth.inactivityDisable.days=90 (rule: V-204750 (SRG-APP-000163, disable accounts after 35 days of inactivity)). Fix: set core.auth.inactivityDisable.days between 1 and 35.

The chart doesn't refuse every weakening. Review these values in your files yourself; the list is the ones that matter most, not every value the chart accepts:

• Report-only CSP with an acknowledgment (Step 4.10). The render succeeds with a WARNING: V-204760 not met: line, and the control is Open.
• MFA exceptions. Extra non-person service users (core.auth.login.serviceUsers with acknowledgeNonPersonServiceUsers) can use the password grant without MFA. core.auth.pat.mfaAmrValues and a broker's mfaClaim pattern decide what counts as MFA; a broad value, such as a pattern of .*, accepts any sign-in. List each exception in your SSP.

The chart also refuses, under stig, global.fips.enabled: false, auth.enabled: false, auth.keycloak.realm.sessionLimit.maxSessionsPerUser above 3, global.identity.exchangedTokenTtlSeconds above 3600 (or set through auth.keycloak.initUsers.extraEnv), core.security.auditFingerprint.enabled: false, a literal KAMIWAZA_AUDIT_FINGERPRINT_KEY value in Core extraEnv, and any setting that turns TLS certificate verification off, such as KAMIWAZA_ALLOW_INSECURE_INTERNAL_RETRY=true in core.scheduler.extraEnv. With a self-signed edge certificate, the KKS handoff adds that variable; use an edge certificate from your PKI, so KKS leaves it out, or delete it from the handoff file, because global.trust.edgeAnchor already gives the platform the CA.

It also refuses extraEnv entries that redefine a chart-managed security variable, such as KAMIWAZA_CSP_MODE, KAMIWAZA_SECURITY_PROFILE or AUTH_LOGIN_MODE. Set the chart value instead.

4.2 Keep the FIPS image set

global:
  fips:
    enabled: true   # the 1.4.0 default; shown here so the intent is recorded

With global.fips.enabled: true, the chart selects the FIPS variants of the platform images, including keycloak-fips with the Bouncy Castle FIPS provider, postgres-fips and model-fetch-fips, and doesn't accept the non-FIPS Argon2 password hash: an explicit argon2 setting is refused. On an upgrade from a non-FIPS Keycloak, a pre-upgrade Job moves the chart's own accounts to PBKDF2, but it can't rehash other local users' Argon2 passwords and stops the upgrade while any remain (see Step 8).

Most third-party FIPS images, such as keycloak-fips, postgres-fips and etcd-fips, are pinned by digest in the chart; a few, such as spicedb-fips, follow the release tag. Kamiwaza's own FIPS images, such as core-fips and frontend-fips, follow the release's image channel tag (release-<version> in a packaged release chart) instead of a digest. Step 7.2 lists every image the install runs so you can check them against the release's image list. Images that stay non-FIPS by design, such as the stock edge proxy, and FIPS builds without -fips in their name, such as the bundled session store image kaizen-valkey, belong in your CRM or POA&M.

4.3 Declare encrypted storage

global:
  storage:
    stateful:
      standard: <encrypted-class>
storage:
  encryptedAtRest: true
  encryptedStorageClasses:
    - <encrypted-class>
datahub:
  enabled: false

Every Kamiwaza PersistentVolumeClaim must name an explicit class. global.storage.stateful.standard sets it for every role; set postgres, etcd, registry, extensions, sessionStore or fast under global.storage.stateful only if they use a different encrypted class, and list each such class. A claim that falls back to the cluster's default StorageClass resolves to an empty class name, which is never accepted. On the KKS route, the handoff file sets standard, postgres, etcd, registry and extensions to the KKS cluster's class, so set every one of those keys to your encrypted class in stig-values.yaml, not only standard. Don't add "" to the list to get past the check: that attests a class nobody checked. If you set core.sessionStore.persistence.storageClassName, list that class too.

This is an attestation, not a setting that turns encryption on. You declare that the classes you list encrypt at rest (Step 2.2). Under stig, the chart refuses to render unless encryptedAtRest is true and every class that a Kamiwaza PersistentVolumeClaim uses is in the list: PostgreSQL, the object store, Core, extensions, the frontend, the upgrade backup and the session store. The bundled DataHub is refused under stig; it is outside the declared boundary.

4.4 Configure the pre-upgrade database backup

Create a PersistentVolumeClaim on your encrypted class for the Core database snapshot. It must be separate from the database's own volume.

kubectl -n kamiwaza apply -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: core-upgrade-backups
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: <encrypted-class>
  resources:
    requests:
      storage: 50Gi
YAML

core:
  postgresql:
    upgradeBackup:
      existingClaim: core-upgrade-backups

Size the claim for at least two snapshots of your Core database. Under stig, the backup is required, and an upgrade fails before it changes anything if the claim is missing, Pending with a warning, or Lost.

4.5 Set the edge TLS policy

global:
  ingress:
    tls:
      minProtocolVersion: TLSV1_2
      cipherSuites:
        - ECDHE-ECDSA-AES128-GCM-SHA256
        - ECDHE-RSA-AES128-GCM-SHA256
        - ECDHE-ECDSA-AES256-GCM-SHA384
        - ECDHE-RSA-AES256-GCM-SHA384

These are the defaults; write them down so the file records the policy. On an edge the chart renders, TLS 1.3 stays enabled. On the KKS route, the gateway runs Istio's fips-140-2 compliance policy, which allows only TLS 1.2, so it refuses TLS 1.3 handshakes. Under stig, a floor below TLS 1.2 is refused, and the Gateway API ingress path is refused because its listener TLS options can't be verified. Use the Istio VirtualService route or the Envoy edge.

If the chart does not render the edge listener itself, because KKS, your platform team or GreyMatter owns the gateway, record the gateway's actual settings. They must match global.ingress.tls exactly, including the cipher order:

global:
  ingress:
    gateway:
      observedTls:
        minProtocolVersion: TLSV1_2
        cipherSuites:
          - ECDHE-ECDSA-AES128-GCM-SHA256
          - ECDHE-RSA-AES128-GCM-SHA256
          - ECDHE-ECDSA-AES256-GCM-SHA384
          - ECDHE-RSA-AES256-GCM-SHA384

The KKS handoff writes this block for you when KKS owns the gateway, in ~/.kamiwaza/evidence/<cluster>/substrate-values.yaml; pass that file to Helm before stig-values.yaml (Step 5.2). Under stig the chart refuses to render when observedTls is missing or differs from global.ingress.tls.

On the standalone Envoy edge, the paths that skip the authorization check before sign-in come from global.authz.extAuthz.exemptPaths, the same list the Istio gateway uses. Leave that list as shipped under stig; see the Administrator Guide for what it covers.

4.6 Keep audit integrity on

The stig profile seals the audit events of the Core processes that receive the key (core-api, the scheduler, the Ray head and workers, and the retrieval streamer) in an HMAC-SHA256 hash chain, and Core refuses to start without the key. By default the chart creates the key on first install, as the Secret kamiwaza-audit-integrity with key hmac-key, and keeps it across upgrades and uninstall. Nothing needs to be set.

If your key management policy requires that you generate the key, create the Secret before installing and tell the chart not to:

(umask 077; openssl rand -hex 32 > ./hmac-key)
kubectl -n kamiwaza create secret generic kamiwaza-audit-integrity --from-file=hmac-key=./hmac-key
shred -u ./hmac-key

The key goes through a file with owner-only permissions, so it never appears in a command line or your shell history. Keep your escrow copy in your key management system, not on the workstation.

core:
  security:
    auditIntegrity:
      createSecret: false

Expected result: the install fails with a named message if the Secret is missing, lacks hmac-key, or holds fewer than 32 bytes. Back up the key with your other platform secrets; you need it to verify audit records later. Back up the audit fingerprint key the same way (Secret kamiwaza-audit-fingerprint, key key, which the chart creates on first install; or the Secret named by core.security.auditFingerprint.existingSecret; or, if you set the key through Core extraEnv, the Secret your entries name): a new fingerprint key breaks joins between new and earlier audit records.

Record two known exceptions for V-204739 in your CRM:

• Install jobs, route reconcilers and the user-jobs Ray cluster emit audit events without the hash chain.
• Code running in Core's Ray workers, such as model and tool deployments, can read the key.

4.7 Configure the logout session store

Under stig, Core keeps a server-side list of signed-out sessions so that a signed-out token stops working at once. Record two residual risks against the logout requirement: the store writes to disk once a second, so a crash can lose about the last second of sign-outs; and losing the store's volume (data-core-session-store-0) lets signed-out tokens work again until they expire, up to about 24 hours. The chart deploys a TLS-only session store (core-session-store, Valkey FIPS) by default. Nothing needs to be set unless you bring your own certificates or your own Redis:

• With core.certs.mode: byo, set core.certs.existingSecrets.sessionStore to a server certificate for the store. The render fails without it.
• To use your own store, set core.sessionStore.mode: external and put a complete rediss:// URL, under the key url, in the Secret named by core.sessionStore.external.existingSecret.

Under stig, core.sessionStore.mode: none is refused, and Core answers 503 while the store is unreachable instead of accepting signed-out tokens.

4.8 Export audit events to your SIEM

Kamiwaza exports logs and audit events over OTLP. Core keeps no durable audit store of its own: your SIEM is the record of truth.

Create Secrets for the collector's credentials and CA, then point the platform at your collector. Put the header value (for example authorization=Bearer%20<token>, with no trailing newline) in a file with owner-only permissions through your secret-management workflow, rather than typing the token on a command line, where it would land in process listings and shell history:

kubectl -n kamiwaza create secret generic customer-otlp-headers \
  --from-file=headers=./otlp-headers
shred -u ./otlp-headers
kubectl -n kamiwaza create secret generic customer-otlp-ca \
  --from-file=ca.crt=./collector-ca.pem

global:
  observability:
    endpoint: https://<collector-host>:4318
    protocol: http/protobuf
    headersSecretName: customer-otlp-headers
    caSecretName: customer-otlp-ca

Export needs only the endpoint. Under stig the chart doesn't refuse an install without one, so check that global.observability.endpoint is set: without it, no audit events leave the cluster. Don't set global.observability.enabled: true unless you also deploy the optional observability release (OpenSearch Dashboards): it makes the install register a Dashboards sign-in client and write into the kamiwaza-observability namespace.

Route audit events in your SIEM on the otel.event.name attribute (and on the legacy event_category: audit). Events from Core's audit streams carry the kamiwaza.audit.integrity.* attributes; the unsealed emitters listed in Step 4.6 don't. Don't turn off certificate verification. See Observability for the event catalog and protocol options.

If you deploy the optional OpenSearch Dashboards in the separate observability release, set global.securityProfile: stig in that release's values as well. Otherwise its sign-in requirement isn't enforced there. Under stig, that release also refuses to render unless global.fips.enabled is true, and its FIPS OpenSearch images need a FIPS keystore, a BCFKS truststore and Secret-backed passwords that the chart doesn't create; its install notes list them.

4.9 List break-glass accounts

core:
  auth:
    inactivityDisable:
      exemptAccounts:
        - <break-glass-username>

Kamiwaza disables local accounts with no sign-in, token refresh or personal access token use for 35 days, revokes their tokens and sessions, and audits each disable. Under stig, the period must be 1 to 35 days and the control can't be turned off. Accounts from your directory or brokered IdP follow that system's lifecycle and aren't swept. List only the break-glass accounts you named in Step 3.3, never the shared bootstrap administrator (Step 6.8).

4.10 Content Security Policy

The Core API and the web console send an enforced Content Security Policy in every profile, default included. Nothing needs to be set.

You can switch one component to report-only only with an explicit acknowledgment. The render then prints a warning that begins WARNING: V-204760 not met: and names the component, and the control shows as Open in your checklist:

frontend:
  security:
    csp:
      mode: report-only
      acknowledgeReportOnly: true

For Core, report-only affects only the API documentation pages; its API responses stay enforced. Use this only to diagnose a breakage, then remove it. (acknowledgeReportOnlyUnderStig is the deprecated spelling of the same key, accepted until 1.5.0.)

Extensions. Under stig, the chart also enforces the extension runtime policy on extension responses under /runtime/ (global.extensionRuntimeCsp.mode follows the profile), on extension pods with Istio sidecars, on the standalone Envoy edge, and on GreyMatter when global.greymatter.renderGslBundle is true. Pods in Istio ambient mode get no extension policy; the render NOTES warn when no edge applies it. Before you install or upgrade with stig, upgrade Kaizen to the Kaizen release in the 1.4 App Garden catalog, or later, and check your other extensions' /runtime/ response headers; see Extension runtime policy under stig. Report-only for extensions needs the same kind of acknowledgment, global.extensionRuntimeCsp.acknowledgeReportOnly: true, and leaves V-204760 Open.

When the built-in documentation pages are enabled, they send a report-only policy (frontend.docsCsp.mode) in both profiles; they don't follow stig. Set frontend.docsCsp.mode: enforce if your assessment covers them.

4.11 Connector endpoint allowlist

Connectors reach only public addresses by default. An endpoint that resolves to a private or other non-public address is refused at registration and on every call, unless you list it site-wide or an administrator approves it for one connector. Loopback, link-local and cloud metadata addresses are refused even when listed. To let connectors reach your internal data sources, list them:

core:
  connectors:
    endpointAllowlist:
      - connector.internal.example.com   # a hostname
      - .data.internal.example.com       # a domain suffix
      - 10.20.0.0/16                     # a CIDR

Entries have no scheme and no port. The render refuses a malformed entry. Under stig, the render also refuses core.connectors.unsafeDevAllowLoopbackAndMetadata, and Core refuses to start with it set by any route; never use it on an assessed install. See Connector destinations for per-connector approvals, how existing connectors behave on upgrade, and the admin view of blocked destinations.

4.12 Complete example

This example renders on its own. Besides the stig blocks from this step, a render needs the values every install needs; on the KKS route, several of them are in the handoff file you pass before stig-values.yaml (Step 5.2):

Value	Where it comes from
global.eula.accepted	You, after reading the EULA. The render stops without it.
core.license.existingSecret	The Secret holding your license.lic (see License File). The render stops without it.
core.inferenceResources enabled, ownerPublicKey, clusterBindingId, bundleConfigMap	The cluster owner, before the install: see Tenant-mode inference. ownerPublicKey is the raw 32-byte Ed25519 key in base64 (44 characters), not a PEM file. The render stops with missing cluster owner trust without the key and binding ID; keep enabled: true.
global.domain	Your platform hostname. KKS route: in the handoff.
global.trust.edgeAnchor	The CA chain that issued your edge certificate from your PKI (pem), or publiclyTrusted: true. Set it here on every route: the KKS handoff writes it only for a self-signed edge certificate, which stig sites shouldn't use (Step 4.1).
core.extensionRuntime.kubernetesApi.egressCidrs	Every Kubernetes API Service and endpoint address, each as /32 (IPv4) or /128 (IPv6); see your environment's install page. KKS route: in the handoff.
global.ingress.gateway.observedTls	The gateway's TLS settings (Step 4.5). KKS route: in the handoff.
global.storage.stateful.*	Your encrypted StorageClass for every role (Step 4.3). KKS route: the handoff sets the KKS cluster's class, so set every key here to override it.

On the KKS route, leave the other handoff values out of stig-values.yaml rather than copying them: the handoff file is passed first, and a value here overrides it. The storage classes are the exception: override them here with your encrypted class (Step 4.3). The example marks each value to delete on the KKS route. Copying the handoff's observedTls here in particular would replace the live gateway reading with fixed text.

global:
  securityProfile: stig
  domain: <platform-host>                    # KKS route: in the handoff; delete this line here
  eula:
    accepted: true                            # you, after reading the EULA
  fips:
    enabled: true
  trust:
    edgeAnchor:                             # the CA chain that issued your edge certificate
      pem: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
  ingress:
    tls:
      minProtocolVersion: TLSV1_2
      cipherSuites:
        - ECDHE-ECDSA-AES128-GCM-SHA256
        - ECDHE-RSA-AES128-GCM-SHA256
        - ECDHE-ECDSA-AES256-GCM-SHA384
        - ECDHE-RSA-AES256-GCM-SHA384
    gateway:
      observedTls:                          # KKS route: in the handoff; delete this block here
        minProtocolVersion: TLSV1_2
        cipherSuites:
          - ECDHE-ECDSA-AES128-GCM-SHA256
          - ECDHE-RSA-AES128-GCM-SHA256
          - ECDHE-ECDSA-AES256-GCM-SHA384
          - ECDHE-RSA-AES256-GCM-SHA384
  observability:
    endpoint: https://<collector-host>:4318
    protocol: http/protobuf
    headersSecretName: customer-otlp-headers
    caSecretName: customer-otlp-ca
  storage:
    stateful:
      standard: <encrypted-class>
      postgres: <encrypted-class>
      etcd: <encrypted-class>
      registry: <encrypted-class>
      extensions: <encrypted-class>
auth:
  keycloak:
    realm:
      securityProfile: stig
storage:
  encryptedAtRest: true
  encryptedStorageClasses:
    - <encrypted-class>
datahub:
  enabled: false
core:
  license:
    existingSecret: kamiwaza-license
  inferenceResources:
    enabled: true
    bundleConfigMap: <profile-configmap>   # from the cluster owner
    ownerPublicKey: <OWNER_PUBLIC_KEY>
    clusterBindingId: <CLUSTER_BINDING_ID>
  extensionRuntime:
    kubernetesApi:
      egressCidrs:
        - <api-service-ip>/32          # KKS route: in the handoff; delete this block here
        - <api-endpoint-ip>/32         # /128 for an IPv6 address
  postgresql:
    upgradeBackup:
      existingClaim: core-upgrade-backups
  auth:
    inactivityDisable:
      exemptAccounts:
        - <break-glass-username>
  security:
    consent:
      enabled: true   # DoD notice and consent banner; set its text in Step 6.7

Step 5: Install

5.1 Check the client

Set the release version you are installing, and check that the chart for it is published:

VERSION=1.4.0
helm version --short
helm show chart oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza --version "$VERSION" | grep -E '^(version|appVersion):'

The last command prints version: and appVersion: equal to $VERSION. An error naming a missing tag or manifest means that version isn't published; use the version from your release notice. Network or authentication errors mean the registry isn't reachable from here; without internet access, use your mirrored chart's reference instead.

Expected result: v4.2.0 or a later 4.x release. The install and upgrade commands in this guide need it: --rollback-on-failure exists only in Helm 4 (Helm 3 called it --atomic). The chart itself doesn't check the client version.

Kamiwaza Kubernetes Setup (KKS) installs Helm 3.19 on the host when it finds no Helm 3.14 or later. If helm version --short shows v3.x, install Helm 4.2 or later before you continue, checking the download against its published checksum:

HELM_VERSION=v4.2.0   # or a later 4.x release
ARCH=$(uname -m | sed 's/x86_64/amd64/; s/aarch64/arm64/')
PKG="helm-${HELM_VERSION}-linux-${ARCH}.tar.gz"
curl -fsSLO "https://get.helm.sh/${PKG}"
curl -fsSLO "https://get.helm.sh/${PKG}.sha256sum"
sha256sum -c "${PKG}.sha256sum"
tar -xzf "${PKG}"
[ -x /usr/local/bin/helm ] && sudo cp -p /usr/local/bin/helm /usr/local/bin/helm3   # keep KKS's Helm 3
sudo install -m 0755 "linux-${ARCH}/helm" /usr/local/bin/helm
sudo fapolicyd-cli --file add /usr/local/bin/helm && sudo fapolicyd-cli --update   # fapolicyd trust (Step 1.2)
hash -r; helm version --short

The checksum proves the download is intact, not who published it, because it comes from the same site. If your site also requires the publisher's signature, download the matching .asc file from the release's GitHub page (https://github.com/helm/helm/releases/download/${HELM_VERSION}/${PKG}.asc) and check it with gpg --verify against the Helm project's published signing keys. Without internet access, copy the files in through your transfer process.

Install Helm 4 after KKS finishes. A later KKS run keeps the Helm it finds, including Helm 4, but KKS itself is qualified only with Helm 3: if you run KKS again, use the Helm 3 client the commands above kept as /usr/local/bin/helm3 for that run (for example, link it as helm in a directory placed first on your PATH, and add it to the fapolicyd trust). Check helm version --short before every Kamiwaza install or upgrade (Step 8).

5.2 Render first

Render against the cluster before you install, so the chart's refusals show up before anything changes. Some checks happen only when Core starts (Step 5.3). First set the values files once. If KKS or your platform team owns the gateway, include the handoff file that records observedTls (Step 4.5), before stig-values.yaml:

VALUES=(--values kamiwaza-values.yaml --values stig-values.yaml)
# KKS route: VALUES=(--values kamiwaza-values.yaml --values ~/.kamiwaza/evidence/<cluster>/substrate-values.yaml --values stig-values.yaml)

kamiwaza-values.yaml is your environment's values file from the install page; stig-values.yaml is the file from Step 4. Pass the same files, in this order, on every install and upgrade.

( set -o pipefail
helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version "$VERSION" \
  --namespace kamiwaza \
  "${VALUES[@]}" \
  --dry-run=server | sed -n '/^NOTES:/,$p'
echo "render exit code: $?" )

Expected result: render exit code: 0 and the release NOTES, which state the stig summary and list every override and warning (for example any WARNING: V-204760 not met: line). Read them before you install. A refusal names the setting, the rule and the fix. Correct the values file and render again.

Under stig, the render also refuses auth.enabled: false, because the profile needs Keycloak (Step 3.1).

5.3 Install

helm upgrade --install kamiwaza \
  oci://ghcr.io/kamiwaza-ai/releases/kamiwaza/charts/kamiwaza \
  --version "$VERSION" \
  --namespace kamiwaza \
  "${VALUES[@]}" \
  --rollback-on-failure \
  --wait --wait-for-jobs \
  --timeout 70m

Don't add --create-namespace; the namespace belongs to the platform administrator. Install with Helm against the cluster, not from rendered manifests: the chart generates and later reads back the platform's keys, including the audit integrity key.

Expected result: the release reaches deployed, and the NOTES state the profile:

helm -n kamiwaza status kamiwaza
kubectl -n kamiwaza get pods

The NOTES list the stig summary and name every per-component override. All pods reach Running or Completed.

Watch the install from a second terminal while Helm waits. If a first install fails, --rollback-on-failure uninstalls the release when the wait ends, and the pods and their logs go with it. If core-api or core-scheduler doesn't become Ready, Core refused to start and the pod log names the reason:

kubectl -n kamiwaza get pods -w
kubectl -n kamiwaza logs deploy/core-api --tail=50

Common causes under stig:

Log message begins	Cause	Fix
AUTH_SECURITY_PROFILE=stig requires the inactive-account disable ..., then but it cannot run: <reason> or but it failed to start (<error>)	The inactive-account sweep can't run. The reason names why: no Keycloak, Lite, SAML or certificate sign-in turned on directly in Core, or the sweep turned off. failed to start means the sweep raised an error at start-up	Fix what the reason names. For the identity reasons, use a Keycloak-backed full install (Step 3.1)
KAMIWAZA_SECURITY_PROFILE=stig requires an audit-log HMAC key ...	The audit integrity Secret is missing	Create it, or set createSecret: true (Step 4.6)
AUTH_SECURITY_PROFILE=stig cannot use AUTH_REBAC_SESSION_REDIS_URL ...	The session store isn't TLS Redis	Use the bundled store or a rediss:// URL (Step 4.7)

Step 6: Configure after the install

6.1 Enroll the first administrator in MFA

Sign in at https://<platform-host>/ with the bootstrap administrator. The console redirects to Keycloak. Keycloak asks for a one-time code and, the first time, shows a QR code to enroll an authenticator app.

Expected result: after the code, you land in the Kamiwaza console. Every local account enrolls the same way at its first sign-in.

Enroll the authenticator on a device other than the workstation you sign in from, such as a mobile phone, and make that a written site policy for every local account. The bootstrap administrator is a shared account: use it only to set up individual administrators, then retire it (Step 6.8).

Under stig, Keycloak sessions end after 15 minutes idle or 8 hours total, and each account can hold at most 3 interactive sessions, and a new sign-in over the limit is refused rather than ending an older session. Personal access tokens and service-account tokens aren't counted.

This covers local accounts and users brokered from your IdP over OIDC, including CAC/PIV users who sign in through your IdP (Step 6.2). (Core's own SAML sign-in is refused under stig. Core's built-in certificate sign-in is off by default and this guide doesn't enable it; if you enable it, its sessions are Core's, counted separately from Keycloak's and held until the Core token expires, 60 minutes by default.) Keycloak shows "There are too many sessions" (Core shows its own message for brokered users) until the user signs out of another session or one idles out. A browser closed without signing out holds its session for up to 15 minutes. To end sessions sooner, use the Keycloak admin CLI (Step 6.5); the Keycloak admin console isn't routed under stig. kc create users/<id>/logout -r kamiwaza ends all of one user's sessions, and kc delete sessions/<session id> -r kamiwaza ends one. kc get users -r kamiwaza -q username=<name> -q exact=true --fields id gives the id, and kc get users/<id>/sessions -r kamiwaza lists the sessions.

The console's access token lasts at most 1 hour (global.identity.exchangedTokenTtlSeconds, 3600 by default under stig; the chart refuses more). auth.keycloak.realm.sessionLimit.maxSessionsPerUser can be 1, 2 or 3; the chart refuses a higher value (Step 4.1). Users change their password on the Keycloak account page. The email password reset is off while MFA is required, because it would sign the user in without the code; administrators reset passwords with Keycloak's admin CLI (Step 6.5).

6.2 Broker your enterprise IdP

Configure your IdP in the values file, not in the Keycloak admin console. A broker added in the console has no MFA claim check, and under stig it fails the next upgrade.

Create the client secret, and the CA if you need one, in the release namespace. Write the client secret to a file with owner-only permissions through your secret-management workflow; never pass it on a command line:

kubectl -n kamiwaza create secret generic customer-idp --from-file=client-secret=./idp-client-secret
shred -u ./idp-client-secret
kubectl -n kamiwaza create configmap customer-idp-ca --from-file=ca.crt=./idp-ca.pem

Add the broker to stig-values.yaml:

auth:
  keycloak:
    identityProviders:
      - alias: agency-sso
        displayName: Agency SSO
        issuer: https://sso.agency.example/realms/agency
        clientId: kamiwaza
        clientSecretRef:
          name: customer-idp
          key: client-secret
        caBundleRef:
          kind: ConfigMap
          name: customer-idp-ca
          key: ca.crt
        mfaClaim:
          claim: acr
          valuePattern: "http://idmanagement\\.gov/ns/assurance/aal/[23].*"
        roleMappings:
          - {claim: groups, value: /kamiwaza-admins, role: admin}
          - {claim: groups, value: /kamiwaza-users, role: user}

Leave out mfaClaim if your IdP sends amr containing mfa; that is the default check. The pattern is a Java regular expression that must match one claim value in full. A user who matches no role mapping can't sign in through the broker.

Apply it with the Step 5.3 command. The realm Job checks the IdP's discovery and TLS before it changes the broker.

Expected result: a CAC/PIV user who chooses your IdP on the Keycloak sign-in page reaches the console without a TOTP prompt, and their token carries amr: ["mfa"]. A user whose IdP sign-in didn't show MFA is refused with a message naming the claim. See the chart's identity-provider documentation for every option.

6.3 Appoint the audit administrator

Audit configuration is separated from platform administration. Assign the audit-admin realm role in Keycloak to a person who does not hold admin. Under stig, an account that holds both is refused on audit routes, and Kamiwaza itself can't grant or revoke audit-admin: only a Keycloak realm administrator appointed by your ISSM can.

Expected result: the audit administrator can read GET /api/security/audit/config; a platform administrator gets 403. See Audit administration for the procedure and SSP wording.

6.4 Use personal access tokens for automation

Under stig, Core refuses the password grant, so the SDK and CLI use personal access tokens (PATs). A user can mint a PAT only from a session that signed in with MFA: by default, a session whose amr claim contains otp (a local account's code) or mfa (a broker that passed the MFA check), set by core.auth.pat.mfaAmrValues. PATs and service tokens can't mint further PATs.

Each PAT has a scope: Read, Write or Admin. A PAT minted without one gets Write (Read for viewers), even for an administrator, and a Write PAT can't call administration routes such as the users API. For the administration steps in this guide (6.7 to 6.9), choose Admin (full access) in the console's token form, or send "scope": "admin" when you mint it. Give each token the shortest lifetime your process allows.

The sign-in that enrolls a new authenticator doesn't count as an MFA session, so sign out and sign in again with the code before you mint a PAT.

Expected result: minting from an MFA session succeeds; minting otherwise is refused with 403 pat_mfa_required.

6.5 Set up the break-glass procedure

When your IdP is unavailable, a listed break-glass account can use the password grant for a bounded window. Turn it on only for the incident, then remove it. Work out the end of the window first, at most 72 hours from now:

date -u -d '+24 hours' +%Y-%m-%dT%H:%M:%SZ      # Linux (GNU date)
date -u -v+24H +%Y-%m-%dT%H:%M:%SZ               # macOS (BSD date)

core:
  auth:
    login:
      passwordGrantBreakGlass: true
      breakGlassUsers:
        - <break-glass-username>
      breakGlassUntil: "<time from the date command>"   # RFC 3339; past or more than 72 hours ahead is refused

Apply with the same helm upgrade --install command as Step 5.3.

Expected result: the listed account can sign in with password and TOTP until the time given; any other account gets an ordinary wrong-credentials answer. Every use writes a WARNING audit event. After the window, remove the block and upgrade again.

Realm administration in an emergency, for example to clear a lost one-time code or re-enable an account, uses Keycloak's admin CLI inside the Keycloak pod. The Keycloak admin console isn't routed through the platform edge and can't be reached through a port-forward either, so use the CLI. Sign in as the master realm's admin, passing the password from the keycloak-admin Secret on standard input rather than the command line:

KC_POD=$(kubectl -n kamiwaza get pod -l app.kubernetes.io/name=keycloak -o jsonpath='{.items[0].metadata.name}')
kubectl -n kamiwaza get secret keycloak-admin -o jsonpath='{.data.password}' | base64 -d \
  | kubectl -n kamiwaza exec -i "$KC_POD" -c keycloak -- sh -c \
    'KC_CLI_PASSWORD=$(cat) /opt/keycloak/bin/kcadm.sh config credentials --server http://localhost:8080 --realm master --user admin --config /tmp/kcadm.config'
kubectl -n kamiwaza exec -it "$KC_POD" -c keycloak -- sh

In that shell, run /opt/keycloak/bin/kcadm.sh <command> -r kamiwaza --config /tmp/kcadm.config for each change, and record each use in your incident log. For example, to set a temporary password that the user must change at next sign-in:

stty -echo; printf 'Temporary password: '; read PW; stty echo; echo
/opt/keycloak/bin/kcadm.sh set-password -r kamiwaza --username <user> --new-password "$PW" --temporary --config /tmp/kcadm.config
unset PW

The password is read without echo inside the Keycloak container, so it stays out of your terminal and shell history; kcadm.sh requires it as an argument, so it is visible to that container's process list only while the command runs. It must meet the realm's password policy (Step 7.5). Hand it to the user through your usual out-of-band channel; Keycloak makes them change it at the next sign-in.

When you finish, remove the CLI's saved admin session from the pod:

kubectl -n kamiwaza exec "$KC_POD" -c keycloak -- rm -f /tmp/kcadm.config

6.6 Confirm SIEM delivery

Sign in and out once, then search your SIEM for the sign-in and sign-out audit events for that user.

Expected result: both events arrive, attributed to the user, with the kamiwaza.audit.integrity.* attributes. Configure your SIEM's retention, backup and alerting for these records; that is your control.

6.7 Show the DoD notice and consent banner

Under stig, users must see the Standard Mandatory DoD Notice and Consent Banner, and accept it, before they sign in. The Kamiwaza console provides this as a consent gate: a modal notice over a blank page, with one accept button and no way to dismiss it. The console shows it before it redirects to the sign-in page, and loads nothing else until the user accepts. The console also asks Core to log each acceptance with the client address and browser, but it lets the user continue if that request fails, so don't treat the log as a complete record of acceptances.

1. Turn the gate on in stig-values.yaml (it is in the Step 4.12 example) and apply it with the Step 5.3 command:

   core:
     security:
       consent:
         enabled: true

   The values file always decides whether the gate is on: the console and the API can't turn it on or off. Consent Gate and Classification Banners describes the endpoints it uses.

2. Put the notice text your ISSO approves in a file, dod-notice.html, as simple HTML: paragraphs, lists, headings, emphasis and links are kept, other markup is removed, and the limit is 10,000 characters. The standard text is:

   <p>You are accessing a U.S. Government (USG) Information System (IS) that is provided for USG-authorized use only.</p>
   <p>By using this IS (which includes any device attached to this IS), you consent to the following conditions:</p>
   <ul>
     <li>The USG routinely intercepts and monitors communications on this IS for purposes including, but not limited to, penetration testing, COMSEC monitoring, network operations and defense, personnel misconduct (PM), law enforcement (LE), and counterintelligence (CI) investigations.</li>
     <li>At any time, the USG may inspect and seize data stored on this IS.</li>
     <li>Communications using, or data stored on, this IS are not private, are subject to routine monitoring, interception, and search, and may be disclosed or used for any USG-authorized purpose.</li>
     <li>This IS includes security measures (e.g., authentication and access controls) to protect USG interests--not for your personal benefit or privacy.</li>
     <li>Notwithstanding the above, using this IS does not constitute consent to PM, LE or CI investigative searching or monitoring of the content of privileged communications, or work product, related to personal representation or services by attorneys, psychotherapists, or clergy, and their assistants. Such communications and work product are private and confidential. See User Agreement for details.</li>
   </ul>

3. Set the text as a named administrator, using an Admin-scope personal access token (Step 6.4). The commands in Steps 6.7 to 6.9 read the token once, without echo, and pass it to curl through a file descriptor, so it stays out of your shell history and the process list. kz prints Kamiwaza's answer, or, when a call fails, Request failed: with the error detail. Run them in bash on your administration workstation, with curl 7.76 or later:

   HOST=<platform-host>
   read -rsp 'Admin-scope PAT: ' TOKEN; echo
   kz() {
     local out
     if out=$(curl -sS --fail-with-body -H @<(printf 'Authorization: Bearer %s\n' "$TOKEN") "$@"); then
       printf '%s\n' "$out"
     else
       printf 'Request failed: %s\n' "$out" >&2; return 1
     fi
   }
   
   ( set -o pipefail
   kz "https://$HOST/api/security/admin/config" \
     | jq --rawfile notice dod-notice.html '{consent_enabled: true, consent_content: $notice,
         consent_button_label, banner_enabled, banner_top_text, banner_top_color,
         banner_bottom_text, banner_bottom_color}' \
     | kz -X PUT "https://$HOST/api/security/admin/config" \
         -H 'Content-Type: application/json' --data-binary @- \
     | jq '{consent_enabled, consent_button_label}' )

   The update replaces the whole display configuration, so the command reads the current settings first and changes only the notice; your classification banner settings are kept. The button label comes from core.security.consent.buttonLabel (default Accept).

Expected result: the update prints "consent_enabled": true. Within a few seconds, every Core replica serves the notice (Step 7.9 checks the text). In a new private browser window, https://<platform-host>/ shows the notice first; the sign-in page appears only after you accept it.

Know the gate's limits, and record them for your assessor:

• It is shown once per browser tab session, and again after the user signs out of the console. An expired session or a failed sign-out doesn't reset it, and a tab opened from a tab that already accepted can carry that acceptance with it.
• It belongs to the Kamiwaza console. Keycloak's own pages, reached directly by URL, don't show it. Brokered users also see your IdP's banner when they sign in there.
• Clients that use personal access tokens, such as the SDK and CLI, don't see it; show the notice in your automation's own onboarding.

6.8 Retire the bootstrap administrator

The install creates one shared administrator in the platform realm, admin, for the first sign-in (Step 6.1). Shared accounts must not remain: once individual administrators exist, disable it. Until you do, Core logs a WARNING that names it at every start. (The chart also creates svc-core, Core's own non-person service account. It is not a sign-in account; Step 3.2 covers it.)

Kamiwaza doesn't stop you from disabling this account, even when no other administrator works, and the disable can't be undone through the API (see Step 6.9 on restoring accounts). Do steps 1 and 2 first, with a different account.

Known limitation in this release: with relationship-based access control (ReBAC) on, the install grants the cluster's owner relation only to the bootstrap administrator, and Kamiwaza has no API to grant it to another account. Other administrators therefore get 403 on the cluster capability check, so the console's cluster hardware views show the hardware as unknown for them. Record this for your assessor if you rely on those views.

1. Name individual administrators. Give each administrator their own account: a brokered user whose IdP group maps to the admin role (Step 6.2), or a local account with the admin role. Each signs in once and enrolls MFA, then signs in again with the code before minting a token (Step 6.4). Your break-glass accounts (Step 3.3) are individually named local accounts too; don't use admin as one.

2. Check a named administrator's access. Sign in as one of them, not as admin, and mint an Admin-scope personal access token (Step 6.4). On your administration workstation, define kz as in Step 6.7 with that token, then count the other working administrators:

   kz "https://$HOST/api/auth/users/" \
     | jq '[.[] | select(.active and .username != "admin" and ((.roles // []) | index("admin")))] | length'

   Expected result: 1 or more. A 403 means the token doesn't carry the Admin scope.

3. Disable the bootstrap administrator through the users API, with that same token. Disabling here also revokes the account's personal access tokens, ends its Kamiwaza and Keycloak sessions, and revokes its refresh and offline tokens:

   if USERS=$(kz "https://$HOST/api/auth/users/"); then
     BOOT_ID=$(jq -r '.[] | select(.username == "admin") | .id' <<<"$USERS")
     echo "${BOOT_ID:-no Kamiwaza record}"
     [ -n "$BOOT_ID" ] && kz -X PUT "https://$HOST/api/auth/users/$BOOT_ID" \
       -H 'Content-Type: application/json' -d '{"active": false}' | jq '{username, active, revocation}'
   fi

   Expected result: "active": false, and revocation reads like User <id> revoked: login disabled, 1 PATs revoked, 2 sessions ended. If it printed no Kamiwaza record, the account never signed in to Kamiwaza, so it holds no Kamiwaza tokens or sessions; disable it in Keycloak in the next step. If a call fails, Request failed: shows the cause. A 403 naming a protected role means the account holds a Keycloak administration role: remove that role with the admin CLI (Step 6.5, with kc defined as in step 4), for example kc remove-roles -r kamiwaza --uusername admin --cclientid realm-management --rolename <role>, then run this step again. Disabling it only in Keycloak would leave its Kamiwaza tokens valid.

4. Confirm it in Keycloak. Open the Keycloak pod's shell and sign in to the admin CLI as in Step 6.5 (Step 6.5 deletes the CLI session when you finish, so sign in again each time). In that shell:

   kc() { /opt/keycloak/bin/kcadm.sh "$@" --config /tmp/kcadm.config; }
   kc get users -r kamiwaza -q username=admin -q exact=true --fields username,enabled

   Expected result: "enabled" : false. If it still shows true because step 3 found no Kamiwaza record, disable it here and check again:

   BOOT_KC_ID=$(kc get users -r kamiwaza -q username=admin -q exact=true --fields id --format csv --noquotes)
   kc update "users/$BOOT_KC_ID" -r kamiwaza -s enabled=false

   Remove the CLI session as in Step 6.5 when you finish.

5. Confirm sign-in is refused. In a private browser window, try to sign in as admin: Keycloak refuses it. After Core's next restart (an upgrade, or kubectl -n kamiwaza rollout restart deploy/core-api), its log says the bootstrap account is disabled instead of the WARNING. A Bootstrap admin check UNKNOWN line means the check couldn't run; it isn't evidence of the disable. From your administration workstation:

   kubectl -n kamiwaza logs deploy/core-api | grep 'Bootstrap admin'

Disable the account; don't delete it. The chart's user seeding recreates a deleted bootstrap account at the next upgrade, enabled and with the password still stored in the kamiwaza-user-admin Secret, but it never re-enables a disabled one. Step 7.9 checks that it stays disabled. Don't list admin in core.auth.inactivityDisable.exemptAccounts (Step 4.9). The shared password stays in that Secret; restrict who can read it, and record who can, in your SSP.

Don't disable the master realm's admin, which you use for the Keycloak admin CLI in Step 6.5. It is the realm-administration credential, not a platform sign-in account: the install and upgrade Jobs use it, and the Keycloak admin console isn't routed. Record its emergency-only use, and who holds the keycloak-admin Secret, in your SSP.

6.9 Let your SIEM or SOAR disable a user

Kamiwaza doesn't run its own intrusion detection. Your IDS and SIEM detect an intrusion; your SOAR, or an operator, responds by calling Kamiwaza's users API to disable the account. Kamiwaza provides three things for this:

• The audit events your SIEM correlates (Step 4.8).
• Automatic lockout. Keycloak's brute-force detection temporarily locks an account after repeated failed sign-ins, with no SOAR action needed. In the Keycloak pod's admin CLI (Step 6.5, with kc defined as in Step 6.8), kc get realms/kamiwaza --fields bruteForceProtected,failureFactor,permanentLockout shows "bruteForceProtected" : true.
• One call that ends all access. PUT /api/auth/users/<id> with {"active": false} disables the account and revokes its personal access tokens, ends its Kamiwaza and Keycloak sessions, revokes its refresh and offline tokens, disables its Keycloak account, and writes an audit entry with the result. It works for local and brokered users.

Set up the integration account:

1. Create a dedicated local account for the SOAR through Kamiwaza's user management, for example soar-integration, with the admin role; the users API requires it. Don't give it audit-admin. Name the person who owns it in your SSP.
2. Sign in as it once and enroll its one-time code, then sign out and sign in again with the code (Step 6.4), and mint a personal access token from that second session with the Admin scope. A token with the default scope can't call the users API. Set its lifetime no longer than your rotation period (the default is 30 days, the maximum 365 days). Store the token in your SOAR's secret store, and rotate it before it expires.
3. Have the SOAR call GET /api/auth/users/ on a schedule as a health check, and alert when it doesn't answer 200. Token use counts as activity, so this also keeps the account from the inactive-account disable (Step 4.9); don't list it as an exemption.

The response playbook, as shell commands, with kz defined as in Step 6.7 using the SOAR's token. First find the user's Kamiwaza id, with one of these lookups:

# By username:
USER_ID=
if USERS=$(kz "https://$HOST/api/auth/users/"); then
  USER_ID=$(jq -r --arg u "<username>" '.[] | select(.username == $u) | .id' <<<"$USERS")
fi

# By the user id (Keycloak subject) in the audit event (a 404 means no such user):
USER_ID=$(kz "https://$HOST/api/auth/users/resolve/<user-id-from-event>" | jq -r '.id // empty')

Then disable the account, and confirm the result from Kamiwaza's own record rather than from the answer alone:

if [ -z "$USER_ID" ]; then
  echo 'No user id: see any Request failed line above, or check the identifier; nothing was disabled' >&2
else
  kz -X PUT "https://$HOST/api/auth/users/$USER_ID" -H 'Content-Type: application/json' \
    -d '{"active": false}' | jq '{username, active, revocation}'
  kz "https://$HOST/api/auth/users/$USER_ID" | jq '{username, active}'
fi

The account is contained only when the final read shows "active": false. For any other answer, kz prints Request failed: with the status detail:

Answer	Meaning	SOAR action
200	Disabled. revocation says what was ended.	Record it in the incident.
503	The detail names the step. If the final read shows "active": false, the disable is committed and Kamiwaza already refuses the account; a later step, such as the Keycloak sign-out, failed. If it shows "active": true, nothing changed yet: under stig, Core couldn't read the account's Keycloak roles before the change.	Retry the same request until it answers 200; every step is safe to repeat. If the account stays active, page your ISSM; a Keycloak-only disable has the token limit in the next row.
403 Admin role required	The SOAR's token doesn't carry the Admin scope, or its account lost admin.	Fix the integration account; nothing was disabled.
403 naming a protected role	Under stig, the users API can't change an account that holds audit-admin or a Keycloak administration role.	Page your ISSM. Disabling the account with the Keycloak admin CLI (Step 6.5) stops new sign-ins and token refresh, but personal access tokens it already holds stay valid until they expire. Record this in your POA&M, and don't issue long-lived tokens to these accounts.
409	The detail names the cause: the change would leave no administrator, the SOAR account targeted itself, or the user's Keycloak account no longer exists.	Escalate to an administrator; nothing was disabled.
401	The SOAR's own token expired or was revoked.	Rotate the integration token; nothing was disabled.
404	No such user.	Check the identifier.

Other replicas may accept a cached access token for up to 5 seconds after the disable. Personal access tokens are checked against the token registry on every request; if the pat_revoke step failed (a 503 naming it), they share the same 5-second window until a retry succeeds. For a brokered user, also disable the account at your IdP, which owns that identity.

Restoring a cleared user depends on how the account was created:

• A local account created through Kamiwaza's user management: send {"active": true} to the same endpoint. Kamiwaza re-enables its Keycloak account too. Revoked tokens and sessions stay revoked, so the user signs in again and mints new tokens.
• A brokered user: the disable is permanent; Kamiwaza refuses {"active": true}. To let the person back in, delete the user (kz -X DELETE "https://$HOST/api/auth/users/$USER_ID"), then delete its account in this cluster's Keycloak realm with the admin CLI (kc delete users/<keycloak-id> -r kamiwaza). Their next sign-in through your IdP creates a new account with none of the old workrooms, grants or tokens. If the account isn't in this cluster's realm, skip the Keycloak delete; an administrator releases the username instead (POST /api/auth/users/<id>/release-username).
• An account created directly in Keycloak (for example with the admin CLI): the disable is also permanent, and nothing recreates the account, so don't delete it in Keycloak. Give the person a new local account through Kamiwaza's user management, under a new username; the old name stays reserved for the disabled account. The new account starts with none of the old workrooms, grants or tokens.

Use the same disable call when a person leaves.

Step 7: Verify the configuration

Run these checks after the install and after every upgrade. Keep the output with your evidence. Set the hostname first:

HOST=kamiwaza.example.mil   # your platform hostname

7.1 The profile is applied

helm -n kamiwaza get values kamiwaza --all -o json | jq -r '.global.securityProfile, .auth.keycloak.realm.securityProfile'
kubectl -n kamiwaza exec deploy/core-api -- printenv KAMIWAZA_SECURITY_PROFILE AUTH_SECURITY_PROFILE

Expected result: the first line is stig, and the second is stig or null (the realm then follows the umbrella); Core prints stig twice. Don't grep every securityProfile line: the subcharts' own global.securityProfile defaults show default on a correct stig install, because the umbrella's global replaces them when the chart renders.

7.2 FIPS mode in the platform

kubectl -n kamiwaza get pods,jobs,cronjobs -o json \
  | jq -r '.. | objects | (.containers?, .initContainers?, .ephemeralContainers?) // empty | .[]?.image' \
  | sort -u
kubectl -n kamiwaza exec deploy/core-api -- python3 -c 'import ssl,_hashlib; print(ssl.OPENSSL_VERSION, _hashlib.get_fips_mode())'
kubectl -n kamiwaza exec deploy/core-api -- python3 -c 'import hmac,hashlib; hmac.new(b"x"*32, b"probe", hashlib.md5)'

Expected result:

• Every image in the list (regular, init and job containers) is a -fips variant, apart from the images your CRM or POA&M lists by name.
• Each image matches the release's image list, by digest or by the release tag (Step 4.2 says which images use which).
• Hook Jobs that Helm already deleted don't appear here; check them against the release's image list.
• Core reports FIPS mode 1, and the HMAC-MD5 call fails with ValueError. (A bare hashlib.md5() call isn't a reliable probe: Python can fall back to its built-in MD5.)

7.3 Edge TLS and exposed ports

Run these from an administration workstation that is not in FIPS mode. A FIPS-mode client refuses TLS 1.1 and ChaCha20 itself, so the negative probes would pass without ever testing the server. OpenSSL 3 also disables TLS 1.1 at its default security level; DEFAULT@SECLEVEL=0 lets the client offer it so that the server's answer is what you test.

openssl s_client -connect "$HOST:443" -tls1_1 -cipher 'DEFAULT@SECLEVEL=0' </dev/null
openssl s_client -connect "$HOST:443" -tls1_2 -cipher ECDHE-RSA-CHACHA20-POLY1305 </dev/null   # ECDSA certificate: ECDHE-ECDSA-CHACHA20-POLY1305
openssl s_client -connect "$HOST:443" -tls1_3 -ciphersuites TLS_CHACHA20_POLY1305_SHA256 </dev/null
openssl s_client -connect "$HOST:443" -tls1_2 -cipher ECDHE-RSA-AES256-GCM-SHA384 </dev/null
sudo nmap -sS -p- "$HOST"   # run this one from outside your administration network

Expected result: port 80 isn't open (the scan reports it closed or filtered); the TLS 1.1 and ChaCha20 handshakes fail with an alert from the server (handshake failure or protocol version), not a local no protocols available or no cipher match error; the TLS 1.3 ChaCha20 handshake either fails or, if your edge proxy accepts it, is recorded in your CRM or POA&M with the edge proxy's module status (Step 4.2; global.ingress.tls.cipherSuites governs TLS 1.2 suites only); the AES-GCM handshake succeeds and names that suite (with an ECDSA server certificate, use the ECDHE-ECDSA- variants of both TLS 1.2 probes); the scan, run from outside your administration network, shows only 443, plus 22 if your site allows SSH administration from there. From the administration network you may also see the Kubernetes API (for example 6443); that is expected.

7.4 Content Security Policy

curl -sI "https://$HOST/" | grep -i '^content-security-policy'
curl -s -D - -o /dev/null "https://$HOST/api/auth/login" | grep -i '^content-security-policy'
curl -s -D - -o /dev/null -H "Cookie: access_token=<value>" "https://$HOST/runtime/apps/<app>/" | grep -i '^content-security-policy'

For the last check, use a signed-in session's access_token cookie and the path of an installed App Garden extension.

Expected result: every response carries Content-Security-Policy (enforced), not Content-Security-Policy-Report-Only. The Core API policy starts with default-src 'none'. The NOTES from Step 5.2 contain no WARNING: V-204760 not met line.

7.5 Sign-in, sessions and MFA

curl -s -D - -o /dev/null -X POST "https://$HOST/api/auth/token" \
  -d 'grant_type=password&username=probe&password=probe' | grep -iE '^HTTP|x-kamiwaza-auth-failure'

Expected result: with no break-glass window open (Step 6.5), 403 with X-Kamiwaza-Auth-Failure: password_login_disabled.

Then check the realm settings with the Keycloak admin CLI (Step 6.5). In the Keycloak pod's shell:

kc() { /opt/keycloak/bin/kcadm.sh "$@" --config /tmp/kcadm.config; }
kc get realms/kamiwaza --fields ssoSessionIdleTimeout,ssoSessionMaxLifespan,offlineSessionIdleTimeout,passwordPolicy
kc get authentication/required-actions/CONFIGURE_TOTP -r kamiwaza
kc get "authentication/flows/kamiwaza%20browser/executions" -r kamiwaza
kc get "authentication/config/<authenticationConfig of the User session count limiter>" -r kamiwaza --fields config
kc get clients -r kamiwaza -q clientId=kamiwaza-platform --fields attributes | grep access.token.lifespan

The first command shows the session lifetimes and password policy, the second the one-time-code action, the third the session limiter in the browser flow, the fourth that limiter's settings (behavior is Deny new session, userRealmLimit is 3 or fewer; take the id from the third command's authenticationConfig), and the fifth the console's access-token lifespan:

Setting	Expected under stig
SSO session idle	15 minutes (900 s)
SSO session max	8 hours (28800 s)
Offline session idle	15 minutes (900 s)
Password policy	hashAlgorithm(pbkdf2-sha512), hashIterations(210000), length(15), one each of upper case, lower case, digit and special character, passwordHistory(5), notUsername
Required action CONFIGURE_TOTP	Enabled
Sessions per user	3 or fewer; deny new session (the chart default under stig; an explicit terminateOldest is accepted but lets an evicted session's token keep working until it expires)
kamiwaza-platform access-token lifespan	3600 s or less

7.6 Logout invalidates the session

Sign in with a test account, copy the value of the access_token cookie from the browser's developer tools, sign out, then call the API with the copied value:

curl -s -w '\nHTTP %{http_code}\n' -H "Cookie: access_token=<copied-value>" "https://$HOST/api/auth/users/me"

Expected result: HTTP 401, with a body containing the message "This session was signed out. Sign in again to continue."

7.7 Audit records verify

The verifier ships in the Core image, and core-api mounts the key, so run it there. Feed each core-api pod its own log; don't add --timestamps or --prefix to kubectl logs, because each line must be JSON:

for POD in $(kubectl -n kamiwaza get pod -l app.kubernetes.io/name=core-api -o jsonpath='{.items[*].metadata.name}'); do
  echo "== $POD"
  kubectl -n kamiwaza logs "$POD" -c core-api \
    | kubectl -n kamiwaza exec -i "$POD" -c core-api -- \
      python3 -m kamiwaza_observability.audit.verify \
        --key-file /app/secrets/audit-integrity/hmac-key --allow-truncated-head /dev/stdin
  echo "verify exit code for $POD: $?"
done

To verify a slice exported from your SIEM instead, pipe the file into the same kubectl exec ... /dev/stdin command in place of kubectl logs. The key never leaves the cluster.

Use --allow-truncated-head for any time-window slice: a slice that starts after the first event of a stream is otherwise reported as tampering. Filter the slice to Core's sealed audit streams. Events from the emitters that carry no seal (Step 4.6) are reported as tampering unless you add --allow-unsealed; if you use it, cite that documented exception in your evidence. If your collector can reorder events, add --allow-reorder. Repeat --key-file for each key that was in use during the slice, for example across a key rotation.

Expected result: verify exit code for <pod>: 0 for every pod. Exit 1 reports each changed, deleted, replayed or reordered event with its file and line. Exit 2 means the slice can't be verified, for example it has no sealed events. The verifier can't detect removal of the newest events; your SIEM's retention control covers that.

7.8 Rescan the hosts

Repeat the OpenSCAP scan from Step 1.2, writing host-post-arf.xml, and explain every difference from the pre-install scan.

On Kamiwaza's qualification host (KKS with the local-path provisioner), the install caused these new failures. Expect them, and record their justification in your CRM:

Rule area	Cause
IPv4 and IPv6 all.forwarding sysctls set to 1	The cluster network (CNI) needs packet forwarding.
World-writable directories	Local-path volume directories are created 0777, under the root-only /opt/local-path-provisioner.
Files with no owning user or group	Files in local-path volumes belong to container user ids that have no host account.

With another storage provisioner, the volume findings may differ.

7.9 Notice shown, no shared accounts

curl -sS "https://$HOST/api/security/public/config" | jq '{consent_enabled}'
curl -sS "https://$HOST/api/security/public/config" | jq -r '.consent_content // ""' > notice-served.html
diff <(tr -s ' \n' ' ' < dod-notice.html) <(tr -s ' \n' ' ' < notice-served.html) && echo 'notice matches'

Then, in the Keycloak pod's shell, signed in to the admin CLI as in Step 6.5:

kc() { /opt/keycloak/bin/kcadm.sh "$@" --config /tmp/kcadm.config; }
kc get users -r kamiwaza -q username=admin -q exact=true --fields username,enabled

Expected result: consent_enabled is true, and the served notice matches your approved file (notice matches; keep notice-served.html with your evidence). If the default <p>Please accept the terms to continue.</p> is served instead, repeat Step 6.7. If other text is served after a successful Step 6.7, a consent file installed with Core (config/security/consent.html) overrides the API text; change the text there. The bootstrap administrator shows "enabled" : false (Step 6.8).

Step 8: Upgrade under stig

1. Read the release notes for the target version and any changes to stig refusals, and check the Helm client again (Step 5.1).

   First upgrade from a non-FIPS Keycloak: if the upgrade stops and lists accounts whose passwords still use Argon2, have each listed account sign in once on the current Keycloak, or reset its password with the Keycloak admin CLI (Step 6.5), then upgrade again. Don't turn off global.fips.enabled to get past it.

2. Stop writes and keep recovery points. Stop ingress and external writers with your site's maintenance procedure before you upgrade: the Core snapshot is consistent, but writes accepted after it are lost if you restore. The snapshot covers only Core's database. If the upgrade can change them, keep coordinated recovery points for SpiceDB, Keycloak, etcd, extension databases and object storage too. Save the current chart version, values and Secrets, including the Fernet key and PostgreSQL credentials:

   helm -n kamiwaza history kamiwaza -o yaml > before-upgrade-history.yaml
   helm -n kamiwaza get values kamiwaza -o yaml > before-upgrade-values.yaml

   Store both files with the rest of your protected backup material; the values can contain secrets.

3. Check the backup claim is Bound and has room. Before the first upgrade, Pending with no Warning event is fine on a class that binds on first use:

   kubectl -n kamiwaza get pvc core-upgrade-backups

4. Render first, with the same values files, as in Step 5.2, with VERSION set to the new version.

5. Upgrade with the Step 5.3 command. Before any schema change, the chart takes a validated snapshot of the Core database to the backup claim. If the upgrade fails before the database migration, Helm rolls back to the previous revision. The realm password and session policy are reapplied on upgrade.

   If it fails after the migration, the automatic rollback fails too, and the release is left FAILED. Helm restores manifests but not the database schema, and the previous version refuses the newer schema. That is the safe outcome. Keep traffic stopped, and restore Core's database from the pre-upgrade snapshot before you roll back the application, as follows.

   To restore, work as the namespace administrator (the installer identity), never cluster-admin, with the namespace owner available for step 3:

   1. Stop traffic and automation. Pick the last successful Helm revision and the snapshot taken before the failed upgrade (helm -n kamiwaza history kamiwaza; don't assume it is the previous revision number).
   2. Mount the backup claim read-only in a temporary pod, and check the snapshot against its .sha256 file. Stop if it doesn't match.
   3. Ask the namespace owner (the platform administrator, since the installer identity can't create quotas) to create a ResourceQuota of count/pods: "0" in the namespace. Then delete every pod except core-postgres-0 and the backup pod.
   4. End the database's open sessions, restore the snapshot with pg_restore --create --clean --if-exists --exit-on-error into core-postgres-0, and roll back the release with helm rollback kamiwaza <revision> -n kamiwaza --no-hooks.
   5. Have the namespace owner remove the quota, delete the backup pod, wait for the rollout, then check the schema version, sign-in, authorization, and a read and a write before you reopen traffic. If any step fails, keep the freeze and retry from the original snapshot.

6. First upgrade from 1.4.0-rc.4 or earlier: sessions and tokens from before the upgrade keep their old terms. Sessions opened before it still count toward the session limit, so users already at the limit are refused, and access tokens issued before it keep their 4-hour lifetime. Once, right after the upgrade, sign every Keycloak session out with the Keycloak admin CLI (Step 6.5): kc create realms/kamiwaza/logout-all. It calls the Admin REST API POST /admin/realms/kamiwaza/logout-all, the same call the field kit's kz-session-reset.sh makes; the Keycloak admin console isn't routed under stig, so you can't use it here. Then either rotate the realm signing key or wait 4 hours. Core checks tokens itself, so ending a session doesn't end a token already issued. kz-session-reset.sh rotates the key: it adds a new RSA key provider with a higher priority, disables the old one, and restarts core-api, core-scheduler and the core-raycluster-* deployments, because each keeps the old key until it reloads its key set. Every user then signs in again. Personal access tokens are signed by Core, not Keycloak, so neither step affects them. The release NOTES from Step 5.2 remind you on every stig upgrade that uses these defaults.

7. Verify again with Step 7.

To skip the snapshot for one revision, for example during a storage incident, set core.postgresql.upgradeBackup.waiveForRevision to the next Helm revision number and give a waiverReason. The waiver is recorded in the release. The restore procedure is in step 5 above.

High security mode (planned)

A higher tier above stig, high security mode, is planned. It will require the stig profile, and under it no override that weakens a control will be allowed, including the acknowledged report-only CSP override that stig permits (Step 4.10). It is not available in this release; everything in this guide describes the stig profile.

Reference

What stig sets

Area	default	stig
Sign-in	Console sign-in or password grant	Browser redirect only; password grant refused except break-glass
MFA	Optional	Required (TOTP for local accounts; IdP-asserted for brokered)
PAT minting	Any session	MFA session only
Sessions	30 min idle, 10 h max, 5 per user; a new sign-in over the limit ends the oldest session	15 min idle, 8 h max, at most 3 per user; a new sign-in over the limit is refused
Console access token	4 h	At most 1 h
Audit session fingerprint key	Chart-managed	Chart-managed; required
Offline sessions	Allowed	Not requested by Core; 15 min idle
Logout	Token valid until expiry	Server-side denylist; 503 if store unreachable
Refresh token	Browser cookie	Held server-side, encrypted
Inactive accounts	Disabled after 35 days	Disabled after at most 35 days; can't be turned off
admin + audit-admin	Allowed with a warning	Refused
Audit integrity	Off unless a key is set	Required
Log level	Any	INFO or DEBUG only, so audit events are never dropped
CSP (console and API)	Enforced	Enforced
Edge TLS	TLS 1.2 floor; lower allowed with a warning	TLS 1.2 floor; lower refused; Gateway API refused
Encrypted storage	Not checked	Declaration required for every class
Pre-upgrade database snapshot	Opt-in	Required
OpenSearch Dashboards sign-in	On; off warns	On; off refused

Your responsibilities

The Security profiles and STIG alignment page has the responsibility table to start your CRM from. The controls this guide asks you to provide are: the host STIG and FIPS mode (Step 1), time synchronization (1.3), etcd encryption (2.1), encrypted storage (2.2), the network boundary and port exposure (2.3), cluster hardening and pod security (2.4), your IdP, its MFA and PKI (3.2, 6.2), your SIEM's retention, backup and alerting (4.8, 6.6), the notice text your ISSO approves (6.7), and the intrusion response that calls Kamiwaza's disable (6.9).

Support

Kamiwaza publishes a support and maintenance policy covering which releases receive security fixes and how quickly.

Related pages

• Security profiles and STIG alignment
• Administrator Guide
• Audit administration
• Observability
• Upgrade
• GPU nodes