Observability Guide

Kamiwaza provides several ways to inspect deployment health, runtime errors, and security-relevant activity. The primary paths are the UI log viewer, Kubernetes-native logs, and OpenTelemetry (OTLP) export to the collector and backend your organization already runs.

Overview

Kamiwaza observability is built around four layers:

• Deployment logs surfaced through the Kamiwaza UI for models, apps, and tools
• Platform logs written as JSON to each container's stdout, available through kubectl logs and any node-level log collector
• OpenTelemetry export of logs, traces, metrics, and audit events to an OTLP endpoint you configure
• Audit events, a separate stream of security-relevant records that you route to your SIEM or log backend

Kamiwaza does not install a monitoring backend in a standard deployment, and it does not depend on host-local log directories or bundled Grafana/Loki services. You choose where telemetry is stored and how long it is kept.

Start Here

For most troubleshooting:

1. use the Kamiwaza UI log viewer first
2. check Kubernetes pod health and container logs if the issue is platform-wide
3. review your OpenTelemetry destination if your environment forwards logs, traces, metrics, or audit events out of the cluster

UI Log Viewer

Kamiwaza includes a logger service that powers the UI log viewer and deployment log APIs.

Common endpoints behind the standard API gateway include:

• GET /api/logger/deployments/all - list deployment logs
• GET /api/logger/deployments/type/{type} - filter by deployment type
• GET /api/logger/deployments/{type}/{id} - fetch log content
• GET /api/logger/deployments/{type}/{id}/patterns - view detected error patterns
• DELETE /api/logger/deployments/{type}/{id}?confirm=true - delete a deployment log file
• DELETE /api/logger/cleanup?days_old=30&dry_run=true - preview or remove older logs

Use the log viewer in the UI when you want to:

• inspect model startup failures
• review app or tool launch problems
• check for common error signatures such as out-of-memory or runtime crashes

Kubernetes-Level Troubleshooting

For cluster administrators, platform issues are usually easiest to diagnose through Kubernetes.

Typical checks include:

• pod status in the Kamiwaza namespace
• container restart counts
• recent pod logs for the affected service
• ingress or routing behavior for the public domain

Use your environment's standard Kubernetes and platform tooling to inspect those signals. If your organization centralizes Kubernetes logs in another system, treat that system as the source of truth for cluster-level troubleshooting.

Native Prometheus metrics

Planned for Kamiwaza 1.4

The controls and engine/database targets in this section are planned for the Kamiwaza 1.4 metrics changes. An earlier chart or Core image might not expose these settings or labels.

Native metrics use HTTP /metrics endpoints independently of OTLP export. Enable the endpoints with global.observability.metrics.enabled: true. Core API metrics continue to use OTLP; this setting does not add a Core API /metrics endpoint.

Choose monitor resources or label discovery

Control and owning chart	Default	Effect
global.observability.metrics.enabled	false	Enable available native endpoints, Core/SpiceDB/Keycloak database sidecars and scrape access policies. DataHub's builtin exporter has a separate switch below.
global.observability.metrics.monitors.enabled	true	Create PodMonitor/ServiceMonitor resources for enabled components and enable managed collector monitor discovery.
alerting.prometheusRules.enabled (separate observability release)	auto	Separately control audit-pipeline PrometheusRule alerts. auto renders them only when the API is served; false never renders them.
datahub.kafkaControllerAlerts.enabled	false	Separately enable DataHub Kafka controller PrometheusRule alerts when their prerequisites are present.
global.namespaces.observability	Release namespace when unset	Namespace admitted by native metrics authorization and default collector ingress.

For a cluster without Prometheus Operator CRDs, use:

global:
  observability:
    metrics:
      enabled: true
      monitors:
        enabled: false
datahub:
  kafkaControllerAlerts:
    enabled: false

datahub.kafkaControllerAlerts.enabled: false is explicit so a retained or site-level opt-in cannot reintroduce PrometheusRule resources into this CRD-free profile.

This profile leaves native listeners, database exporters, Services, credentials and access policies enabled. It suppresses the umbrella's first-party PodMonitors and ServiceMonitors. Keep vendored components' serviceMonitor.enabled and serviceMonitor.create options disabled; conflicting opt-ins fail rendering with the dependency name. The monitor switch does not disable PrometheusRule alerts or unrelated operator prerequisites. The observability chart is installed separately, outside the umbrella. Pass the monitor switch to that release too, alongside its local alerting control, to suppress its monitors and target allocator Prometheus-CR discovery:

global:
  observability:
    metrics:
      monitors:
        enabled: false
alerting:
  prometheusRules:
    enabled: false

Setting observability.alerting.* on the umbrella does not configure that separate release.

For a cluster using monitors, set global.observability.metrics.monitors.enabled: true. The platform owner must supply the podmonitors.monitoring.coreos.com and servicemonitors.monitoring.coreos.com APIs, plus installer permission to get, list, watch, create, update, patch and delete their namespaced objects.

Helm globals do not cross release boundaries. Pass the switches to each separately installed chart, including the CA chart, and apply alerting settings at that chart's local values path.

Prepare monitor CRDs

The platform owner performs these steps before installing Kamiwaza. Set PLATFORM_OWNER_KUBECONFIG to the owner's kubeconfig and PLATFORM_MONITORING_NAMESPACE to an existing namespace that will hold the Helm release record.

Inspect every existing CRD in the monitoring API group first. The package installs additional monitoring CRDs beyond PodMonitor and ServiceMonitor:

kubectl --kubeconfig "$PLATFORM_OWNER_KUBECONFIG" get crd \
  -o jsonpath='{range .items[?(@.spec.group=="monitoring.coreos.com")]}{.metadata.name}{"\n"}{end}'

If any of these CRDs exist, reuse their current ownership and coordinate any missing PodMonitor or ServiceMonitor API with that owner. Both required APIs must serve monitoring.coreos.com/v1. Do not install this package over another owner's CRDs. If the inspection returns no monitoring CRDs and monitor support is wanted, obtain the pinned Prometheus Operator CRD chart, version 29.0.0, on a connected machine:

helm pull prometheus-operator-crds \
  --repo https://prometheus-community.github.io/helm-charts --version 29.0.0

For an air-gapped cluster, transfer prometheus-operator-crds-29.0.0.tgz through the platform's approved artifact-transfer process to the owner's installation machine. An approved chart mirror can supply the same pinned package instead. Install from the directory containing the package:

helm upgrade --install prometheus-operator-crds \
  ./prometheus-operator-crds-29.0.0.tgz \
  --namespace "$PLATFORM_MONITORING_NAMESPACE" \
  --kubeconfig "$PLATFORM_OWNER_KUBECONFIG" --wait --timeout 5m
kubectl --kubeconfig "$PLATFORM_OWNER_KUBECONFIG" wait \
  --for=condition=Established --timeout=60s \
  crd/podmonitors.monitoring.coreos.com crd/servicemonitors.monitoring.coreos.com

The platform owner grants the installer the namespaced monitor permissions listed above through the platform's namespace-owner process. Product Helm installation retains the restricted installer identity and --skip-crds; the installer does not need CRD-write permission. This package installs API definitions only. Configure the platform's Prometheus Operator or compatible target allocator to discover monitors in the Kamiwaza namespace and to reach their scrape endpoints.

The package also registers the PrometheusRule API. A separately installed observability chart at alerting.prometheusRules.enabled: auto can then render alert rules. Set that release's local value to false if those rules are not wanted or its installer lacks PrometheusRule permissions; otherwise coordinate the required namespaced rule permissions with the platform owner. The monitor switch does not control these alert rules.

Discover targets without monitor CRDs

Configure the customer scraper's Kubernetes discovery and relabeling. Monitor opt-out does not install a replacement discovery job or add prometheus.io/* annotations.

Target	Pod label	Named port	Endpoint
vLLM and llama.cpp engines	kamiwaza.io/engine=vllm or llamacpp	http, tenant API port 8080	/metrics
Core, SpiceDB and Keycloak PostgreSQL sidecars	app.kubernetes.io/name is core-postgres, core-spicedb-postgres or keycloak-postgres	metrics, port 9187	/metrics
DataHub PostgreSQL builtin exporter	app.kubernetes.io/name=postgresql, app.kubernetes.io/component=primary	http-metrics, port 9187	/metrics

Engine labels identify runtime-created pods. Relaunch engine allocations persisted before the metrics change to receive the new label and llama.cpp metrics flag. Engines deployed before the upgrade keep their pod shape until relaunched.

The following jobs use Prometheus Kubernetes pod discovery. They assume the product namespace and Helm release are both kamiwaza. Change the namespace and app.kubernetes.io/instance match to the installed values and merge the jobs into the existing scraper configuration. Exporters run inside database pods, so discovery selects the database labels and the named exporter port.

scrape_configs:
  - job_name: kamiwaza-inference
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [kamiwaza]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_phase]
        regex: Running
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_kamiwaza_io_engine]
        regex: vllm|llamacpp
        action: keep
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: http
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_kamiwaza_io_engine]
        target_label: engine
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
  - job_name: kamiwaza-postgres
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [kamiwaza]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_phase]
        regex: Running
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
        regex: kamiwaza
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
        regex: core-postgres|core-spicedb-postgres|keycloak-postgres
        action: keep
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: metrics
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
        target_label: instance
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
  - job_name: kamiwaza-datahub-postgres
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [kamiwaza]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_phase]
        regex: Running
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
        regex: kamiwaza
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
        regex: postgresql
        action: keep
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
        regex: primary
        action: keep
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: http-metrics
        action: keep
      - target_label: instance
        replacement: datahub-postgresql
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

The database instance relabeling retains four distinct database identities even when PodMonitors are disabled. Use one scrape owner for each endpoint; disable overlapping jobs before enabling these jobs.

Authorize the customer scraper

The access examples assume the existing observability namespace is monitoring and the product namespace is kamiwaza. Replace both names with the installed values. Run the scraper in the namespace already selected by global.namespaces.observability; when unset, that namespace is the release namespace. Changing this global also changes default managed collector/gateway OTLP addresses and collector access rules. Coordinate collector placement and explicit OTLP endpoints before changing it; the value does not move running collectors.

Discovery and network access are separate requirements. The scraper ServiceAccount needs namespaced get, list and watch permission on pods. The following owner-managed objects grant pod discovery in kamiwaza to an existing ServiceAccount named prometheus in monitoring:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kamiwaza-metrics-discovery
  namespace: kamiwaza
rules:
  - apiGroups: ['']
    resources: [pods]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kamiwaza-metrics-discovery
  namespace: kamiwaza
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kamiwaza-metrics-discovery
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring

Default exporter ingress admits only OTel collector pods labeled app.kubernetes.io/component=opentelemetry-collector in global.namespaces.observability. For a Prometheus pod with app.kubernetes.io/name=prometheus, the namespace owner can add this scoped NetworkPolicy, substituting the actual namespace and pod labels:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: customer-prometheus-postgres-metrics
  namespace: kamiwaza
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: kamiwaza
    matchExpressions:
      - key: app.kubernetes.io/name
        operator: In
        values: [core-postgres, core-spicedb-postgres, keycloak-postgres, postgresql]
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus
      ports:
        - port: 9187
          protocol: TCP

The scraper's own egress policy must allow its Kubernetes API discovery and the selected endpoint ports. Other target ingress policies still apply. When Istio is enabled, the scraper needs an authenticated mesh identity in the configured observability namespace so the chart's metrics-only authorization admits it. Under STRICT mTLS, obtain the platform owner's approved authenticated scrape path and certificates or mesh configuration before using these HTTP jobs. NetworkPolicy alone does not provide mesh identity. Do not label Prometheus as an OTel collector, disable TLS verification or weaken target mTLS to make a scrape succeed.

Inference metric families

At llama.cpp b8775, all names begin with llamacpp:. Counters are prompt_tokens_total, prompt_seconds_total, tokens_predicted_total, tokens_predicted_seconds_total, n_decode_total, n_tokens_max and n_busy_slots_per_decode. Gauges are prompt_tokens_seconds, predicted_tokens_seconds, requests_processing and requests_deferred. Despite its name, n_busy_slots_per_decode is a counter at this pin. New llama.cpp allocations include --metrics.

At vLLM 0.28.0, the families used for this verification include vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:kv_cache_usage_perc, vllm:prompt_tokens_total, vllm:generation_tokens_total and vllm:request_success_total with finished_reason. Native metrics use the API listener; do not disable vLLM statistics logging. Histogram families expose their _bucket, _sum and _count samples.

Managed PostgreSQL exporters

The targets are core-postgres, core-spicedb-postgres, keycloak-postgres and datahub-postgresql. Core, SpiceDB and Keycloak use sidecars in their database pods. SpiceDB's exporter requires its enabled, managed PostgreSQL datastore; an external datastore Secret suppresses it. External databases need an exporter and credential provisioning owned by their database administrator.

DataHub uses the pinned PostgreSQL chart's builtin exporter. For an enabled DataHub chart with bundled PostgreSQL, enable it separately and disable the vendor's broad database policy so the chart's scoped policies apply:

global:
  observability:
    metrics:
      enabled: true
datahub:
  postgresql:
    enabled: true
    metrics:
      enabled: true
    primary:
      networkPolicy:
        enabled: false

Keep the global metrics switch enabled for managed collector egress. This example configures an already-enabled DataHub chart; it does not install DataHub by itself.

The chart pins Kamiwaza's FIPS postgres_exporter v0.19.1 image, built on a Chainguard base, at digest sha256:203fae5c928c3b2888375d63bad54b1d4a1d58d389d7ad88a42b7a427a517a5d, qualified against PostgreSQL 18.6. The stat_user_tables, statio_user_tables, statio_user_indexes and stat_statements collectors are disabled. Each exporter connects over loopback as the dedicated kamiwaza_metrics login with pg_monitor and CONNECT on postgres; its password comes from a Secret. The bootstrap Job uses the database administrator credential to provision that login. Exporters do not receive the administrator credential; the database server retains its own credentials.

By default, Helm generates and retains a separate <target>-exporter Secret with key password, reusing it on upgrades through a live-cluster lookup. DataHub's generated Secret also contains dsn. Renders without cluster access, including helm template and GitOps rendering, cannot reuse the live password; supply existing Secrets for those workflows.

For Core, SpiceDB and Keycloak, create a Secret in the database namespace with key password and set the owning subchart's postgresExporter.existingSecrets.<target>:

core:
  postgresExporter:
    existingSecrets:
      core-postgres: core-postgres-metrics-password
      core-spicedb-postgres: spicedb-metrics-password
auth:
  postgresExporter:
    existingSecrets:
      keycloak-postgres: keycloak-postgres-metrics-password

DataHub instead reads DATA_SOURCE_NAME from datahub.postgresql.metrics.extraEnvVars. Its supplied Secret needs both password for bootstrap and dsn for the exporter, using the same monitoring password. With the default database container port, the DSN is postgresql://kamiwaza_metrics:<URL-encoded-password>@127.0.0.1:5432/postgres?sslmode=disable. Match a changed container port in that DSN; bootstrap uses the database Service port.

datahub:
  postgresql:
    metrics:
      extraEnvVars:
        - name: DATA_SOURCE_NAME
          valueFrom:
            secretKeyRef:
              name: datahub-postgres-metrics-password
              key: dsn

Bootstrap provisions the login on installation and each native Helm revision. Its Job name includes the revision and a hash of its pod specification. After rotating a supplied password, run a native Helm upgrade with --wait-for-jobs to update the role, then restart the database pod in a maintenance window so its exporter reloads the credential. A Secret change alone neither reruns bootstrap nor reloads a running exporter's environment. Fixed-revision GitOps rendering also requires an explicit bootstrap rerun after a credential-only change.

Enabling or disabling the three shared sidecars rolls the single-replica Core, SpiceDB and Keycloak database StatefulSets; plan for a database restart. Their exporter password references are optional: a missing exporter Secret does not block container creation, and these sidecars have no readiness probe. PostgreSQL can remain Ready and in its Service endpoints while exporter authentication fails and pg_up=0. Bootstrap still requires that Secret, so a missing credential can fail its Job and a Helm installation that waits for Jobs. An exporter that cannot start, such as an image-pull failure, still keeps the pod NotReady. Mirror the exporter image before enabling metrics in a disconnected installation. DataHub uses its builtin exporter's own probe and credential configuration.

Disabling exporters removes their bootstrap Jobs while retaining generated Secrets and database roles. For DataHub, disable datahub.postgresql.metrics.enabled separately. The database owner can revoke the login with ALTER ROLE kamiwaza_metrics NOLOGIN and remove the retained Secret when no longer needed. Re-enabling the exporter provisions the login again.

PostgreSQL client access

Exporters use loopback inside database pods. The <target>-exporter-database NetworkPolicy admits bootstrap and the following same-namespace database clients by their exact app.kubernetes.io/name labels; it does not admit every core-* pod or grant collectors SQL access:

Database	Allowed client names, in addition to its bootstrap pod
core-postgres	core-api, core-runtime-route-reconciler, core-scheduler, core-raycluster, core-db-init, core-offline-embedding-import, core-storage-janitor, core-retrieval-streamer, init-keycloak-users, core-spicedb, spicedb-cutover, rebac-bootstrap
core-spicedb-postgres	core-spicedb, spicedb-datastore-migrate
keycloak-postgres	keycloak
datahub-postgresql	datahub-gms, datahub-mce-consumer, datahub, postgresql

DataHub also admits same-namespace SQL Jobs only when they carry both kamiwaza.io/datahub-sql-client=true and a batch.kubernetes.io/job-name label. New database clients need corresponding owner-reviewed client access; runtime-created pods are not covered by the chart's rendered-client tests. Existing Istio ServiceAccount authorization also applies when enabled. NetworkPolicies are additive, so another policy must not independently grant broader database access.

The bootstrap egress policy permits the matching database port, DNS and, when enabled, Istiod. Other egress policies remain additive. Scrapers connect to exporter port 9187, never the SQL port (5432 by default).

Verify the scrape

Validate the configuration with the scraper distribution's validator, such as promtool check config prometheus.yml, before reloading it. Confirm both inference engines appear in discovered targets while running and each target reports up=1. Check that the listed families are present and token counters increase during driven requests. For vLLM, also check that request counters and latency histogram counts increase. Sample gauges while requests are active; queue gauges change only under contention and can remain zero under light load. For PostgreSQL, confirm pg_up=1 for every managed target and verify that each has a distinct instance label. An absent expected exporter remains a failure even if the other targets are healthy.

What Kamiwaza Emits

Logs on stdout

Core services that initialize Kamiwaza telemetry write stdout records as one JSON object per line. With an OTLP log bridge active, stdout retains audit events, startup and exporter diagnostics, and ordinary records at WARNING or above by default. Ordinary INFO and DEBUG records can go only to OTLP. Without the bridge, stdout uses the configured application log level. Each JSON record has this envelope:

{
  "timestamp": "2026-10-27T12:00:00.000000Z",
  "severity_text": "INFO",
  "severity_number": 9,
  "body": "message text",
  "trace_id": "…",
  "span_id": "…",
  "resource": { "service.instance.id": "…" },
  "attributes": { "…": "…" }
}

trace_id and span_id appear when the record was written inside a traced operation. Values that look like credentials in URLs are redacted before the line is written.

Third-party components that Kamiwaza deploys, such as Keycloak and PostgreSQL, write logs in their own formats to stdout.

OTLP export

When you configure an OTLP endpoint (see Configure OpenTelemetry export), the core API, scheduler, and retrieval streamer initialize the OpenTelemetry SDK. They export the logs, traces, metrics, and audit events their code emits. Their stdout keeps the records described above rather than a complete copy of ordinary OTLP logs.

OTLP log export uses two bounded in-memory queues, one for audit events and one for ordinary logs. Records still queued when the process is killed are lost.

Ray head and worker containers also receive the endpoint and export settings. This does not initialize telemetry in every Ray task: task code must initialize a provider to export, and task-specific environment forwarding can replace the pod's service identity. Do not assume that database initialization or other utility jobs export telemetry merely because they consume the shared core configuration.

With no endpoint configured, the OpenTelemetry SDK is turned off in these workloads and nothing is exported. Logs still go to stdout.

Available in Kamiwaza 1.4.0

The charts set service.name on each exporting container: core-api, core-scheduler, core-retrieval-streamer, core-ray-head, and core-ray-worker. They also set service.namespace=kamiwaza, deployment.environment.name from global.environment, and k8s.namespace.name, k8s.pod.name, k8s.pod.uid, and k8s.node.name. The SDK supplies service.instance.id. These settings render only when an effective OTLP endpoint is configured.

Prometheus metrics endpoints

The charts render Prometheus Operator ServiceMonitor and PodMonitor resources when global.observability.metrics.enabled: true, unless global.observability.metrics.monitors.enabled is set to false (it defaults to true). When monitor integration is enabled, the cluster must already provide both custom resource definitions and a collector that discovers these resources. OTLP export and Prometheus scraping have separate settings; configuring an OTLP endpoint does not enable these monitors.

With the global metrics flag enabled, the charts can render the targets below for installed components. A monitor resource alone does not prove that a collector can reach its target; check your NetworkPolicies, mesh authorization, and collector discovery configuration.

The istiod, OpenTelemetry operator and kube-state-metrics monitors also require enabled: true in the optional Observability chart and their respective nativeMetrics flags. That chart defaults to enabled: false; the global metrics flag alone does not install it or render its monitors. Contact Kamiwaza support before using it.

The chart-managed metrics policies use global.namespaces.observability, falling back to the release namespace when that value is unset. Cross-namespace metrics rules in the Keycloak, DataHub and operator NetworkPolicies, and in the registry's policy without Istio, select scraper pods labeled app.kubernetes.io/component: opentelemetry-collector. Match the configured namespace and collector labels, or adapt the policies for your collector.

Component	Port	Path	Monitor owner
Keycloak	9000	/metrics	Auth chart
Ray head and workers	8080	/metrics	Core chart
Native vLLM and llama.cpp engines	Named http port on the engine API listener	/metrics	Core chart; selects pods labeled kamiwaza.io/engine: vllm or llamacpp
SpiceDB	9090	/metrics	Core chart, when SpiceDB is enabled
etcd	8080	/metrics	Core chart
SeaweedFS	9324	/metrics	SeaweedFS chart
Registry	9090	/metrics	Registry chart; access gate described below
DataHub GMS	4318 (named jmx)	/metrics	DataHub chart; additional monitoring flags described below
Extension operator	8080	/metrics	Extension operator chart
Placement operator	8080	/metrics	Placement operator chart
istiod	15014	/metrics	Observability chart, nativeMetrics.istiod
OpenTelemetry operator	8080	/metrics	Observability chart, nativeMetrics.otelOperator
kube-state-metrics	8080	/metrics	Observability chart, nativeMetrics.kubeStateMetrics
cert-manager	9402	/metrics	CA chart, when installed separately

Native engine allocations persisted before the metrics change require relaunch to acquire the engine label and, for llama.cpp, the --metrics argument. Enabling discovery alone does not update those existing workloads.

The DataHub monitor also requires serviceMonitor.enabled and global.datahub.monitoring.enablePrometheus in the DataHub release; both default to true. Without Istio, registry metrics remain disabled until metrics.applicationIngress in the Registry chart (registry.metrics.applicationIngress in the umbrella chart) explicitly lists the application's allowed NetworkPolicy peers. An empty list permits no application peers. With Istio, the registry metrics access policy is enabled with the global flag.

For the optional observability chart, nativeMetrics.*.enabled selects the three substrate targets above. Disable targets that are absent and adjust their namespace and selectors to match the site. These settings discover existing components; they do not install them.

These settings do not add a core API /metrics endpoint. Core API metrics use OTLP push export. Only one collector should own each scrape target.

Pass global.observability.metrics.enabled, global.observability.metrics.monitors.enabled and global.namespaces.observability to every Helm release you configure, including the optional observability and standalone operator releases; Helm globals do not cross release boundaries. If your installation uses cert-manager and the standalone ca chart, pass the flags to the CA release explicitly. An installation without cert-manager has no cert-manager target to scrape.

Without the monitor CRDs, leave metrics disabled or set global.observability.metrics.monitors.enabled: false and use your collector's label-based discovery. The latter keeps native metrics configuration and access rules enabled while omitting monitor resources and managed collector monitor integration. Disable any separately enabled dependency-chart ServiceMonitors too; the chart rejects contradictory settings. After configuring discovery, check that every enabled target appears in the collector's discovered targets and that its up series is 1.

Configure OpenTelemetry Export

Set OTLP export in your Helm values file, under global.observability. These are umbrella chart paths; when installing the Core chart directly, the scheduler override is scheduler.otel.endpoint.

Value	Default	Purpose
global.observability.enabled	false	Controls other observability integration, including the optional Ray stdout setting. It is not required to export to an external OTLP endpoint.
global.observability.endpoint	""	The OTLP endpoint of your collector, for example http://otel-collector.monitoring:4318. Without an effective endpoint, the core SDK is disabled. An existing core.scheduler.otel.endpoint override takes precedence over this global value.
global.observability.protocol	http/protobuf	The OTLP protocol: http/protobuf (usually port 4318) or grpc (usually port 4317). This global value applies when the global endpoint is used and no explicit core.scheduler.otel.protocol is set. With a core.scheduler.otel.endpoint override, set core.scheduler.otel.protocol too; otherwise it defaults to http/protobuf.
global.observability.metrics.enabled	false	Enables chart-managed native metrics configuration and access rules, with monitors by default.
global.observability.metrics.monitors.enabled	true	Enables monitor resources and managed collector monitor integration; set false for discovery without monitor CRDs.
global.observability.managedByKamiwaza	false	Leave false. It is for the optional in-cluster stack described below.
global.observability.dashboards.enabled	false	Leave disabled when using your own backend. This applies to the optional in-cluster stack.

Example

global:
  observability:
    endpoint: http://otel-collector.monitoring.svc:4318
    protocol: http/protobuf
    metrics:
      enabled: true   # only if the Prometheus Operator CRDs are installed

With http/protobuf, set the endpoint to the collector's base URL. The SDK appends /v1/logs, /v1/traces, and /v1/metrics.

Migration note: Core still honors the legacy KAMIWAZA_OTEL_ENABLED environment variable when OTEL_SDK_DISABLED is absent. Remove a leftover KAMIWAZA_OTEL_ENABLED=false when migrating to endpoint-based configuration; it can disable export even with an endpoint set.

Credentials and TLS

Available in Kamiwaza 1.4.0

Kamiwaza can send OTLP request headers, such as an authorization token, and trust a private CA for the OTLP endpoint. Both are read from Kubernetes Secrets you create:

Value	Default	Purpose
global.observability.headersSecretName	""	Existing Secret containing the OTLP request headers
global.observability.headersSecretKey	headers	Secret data key passed to OTEL_EXPORTER_OTLP_HEADERS through secretKeyRef
global.observability.caSecretName	""	Existing Secret containing the PEM CA bundle for the OTLP endpoint
global.observability.caSecretKey	ca.crt	Secret data key projected read-only at /app/observability-tls/ca.crt, the path in OTEL_EXPORTER_OTLP_CERTIFICATE

Headers and a private CA are optional independently. No new authentication or CA wiring renders without an effective endpoint. Create the Secrets and selected keys in every exporting workload's namespace before enabling them; a missing Secret or key prevents pod startup. Keep credentials in Kubernetes Secrets or your approved secret manager, never in a values file. Use TLS between the cluster and your backend in production.

Configure and verify delivery

1. Configure your collector's OTLP receiver. Ensure DNS, network policy, and mesh rules allow the core workloads to reach it.

2. For a collector requiring a bearer token and a private CA, prepare a local headers file containing authorization=Bearer%20YOUR_TOKEN and obtain the collector's PEM CA bundle as ca.crt. Percent-encode spaces and other reserved characters in OTLP header values. Create the Secrets in the workload namespace, substituting your namespace:

   kubectl -n kamiwaza create secret generic customer-otlp-headers --from-file=headers=./headers
   kubectl -n kamiwaza create secret generic customer-otlp-ca --from-file=ca.crt=./ca.crt

   Protect the local header file using your secret-management procedures and remove it after provisioning. Omit either Secret when your collector does not need it.

3. Add the following overlay to your existing installation values. Replace the endpoint with the hostname covered by the collector's certificate:

   global:
     observability:
       endpoint: https://otel-collector.monitoring.svc:4318
       protocol: http/protobuf
       headersSecretName: customer-otlp-headers
       headersSecretKey: headers
       caSecretName: customer-otlp-ca
       caSecretKey: ca.crt

4. Apply the overlay through the same Helm release or installation procedure you use to manage Kamiwaza, retaining your existing installation values. Wait for the core workloads to become Ready. Check that each container has the expected Secret references and CA mount without printing Secret data.

5. Make a normal API request and an unauthenticated request to a protected API route. At the collector or backend, verify a core log record, a kamiwaza.audit.authentication.attempt with kamiwaza.audit.result=failure, and resource service.name=core-api. Check the configured Kubernetes and environment attributes. For a trace check, make an authenticated GET /api/models/ request and look for the models.list_models span. Verify metrics separately at the collector; an HTTP success response alone does not prove telemetry delivery.

6. Check kubectl logs for the same core container. Audit records and warning/error diagnostics remain on stdout; an ordinary INFO log received over OTLP need not appear there. If the receiver rejects requests, check header formatting and the referenced key; for certificate errors, check the CA bundle and endpoint hostname. Do not disable certificate verification to make the test pass.

Send Telemetry to Your Own Backend

Kamiwaza publishes a CI-tested reference collector configuration, covered in the 1.5 integration guide. Until then, point the platform's OTLP endpoint at your own collector.

Optional in-cluster stack

The Kamiwaza deployment also contains an observability Helm chart: OpenTelemetry collectors, Data Prepper, OpenSearch, and OpenSearch Dashboards. It is not part of the standard install and is off by default (managedByKamiwaza: false). Contact Kamiwaza support before using it. It can forward logs and traces to an OTLP, OpenSearch, or Splunk HEC destination, and metrics to OTLP or Splunk HEC. It stores audit events in its own OpenSearch and also forwards them to any configured customer OTLP, OpenSearch, or Splunk HEC exporters. The chart generates its own certificates by default (global.observability.certificates.mode: generated) and does not require cert-manager. Sites that manage certificates can instead set the mode to byo (their own Secrets) or cert-manager, supplying the same value to the observability and observability-operator charts in both Helm releases. Existing cert-manager installations must keep mode cert-manager when upgrading. If you already run a backend, use your own collector as described above.

Audit Events

Kamiwaza emits audit events as a separate stream from operational logs. Route them to the system your organization uses as its audit record of truth. Core does not provide a durable audit database; set retention in your audit backend. Keycloak separately stores selected user events in its database for seven days by default (auth.keycloak.realm.eventsExpirationSeconds: 604800 in the umbrella chart). That operational event history is not a replacement for your audit archive.

Event names

Audit events follow a fixed catalog:

Event name	Records
kamiwaza.audit.api_action	HTTP outcomes of authenticated mutations and privileged API requests; a 202 records acceptance, not asynchronous completion
kamiwaza.audit.authentication.attempt	An authentication attempt and its result
kamiwaza.audit.authorization.decision	A permit or deny decision for a protected action or API
kamiwaza.audit.sensitive_data.access	Access to data designated as sensitive, including failed attempts
kamiwaza.audit.sensitive_data.change	Modification of sensitive data, including denied or failed attempts
kamiwaza.audit.sensitive_data.disposal	Successful deletion or disposal of sensitive data
kamiwaza.audit.sensitive_data.export	Successful export of sensitive data
kamiwaza.audit.session.end	Known logout, session revocation or observed login-session expiry
kamiwaza.audit.user_management.change	User, role membership, or security-relevant claim changes

Catalog records carry kamiwaza.audit.result (success or failure); check it rather than inferring success from the event name. Authentication, authorization and API-action producers must supply a result. For other known events, the emitter fills in success when the producer omits it, while preserving an explicit failure. Records also carry log.record.uid, retained across delivery retries. Actor and target attributes depend on the event and the available authenticated context.

How audit events arrive

• Over OTLP: as log records from the kamiwaza.audit logger, with the event name in the otel.event.name attribute. Route on that attribute.
• On stdout: in the same JSON envelope as other logs, with the event name at attributes."otel.event.name".

Some audit records still use an older format: an ordinary log record with event_category: "audit" instead of otel.event.name. Examples include buffered rebac_decision records and records emitted through logger.audit_info. To capture them, also route records whose event_category attribute is audit to your audit destination.

What is audited today

• API outcomes: on matched API routes, authenticated POST, PUT, PATCH and DELETE requests emit kamiwaza.audit.api_action. Privileged requests, including requests by verified platform administrators and admin-gated routes, are also audited, including reads. Records identify the method, route, endpoint target and HTTP outcome; they exclude request bodies, query strings and response contents. A 202 records acceptance rather than completion of the background operation. A response that is not fully delivered is recorded as a failure even if success headers were sent.
• Rejected requests: a 401 emits kamiwaza.audit.authentication.attempt with result failure, and a 403 emits kamiwaza.audit.authorization.decision with result failure. An authentication or authorization component can also identify a denial behind another response status.
• Core-owned sign-ins and credentials: SAML, certificate and Lite authentication attempts emit kamiwaza.audit.authentication.attempt with result success or failure. A successful sign-in identifies the minted credential and session when safely available. Native sign-in failures name a user only after resolving an existing account, and do not inherit an incoming browser's session. Browser-child and runtime credential issuance also emit authentication outcomes; token creation and revocation retain their user-management records.
• Session ends: kamiwaza.audit.session.end records known logout, revocation and expiry observed through Core's session tracking. Logout and expiry identify the session owner; administrative revocation identifies the acting administrator separately from the owner in enduser.id and the ended session in kamiwaza.audit.target.id. Duration is included when the original start is known. An expiry that Core never observes has no guaranteed end event; native Keycloak listener events still require separate mapping.
• Document classification denials: DDE marking enforcement emits kamiwaza.audit.authorization.decision for a rejected document, with result failure, a document target and a reason such as insufficient_clearance. The event contains identifiers and an optional normalized classification level, not document content or raw markings. Rejected ingest inputs use a hashed source reference as the target ID because no persisted document ID exists yet.
• Service operations: user management, role and permission changes, personal access token creation and revocation, and sensitive-data access, change, disposal and export are emitted by the services that perform them.
• Keycloak: the auth chart registers and reconciles the logging event listener, writing successful events at INFO and errors at WARN as JSON console records. Login, logout and admin events are available from the Keycloak container's stdout; admin representations are excluded. Keycloak's native event details are in the JSON message field and need source-specific parsing and mapping; they are not Core's catalog envelopes.

Successful non-admin reads are not generically audited; explicitly instrumented sensitive-data reads are covered. An API action outcome does not replace service-specific audit records for object changes or asynchronous completion.

Calls that authenticate their own non-platform credential, such as delegated-workload capability, effect and credential-use calls, are outside the generic API-action hook. They use a workload principal and depend on service-specific events.

Although API-action records exclude bodies and query strings, kamiwaza.audit.target.id can contain usernames, file paths and document or connector URNs derived from filenames. Treat the audit backend as containing personal and sensitive metadata, and restrict access and retention accordingly.

Every request by a verified global administrator is audited, including UI polling and admin-role automation. A view polling every three to five seconds produces roughly 720–1,200 records per hour while open; concurrent views and administrators add to that volume. Size transport and retention for your actual polling and automation load.

Available in Kamiwaza 1.4.0

• Authenticated actors: catalog audit events within authenticated bearer, personal access token, and signed platform-gateway request contexts carry the acting user (user.id). Token creation and self-revocation records name the acting owner. Administrative PAT revocation names the administrator in user.id, the affected owner in enduser.id and the token identifier in kamiwaza.audit.target.id. Unverified credentials do not establish a user; native sign-in failures can identify a previously resolved account.
• Available context: audit records include user.name, kamiwaza.principal.type, kamiwaza.auth.method, kamiwaza.auth.client_id, client.address, and user_agent.original when established by authentication. kamiwaza.auth.credential.id identifies the credential, and kamiwaza.request.id correlates events within a request; correlation is not proof of identity. A Keycloak bearer request can carry the existing Keycloak sid as session.id; personal access tokens do not reuse a login session. Core-owned session IDs and rejected credential identifiers use keyed fingerprints rather than raw bearer values. Missing fingerprint-key configuration reports a diagnostic and omits unsafe identifiers; session-end records requiring such an identifier cannot be emitted. Trace and span IDs are included when an active span exists. Caller-supplied event attributes cannot replace authenticated identity fields. This does not add actor fields to ordinary application logs or spans.

For those fingerprints, configure exactly one of KAMIWAZA_AUDIT_FINGERPRINT_KEY_FILE (a mounted Secret file) or KAMIWAZA_AUDIT_FINGERPRINT_KEY (a Secret-backed environment variable), containing at least 32 bytes after trimming surrounding whitespace. Use the same stable, per-installation key across producers; changing it breaks identifier joins, and a mounted key is read once until process restart.

The Core chart provides this key in every security profile. On first install it creates Secret kamiwaza-audit-fingerprint with a random key, reuses that key on every upgrade, and keeps the Secret on uninstall. It mounts the key into the API, scheduler, Ray and retrieval streamer containers and sets KAMIWAZA_AUDIT_FINGERPRINT_KEY_FILE. A Secret of that name that the release doesn't own is used as is. The chart generates the key only when the render can look up the cluster: helm install or helm upgrade into an existing namespace. A first helm install --create-namespace, GitOps and helm template can't, so they render no Secret; under stig Core pods then don't start until it exists, and under default Core drops those session IDs until it does (run helm upgrade once more after --create-namespace). For GitOps or helm template, create the Secret once from a file, as for the audit HMAC key in the hardened install guide ((umask 077; openssl rand -hex 32 > ./key), then kubectl -n <namespace> create secret generic <name> --from-file=key=./key and shred -u ./key), or copy the key of an existing kamiwaza-audit-fingerprint Secret to keep audit joins, and set core.security.auditFingerprint.existingSecret to its name. Under stig the key is required and core.security.auditFingerprint.enabled: false is refused; under default, false opts out.

If you already set the key yourself, through a valueFrom.secretKeyRef entry in core.scheduler.extraEnv and core.retrievalStreamer.extraEnv (no core. prefix for the standalone Core chart), or with the 1.4.0 field kit's fingerprint overlay, your entries keep working: they replace the chart's key for those workloads, so your key and its joins are kept. Set it in both lists, or the streamer and the other workloads use different keys; NOTES warn when only one list sets it. Under stig a literal key value is refused; use valueFrom or the _FILE form. To move to the chart-managed key without changing the key, set core.security.auditFingerprint.existingSecret and secretKey (security.auditFingerprint.* for the standalone Core chart) to the Secret and key your entries use, then remove the entries and any extraVolumes and extraVolumeMounts entries you added for them. The field kit's Secret is already kamiwaza-audit-fingerprint with key key, which the chart uses as is, so kit sites can skip the first step. If your Secret has another name or key, removing the entries without the first step makes the chart generate a new key.

Log Types

The platform emits several categories of logs:

Application logs

Used for API operations, business logic, and service errors.

Typical examples:

• model deployment failures
• connector execution errors
• service exceptions

Audit logs

Security-relevant records described in Audit Events.

Deployment logs

Used for model, app, and tool runtime troubleshooting.

Typical examples:

• engine startup output
• container stderr or stdout
• launch-time readiness failures

Ingress and access logs

Used for request-level routing and gateway debugging.

Typical examples:

• upstream routing errors
• unexpected status codes
• repeated auth or callback failures

Troubleshooting Common Issues

A model, app, or tool fails to start

• start with the UI log viewer for that deployment
• look for detected patterns such as out-of-memory, startup timeout, or container exit
• if the problem affects multiple deployments, check Kubernetes pod status and cluster logs

A service is reachable but behaving incorrectly

• inspect application logs for the affected service
• compare the failing request path against the expected routed URL
• verify the deployment still matches the intended configuration

Your OTLP destination is missing data

• confirm an effective global.observability.endpoint or core.scheduler.otel.endpoint is set in the release values
• confirm the core configuration has the endpoint: OTEL_EXPORTER_OTLP_ENDPOINT should be set, and OTEL_SDK_DISABLED should be absent. Pods read the value at startup, so restart them after changing it.
• remove a leftover KAMIWAZA_OTEL_ENABLED=false when relying on endpoint-based SDK enablement
• confirm protocol matches the collector port: http/protobuf for 4318, grpc for 4317
• check the Kamiwaza pod logs on stdout for exporter errors, then your collector's logs
• for Core-side catalog-audit losses, look for stdout ERROR diagnostics with event.name=kamiwaza.audit.export.failure and the kamiwaza.audit.export.failures counter, labeled by error.type. These cover losses detected by a live producer during export, queue overflow or graceful shutdown; a killed process cannot report records left in its queues. Legacy event_category: audit records and OTLP partial-success rejections are not counted. Check stdout when the telemetry path itself is unavailable.
• confirm the destination is receiving data for the correct environment, index, or tenant

Authentication or access issues need correlation

• review audit events for the user and time window
• confirm the request path, user context, and deny behavior
• use the security validation docs when the issue is tied to ReBAC or identity setup

Related Docs

• Quickstart
• Configuration Reference
• Administrator Guide