AWS S3 Workroom Storage

Kamiwaza uses S3-compatible object storage for two platform workflows:

• workroom context file persistence
• Skills Library package import and download

If object storage is not configured, file-backed user workflows can fail with errors such as:

Workroom storage is not configured for Skills Library.

Use this guide when your deployment stores workroom content in external AWS S3 instead of the default in-cluster object storage.

Default SeaweedFS installs need no configuration — do not apply this page to them

By default the chart deploys SeaweedFS in the kamiwaza namespace and wires workroom storage to it. On those installs workroom storage already works, and applying this page's values without a real external-S3 requirement — or leaving them behind after one — will break it.

In particular, a core.context.objectStorage block left in your values file from an earlier external-S3 setup replaces the built-in wiring, and core then references a Secret that may not exist. The install never completes: helm upgrade --install waits on post-install jobs while core pods sit in CreateContainerConfigError: secret "core-s3" not found. See Troubleshooting for diagnosis and cleanup.

When You Need This

Configure AWS S3 if your workroom content must live in an external AWS S3 bucket and you want the following to work reliably:

• context file upload and persistence
• Skills Library package import
• Skills Library package download

If the default SeaweedFS backend meets your requirements, you do not need this page.

Prerequisites

Before you start, make sure you have:

• an AWS S3 bucket
• an IAM user or role with access to that bucket
• the installer kubeconfig and the values file from your environment's install page (kamiwaza-values.yaml in the examples)

The IAM identity should be allowed to perform at least:

• s3:ListBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject

Choose S3 Before the First Install

Changing values does not migrate existing objects. On a running install, workroom files and Skills Library packages already in SeaweedFS are not copied to S3, and the values on this page set seaweedfs.enabled: false, which makes the next upgrade delete the SeaweedFS volume and every object on it. Skills Library packages stored there stay listed, but downloading them fails. To keep that data, leave seaweedfs.enabled: true until a separately planned and verified transfer has copied it. Moving from S3 back to SeaweedFS needs the same transfer.

Configuration Model

External S3 is configured in one block of your values file, under core.context.objectStorage:

core:
  context:
    objectStorage:
      enabled: true
      defaultBucket: ""
      defaultRegion: ""
      defaultPrefix: "context/raw"
      endpointUrl: "https://s3.<region>.amazonaws.com"
      credentialsSecretRef:
        name: ""
        accessKeyIdKey: "access_key_id"
        secretAccessKeyKey: "secret_access_key"
        sessionTokenKey: "session_token"

• Set endpointUrl to your bucket's regional endpoint, https://s3.<region>.amazonaws.com, or to your S3-compatible service's endpoint. Do not leave it empty: on an install that ran with SeaweedFS, an empty value can leave core using the old SeaweedFS address.
• Set enabled: true explicitly. Without it, the ambient-credentials path (empty Secret name) does not turn object storage on.
• When this block is set, the chart uses it as written and ignores the SeaweedFS wiring. Setting seaweedfs.enabled: false alongside it removes the unused SeaweedFS deployment and volume.
• Setting storage.workrooms.backend: s3 does not configure core. Use core.context.objectStorage.

This configuration is rendered into:

• the core-config ConfigMap for non-secret S3 settings
• the core-scheduler, core-raycluster-head, and core-raycluster-worker deployments for secret-backed AWS credentials

Option 1: Static AWS Access Keys

Use this option when your cluster does not already provide AWS credentials to the pods.

1. Create a Kubernetes Secret

Core pods wait on this Secret, so create it before you install or upgrade, in the kamiwaza namespace your platform administrator created in Platform Prerequisites (Kubernetes with Istio or OpenShift with GreyMatter). Build it from files, so the keys never appear in a command line or your shell history:

printf 'AWS secret access key: '; read -rs AWS_SECRET_ACCESS_KEY; echo
printf 'AWS session token (Enter if none): '; read -rs AWS_SESSION_TOKEN; echo
(umask 077; d=$(mktemp -d)
 printf '%s' '<aws-access-key-id>' > "$d/access_key_id"
 printf '%s' "$AWS_SECRET_ACCESS_KEY" > "$d/secret_access_key"
 [ -n "$AWS_SESSION_TOKEN" ] && printf '%s' "$AWS_SESSION_TOKEN" > "$d/session_token"
 kubectl -n kamiwaza create secret generic core-s3 --from-file="$d" \
   --dry-run=client -o yaml | kubectl apply -f -
 rm -rf "$d")
unset AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN

The same command creates the Secret or replaces its keys. Core reads session_token when it is present; no values change is needed. Session credentials expire, so rotate the Secret before they do, as in Rotate credentials.

2. Add the S3 settings to your values file

Merge this into kamiwaza-values.yaml:

core:
  context:
    objectStorage:
      enabled: true
      defaultBucket: "my-kamiwaza-artifacts"
      defaultRegion: "us-west-2"
      defaultPrefix: "context/raw"
      endpointUrl: "https://s3.us-west-2.amazonaws.com"
      credentialsSecretRef:
        name: "core-s3"
        accessKeyIdKey: "access_key_id"
        secretAccessKeyKey: "secret_access_key"

seaweedfs:
  enabled: false

Notes:

• defaultBucket is required; the render fails without it.
• If your Secret uses different key names, set accessKeyIdKey and secretAccessKeyKey to match.
• endpointUrl must match the bucket's region.
• This changes workroom storage only. Model storage keeps its configured backend.

3. Apply the values

Run the helm upgrade --install command from your environment's install page, for example Install on Kubernetes with Istio, step 3, with the same chart version and the updated values file. The upgrade restarts the core deployments when their S3 settings change.

Option 2: Ambient AWS Credentials

Use this option if your cluster already provides AWS credentials to pods through IAM roles or another AWS-native mechanism.

In this case, do not create a Secret. Leave credentialsSecretRef.name empty, so no credential environment variables are rendered and the AWS SDK falls back to ambient credentials. enabled: true is required here: without it, the chart cannot tell that object storage is configured.

core:
  context:
    objectStorage:
      enabled: true
      defaultBucket: "my-kamiwaza-artifacts"
      defaultRegion: "us-west-2"
      defaultPrefix: "context/raw"
      endpointUrl: "https://s3.us-west-2.amazonaws.com"
      credentialsSecretRef:
        name: ""

seaweedfs:
  enabled: false

Apply the values as in Option 1, step 3.

Rotate credentials

To replace the keys, rerun Option 1, step 1 with the new values. The chart does not track the Secret's contents, so then restart the core deployments to make them read the new values. Confirm the Secret exists first; if it does not, the restarted core pods cannot start and the API is unavailable until it does:

kubectl -n kamiwaza get secret core-s3

kubectl -n kamiwaza rollout restart \
  deployment/core-scheduler deployment/core-raycluster-head deployment/core-raycluster-worker

Verify the Configuration

Check the ConfigMap

Verify that the non-secret S3 settings are present:

kubectl -n kamiwaza get configmap core-config -o yaml | grep CONTEXT_SERVICE_S3

You should see values like:

CONTEXT_SERVICE_S3_DEFAULT_BUCKET: my-kamiwaza-artifacts
CONTEXT_SERVICE_S3_DEFAULT_PREFIX: context/raw
CONTEXT_SERVICE_S3_DEFAULT_REGION: us-west-2
CONTEXT_SERVICE_S3_ENABLED: "true"
CONTEXT_SERVICE_S3_ENDPOINT_URL: https://s3.us-west-2.amazonaws.com

Check the Credential Reference

If you are using Secret-backed credentials, verify the rendered credential reference without echoing any secret values:

kubectl -n kamiwaza get deploy core-scheduler -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID
kubectl -n kamiwaza get deploy core-raycluster-head -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID

The secretKeyRef should name your Secret (core-s3) and key (access_key_id).

Validate in the Product

Import a Skills Library package, then download it. Restart the core deployments as in Rotate credentials and download it again; the package should still be there. Its objects appear in the bucket under the defaultPrefix.

Troubleshooting

Error: secret "core-s3" not found (CreateContainerConfigError)

Core pods (core-scheduler, core-raycluster-head, core-raycluster-worker) are stuck in CreateContainerConfigError (shown as 0/1, or 1/2 on Istio-injected installs), the pod events show secret "core-s3" not found, and the install never completes: helm upgrade --install waits on post-install jobs while most other pods run normally.

This means core was configured to read S3 credentials from a Secret that does not exist in the kamiwaza namespace. It usually means a core.context.objectStorage block left in your values file from an earlier external-S3 setup is overriding the default SeaweedFS wiring.

Check which Secret core references:

kubectl -n kamiwaza get deploy core-scheduler -o yaml | grep -A4 CONTEXT_SERVICE_S3_ACCESS_KEY_ID

• name: core-s3 — this page's external-S3 settings are in effect
• name: seaweedfs-s3-credentials — the default wiring is in effect

Fix:

• If you intended the default SeaweedFS storage: remove the core.context.objectStorage block from your values file, set seaweedfs.enabled back to true if you changed it, and rerun helm upgrade --install.
• If you intended external S3: create the Secret (Option 1, step 1). The kubelet retries automatically and the pods recover within a few minutes (container-creation backoff caps at 5 minutes). To retry immediately, restart the deployments as in Rotate credentials.

Error: Workroom storage is not configured for Skills Library

This usually means core does not see CONTEXT_SERVICE_S3_DEFAULT_BUCKET.

Check:

• the block is nested under core: — core.context.objectStorage, not top-level context.objectStorage. helm get values kamiwaza -n kamiwaza shows what Helm received.
• enabled: true is set
• the core-config ConfigMap contains CONTEXT_SERVICE_S3_DEFAULT_BUCKET

Error: Unable to locate credentials

This means the bucket settings are present but AWS authentication is not.

Check:

• the Secret exists in the kamiwaza namespace
• credentialsSecretRef.name matches the Secret name
• the Secret keys match the configured key names
• the core deployments were restarted after the Secret was created or changed

Error: AccessDenied or SignatureDoesNotMatch

This usually means the credentials are present but not valid for the target bucket or region.

Check:

• the bucket name
• the AWS region
• the IAM policy attached to the user or role
• whether the credentials belong to the AWS account that owns the bucket

Security Notes

• Do not store AWS secrets in workroom attributes or user-editable metadata.
• Prefer Kubernetes Secrets or ambient AWS identity over plain-text credentials in values files.
• Use an https:// endpoint in endpointUrl.