Skip to main content
Version: 0.9.0

Administrator Guide

1. Authentication & Access Control​

Kamiwaza provides enterprise-grade authentication built on Keycloak with OpenID Connect (OIDC) and JWT token validation.

1.1 Authentication Architecture​

User → Keycloak (IdP) → JWT Token → Traefik → ForwardAuth → API Services
                                       ↓
                                  [Validated] → Access Granted
                                       ↓
                                  [Rejected] → 401/403 Error

Components:

  • Keycloak: Identity provider managing users, authentication, and token issuance
  • ForwardAuth Service: Validates JWT tokens and enforces access policies
  • Traefik: Reverse proxy routing requests through ForwardAuth middleware
  • RBAC Policy Engine: YAML-based endpoint access control

1.2 Authentication Modes​

Kamiwaza supports two operational modes:

ModeUse CaseConfiguration
With AuthenticationProduction, staging, secure environmentsKAMIWAZA_USE_AUTH=true
Bypass ModeLocal development, debuggingKAMIWAZA_USE_AUTH=false

To enable authentication:

# In env.sh or environment
export KAMIWAZA_USE_AUTH=true
bash startup/kamiwazad.sh restart

Expected output:

Stopping kamiwazad ...
Starting kamiwazad ...
kamiwazad status: active (running)

⚠️ Warning: Bypass mode (KAMIWAZA_USE_AUTH=false) disables all authentication. Use only in secure development environments.

1.3 Token-Based Authentication​

Kamiwaza uses RS256 JWT tokens with asymmetric cryptographic signatures.

Token Lifecycle:

  1. Acquisition: User authenticates with Keycloak via username/password or SSO
  2. Validation: ForwardAuth validates token signature against JWKS endpoint
  3. Authorization: User roles checked against RBAC policy
  4. Expiration: Access tokens expire (default: 1 hour), require refresh
  5. Revocation: Logout invalidates tokens

Token Delivery Methods:

  • HTTP Authorization: Bearer <token> header (recommended for APIs)
  • Secure HTTP-only cookie (automatic for browser sessions)

2. User Management​

2.1 Accessing Keycloak Admin Console​

Default Credentials (change immediately in production):

  • URL: http://localhost:8080 (or your configured Keycloak URL)
  • Username: admin
  • Password: Set via KEYCLOAK_ADMIN_PASSWORD environment variable

Production Setup:

# Set secure admin password in env.sh
export KEYCLOAK_ADMIN_PASSWORD="<strong-random-password>"

2.2 Creating User Accounts​

Via Keycloak Admin Console:

  1. Navigate to Users in left sidebar
  2. Click Add User
  3. Fill in required fields:
    • Username (required)
    • Email (required for password reset)
    • First Name / Last Name (optional)
  4. Toggle Email Verified to ON
  5. Click Save
  6. Go to Credentials tab
  7. Set temporary or permanent password
  8. Assign roles (see Role Management below)

Pre-configured Test Users:

UsernamePasswordRolesUse Case
testusertestpassviewerRead-only testing
testadmintestpassadminAdministrative testing

⚠️ Important: Remove or secure test users before production deployment.

2.3 User Roles and Permissions​

Kamiwaza defines three primary roles:

RolePermissionsTypical Users
adminFull access: read, write, delete, configureSystem administrators, platform operators
userStandard access: read, write (no delete/admin)Data scientists, developers, analysts
viewerRead-only accessAuditors, observers, stakeholders

Assigning Roles:

  1. Navigate to Users → Select user
  2. Go to Role Mappings tab
  3. Under Realm Roles, select appropriate roles
  4. Click Add selected
  5. Changes take effect immediately (no logout required)

2.4 Password Policies​

Configuring Password Requirements:

  1. Navigate to Realm Settings → Security Defenses → Password Policy
  2. Add policies:
    • Minimum Length: 12 characters (recommended)
    • Uppercase Characters: Require at least 1
    • Lowercase Characters: Require at least 1
    • Digits: Require at least 1
    • Special Characters: Require at least 1
    • Not Username: Prevent username as password
    • Password History: Prevent last 3 passwords
    • Expire Password: 90 days (recommended)

Password Reset Flow:

  1. User clicks "Forgot Password" on login page
  2. Keycloak sends password reset email
  3. User follows link and sets new password
  4. New password must meet policy requirements

⚠️ Important: Configure SMTP settings in Keycloak for email-based password reset to function.

3. Role-Based Access Control (RBAC)​

3.1 RBAC Policy File​

Access control is defined in YAML policy files that map endpoints to required roles.

Default Location:

  • Host installs: $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml
  • Docker installs: Mounted at /app/config/auth_gateway_policy.yaml

ForwardAuth is stateless—set AUTH_GATEWAY_POLICY_FILE=$KAMIWAZA_ROOT/config/auth_gateway_policy.yaml (or the mounted path) so every restart reloads the same policy file.

Policy File Structure (default config/auth_gateway_policy.yaml):

version: 1
env: dev
default_deny: true

roles:
  - id: admin
    description: "Full system access"
  - id: user
    description: "Standard user access"
  - id: viewer
    description: "Read-only access"
  - id: guest
    description: "Minimal guest access"

endpoints:
  # Health checks
  - path: "/health"
    methods: ["GET"]
    roles: ["*"]
  - path: "/api/health"
    methods: ["GET"]
    roles: ["*"]

  # Auth endpoints (login/logout)
  - path: "/auth/login"
    methods: ["POST"]
    roles: ["*"]
  - path: "/auth/logout"
    methods: ["POST"]
    roles: ["*"]

  # Who am I
  - path: "/api/whoami"
    methods: ["GET"]
    roles: ["admin", "user", "viewer", "guest"]

  # Models
  - path: "/api/models*"
    methods: ["GET"]
    roles: ["admin", "user", "viewer"]
  - path: "/api/models*"
    methods: ["POST", "PUT", "DELETE"]
    roles: ["admin", "user"]

  # Serving deployments
  - path: "/api/serving/deployments*"
    methods: ["GET"]
    roles: ["admin", "user", "viewer"]
  - path: "/api/serving/deployments*"
    methods: ["POST", "PUT", "DELETE"]
    roles: ["admin", "user"]

  # Admin-only APIs
  - path: "/api/cluster*"
    methods: ["*"]
    roles: ["admin"]
  - path: "/api/activity*"
    methods: ["*"]
    roles: ["admin"]

  # Garden apps + tools
  - path: "/api/apps*"
    methods: ["GET"]
    roles: ["admin", "user", "viewer"]
  - path: "/api/apps*"
    methods: ["POST", "PUT", "DELETE"]
    roles: ["admin", "user"]
  - path: "/api/tools*"
    methods: ["GET"]
    roles: ["admin", "user", "viewer"]
  - path: "/api/tools*"
    methods: ["POST", "PUT", "DELETE"]
    roles: ["admin", "user"]

  # Data Discovery Engine
  - path: "/api/dde/status"
    methods: ["GET"]
    roles: ["viewer", "admin"]
  - path: "/api/dde/search"
    methods: ["POST"]
    roles: ["viewer", "admin"]
  - path: "/api/dde/reindex"
    methods: ["POST"]
    roles: ["admin"]

  # Static assets
  - path: "/static/*"
    methods: ["GET"]
    roles: ["admin", "user", "viewer", "guest"]
  - path: "/assets/*"
    methods: ["GET"]
    roles: ["admin", "user", "viewer", "guest"]
  - path: "/favicon.ico"
    methods: ["GET"]
    roles: ["admin", "user", "viewer", "guest"]
  - path: "/manifest.json"
    methods: ["GET"]
    roles: ["admin", "user", "viewer", "guest"]

3.2 Path Matching Rules​

Wildcard Patterns:

  • * matches zero or more characters within a path segment
  • ** matches across multiple path segments
  • Patterns are case-sensitive

Examples:

  • /api/models* matches /api/models, /api/models/123, /api/models/search
  • /api/*/health matches /api/models/health, /api/cluster/health
  • /api/** matches all paths under /api/

3.3 Relationship-Based Access Control (ReBAC)​

Roles gate entire endpoints, while ReBAC expresses who can act on a specific resource (model, dataset, container, etc.). When ReBAC is enabled:

  1. Turn on the feature flags – set AUTH_REBAC_ENABLED=true, AUTH_REBAC_DEFAULT_TENANT_ID, and PAT tagging variables as described in the ReBAC Deployment Guide.
  2. Bootstrap tenant tuples – run
    python scripts/rebac_tenant.py bootstrap configs/rebac/tenants/__default__.yaml
    ```<br/>
    This seeds owner/editor/clearance relationships for every default resource.
  3. Share resources by relationship – edit the tenant manifest and reapply it. Example snippet (configs/rebac/tenants/__default__.yaml):
    relationships:
      - subject: user:testuser
        relation: viewer
        object: model:catalog-sdk
    ```<br/>
    Save the file, preview the change with `python scripts/rebac_tenant.py plan configs/rebac/tenants/__default__.yaml`, then apply it with the same `bootstrap` command. The CLI ensures tuples are deduplicated and can target any tenant with `--tenant <id>`.
  4. Validate the experience – follow the ReBAC Validation Checklist to exercise both allow and deny flows from the SDK/UI. The checklist calls out the expected log messages and API responses so you can sign off without digging into tuples manually.

At a glance: every tuple stored in the relationship service takes the form subject --(relation)--> object. Common relations:

RelationDescriptionExample
ownerFull control over the resourceuser:testadmin owner model:demo-llm
editorUpdate/delete rights without being the original ownerrole:user editor dataset:sales-ingest
viewerRead-only accessuser:testuser viewer container:govdocs

Once a tuple exists, the UI/API automatically enforces it—no redeploy or restart required.

3.3 Hot Reload (No Restart Required)​

The RBAC policy file is automatically reloaded when modified:

  1. Edit auth_gateway_policy.yaml
  2. Save the file
  3. Changes take effect within seconds
  4. Monitor logs for reload confirmation: 
    INFO: Policy reloaded successfully from /app/config/auth_gateway_policy.yaml

⚠️ Important: Invalid YAML syntax will prevent reload and retain the previous valid configuration.

3.4 Adding Custom Endpoints​

Example: Protecting a new analytics endpoint

endpoints:
  # Add new analytics endpoint
  - path: "/api/analytics/reports*"
    methods: ["GET"]
    roles: ["user", "admin"]

  - path: "/api/analytics/reports*"
    methods: ["POST", "DELETE"]
    roles: ["admin"]

Testing Access Control:

# Get token for viewer role (should be denied POST)
VIEWER_TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \
  -d "grant_type=password" \
  -d "client_id=kamiwaza-platform" \
  -d "username=testuser" \
  -d "password=testpass" | jq -r .access_token)

# Test (expect 403 Forbidden)
curl -H "Authorization: Bearer $VIEWER_TOKEN" \
  -X POST http://localhost:7777/api/analytics/reports

# Get token for admin role (should succeed)
ADMIN_TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \
  -d "grant_type=password" \
  -d "client_id=kamiwaza-platform" \
  -d "username=testadmin" \
  -d "password=testpass" | jq -r .access_token)

# Test (expect 200 OK)
curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  -X POST http://localhost:7777/api/analytics/reports

4. Identity Provider Integration​

4.1 Keycloak Configuration​

Realm: kamiwaza Client ID: kamiwaza-platform

Client Configuration Settings:

SettingValuePurpose
Access TypePublic (SPA) or Confidential (backend)Authentication flow type
Valid Redirect URIshttps://your-domain.com/*Allowed OAuth callback URLs
Web Originshttps://your-domain.comCORS configuration
Direct Access GrantsEnabled (dev), Disabled (prod)Password grant for testing

4.2 OAuth 2.0 / OpenID Connect Integration​

Kamiwaza supports standard OIDC authentication flows.

Environment Configuration:

# Keycloak OIDC Settings
AUTH_GATEWAY_KEYCLOAK_URL=https://auth.yourdomain.com
AUTH_GATEWAY_KEYCLOAK_REALM=kamiwaza
AUTH_GATEWAY_KEYCLOAK_CLIENT_ID=kamiwaza-platform

# JWT Validation
AUTH_GATEWAY_JWT_ISSUER=https://auth.yourdomain.com/realms/kamiwaza
AUTH_GATEWAY_JWT_AUDIENCE=kamiwaza-platform
AUTH_GATEWAY_JWKS_URL=https://auth.yourdomain.com/realms/kamiwaza/protocol/openid-connect/certs

OIDC Discovery Endpoint:

https://auth.yourdomain.com/realms/kamiwaza/.well-known/openid-configuration

4.3 SAML Integration​

Configure SAML Identity Provider in Keycloak:

  1. Navigate to Identity Providers in Keycloak admin console
  2. Select SAML v2.0
  3. Configure SAML settings:
    • Single Sign-On Service URL: Your IdP's SSO endpoint
    • Single Logout Service URL: Your IdP's logout endpoint
    • NameID Policy Format: urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress
    • Principal Type: Subject NameID
  4. Upload IdP metadata XML or configure manually
  5. Map SAML attributes to Keycloak user attributes
  6. Enable identity provider in login flow

Attribute Mapping Example:

SAML Attribute        → Keycloak Attribute
-----------------       -------------------
email                 → email
firstName             → firstName
lastName              → lastName
memberOf              → roles

4.4 LDAP / Active Directory Integration​

Configure LDAP Federation:

  1. Navigate to User Federation → Add provider → ldap
  2. Configure connection settings:
    • Connection URL: ldap://ldap.company.com:389 or ldaps:// for SSL
    • Bind DN: cn=admin,dc=company,dc=com
    • Bind Credential: LDAP admin password
  3. Configure LDAP search settings:
    • Users DN: ou=users,dc=company,dc=com
    • User Object Classes: inetOrgPerson, organizationalPerson
    • Username LDAP attribute: uid or sAMAccountName (AD)
    • RDN LDAP attribute: uid or cn
    • UUID LDAP attribute: entryUUID or objectGUID (AD)
  4. Save and test connection
  5. Synchronize users: Synchronize all users button

Active Directory Specific Settings:

  • Vendor: Active Directory
  • Username LDAP attribute: sAMAccountName
  • RDN LDAP attribute: cn
  • UUID LDAP attribute: objectGUID
  • User Object Classes: person, organizationalPerson, user

Role Mapping from LDAP Groups:

  1. Go to Mappers tab in LDAP federation
  2. Create new mapper: group-ldap-mapper
    • Mapper Type: group-ldap-mapper
    • LDAP Groups DN: ou=groups,dc=company,dc=com
    • Group Name LDAP Attribute: cn
    • Group Object Classes: groupOfNames
    • Membership LDAP Attribute: member
    • Mode: READ_ONLY or LDAP_ONLY
  3. Map LDAP groups to Keycloak roles in Role Mappings

4.5 Single Sign-On (SSO) Setup​

Google SSO Integration:

  1. Create OAuth 2.0 credentials in Google Cloud Console
  2. Configure authorized redirect URI: 
    https://auth.yourdomain.com/realms/kamiwaza/broker/google/endpoint
  3. In Keycloak, navigate to Identity Providers → Google
  4. Enter Client ID and Client Secret from Google Console
  5. Save and enable

Environment Configuration:

# Google SSO
GOOGLE_CLIENT_ID=your-google-client-id
GOOGLE_CLIENT_SECRET=your-google-client-secret

Microsoft Azure AD / Office 365:

  1. Register application in Azure Portal
  2. Configure redirect URI: https://auth.yourdomain.com/realms/kamiwaza/broker/oidc/endpoint
  3. In Keycloak, add OpenID Connect v1.0 provider
  4. Configure with Azure AD settings:
    • Authorization URL: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/authorize
    • Token URL: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
    • Client ID: Azure application ID
    • Client Secret: Azure client secret

Testing SSO:

  1. Navigate to Kamiwaza login page
  2. Click SSO provider button (Google, Azure, etc.)
  3. Authenticate with external identity provider
  4. First-time users automatically create Keycloak account
  5. Subsequent logins use existing account

5. Security Configuration​

5.1 JWT Token Configuration​

Token Security Settings:

# JWT Validation (in env.sh)
AUTH_GATEWAY_JWT_AUDIENCE=kamiwaza-platform  # Required audience claim
AUTH_GATEWAY_JWT_ISSUER=https://auth.yourdomain.com/realms/kamiwaza
AUTH_GATEWAY_JWKS_URL=https://auth.yourdomain.com/realms/kamiwaza/protocol/openid-connect/certs

# Security Hardening
AUTH_REQUIRE_SUB=true  # Require 'sub' claim (user ID) in tokens
AUTH_EXPOSE_TOKEN_HEADER=false  # Don't expose tokens in response headers (production)
AUTH_ALLOW_UNSIGNED_STATE=false  # Require signed OIDC state parameter (production)

Token Algorithms:

  • Supported: RS256 (RSA with SHA-256) - asymmetric cryptography
  • Not Supported: HS256, ES256, or other algorithms

5.2 Session Management​

Access Token Expiration:

Configure in Keycloak: Realm Settings → Tokens

  • Access Token Lifespan: 1 hour (default), 5-15 minutes (high security)
  • Refresh Token Lifespan: 30 days (default)
  • SSO Session Idle: 30 minutes
  • SSO Session Max: 10 hours

Session Timeout Configuration:

# In env.sh
AUTH_GATEWAY_TOKEN_LEEWAY=30  # Clock skew tolerance (seconds)
AUTH_GATEWAY_JWKS_CACHE_TTL=300  # JWKS cache duration (5 minutes)

Best Practices:

  • Short-lived access tokens (5-15 minutes) for high-security environments
  • Longer refresh tokens (days) for user convenience
  • Implement token refresh in client applications
  • Use secure, HTTP-only cookies for browser sessions

5.3 HTTPS Enforcement​

Production HTTPS Requirements:

Kamiwaza enforces HTTPS in production and CI environments when CI=true or KAMIWAZA_ENV=production.

TLS Configuration:

  1. Obtain SSL/TLS certificates (Let's Encrypt, commercial CA, etc.)
  2. Configure Traefik with TLS:
# traefik-dynamic.yml
tls:
  certificates:
    - certFile: /certs/your-domain.crt
      keyFile: /certs/your-domain.key
  options:
    default:
      minVersion: VersionTLS12
      cipherSuites:
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  1. Update environment:
AUTH_GATEWAY_KEYCLOAK_URL=https://auth.yourdomain.com
KAMIWAZA_HTTPS=true

5.4 Rate Limiting (Optional - Requires Redis)​

Rate limiting requires Redis configuration:

# Redis connection for rate limiting
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0

Rate Limit Configuration:

# In auth_gateway_policy.yaml
rate_limits:
  - path: "/api/models*"
    requests_per_minute: 100
    per_user: true

  - path: "/api/auth/token"
    requests_per_minute: 10
    per_ip: true

5.5 Ephemeral Sessions for App Garden​

Ephemeral sessions automatically clean up App Garden deployments when users log out or their session expires. This prevents orphaned containers and ensures sensitive workloads don't persist beyond the user's session.

Understanding Ephemeral Deployments​

When a user deploys an application from App Garden with "Ephemeral session" enabled:

  1. The deployment is tied to the user's authentication session
  2. When the user logs out or their session expires, the deployment is automatically purged
  3. All containers, data, and resources associated with that deployment are cleaned up
  4. On Kamiwaza core restart, any orphaned ephemeral deployments are also purged

This is ideal for:

  • Demo environments where users should not leave resources running
  • Sensitive workloads that must not persist beyond a session
  • Multi-tenant environments where cleanup is critical
  • Development/testing scenarios

Configuration Options​

Default Behavior (Per-Deployment Choice):

By default, users can choose whether each deployment is ephemeral via a checkbox in the App Garden deploy modal:

# Default behavior - users choose per deployment
# No environment variable needed

Force All Deployments to be Ephemeral:

For environments where all App Garden deployments must be ephemeral (e.g., demo servers, shared labs), set:

# In env.sh or environment
export KAMIWAZA_EPHEMERAL_EXTENSIONS=true

When enabled:

  • All App Garden deployments are automatically ephemeral
  • The "Ephemeral session" checkbox is checked and disabled in the UI
  • Users cannot create non-ephemeral deployments
  • The API ignores any is_ephemeral_session=false requests

Configure Default Value (Without Forcing):

To change the default checkbox state without forcing ephemeral mode:

# In env.sh - sets default to ephemeral, but users can uncheck
export KAMIWAZA_APP_SESSION_EPHEMERAL_DEFAULT=true

Environment Variable Reference​

VariableDescriptionDefault
KAMIWAZA_EPHEMERAL_EXTENSIONSForce all App Garden deployments to be ephemeralfalse
KAMIWAZA_APP_SESSION_EPHEMERAL_DEFAULTDefault value for ephemeral checkbox when not forcedfalse

Verifying Ephemeral Session Compliance​

Check Current Configuration:

# Verify environment variable is set
echo $KAMIWAZA_EPHEMERAL_EXTENSIONS

# Check via API
curl -s https://localhost/api/v1/apps/config/ephemeral_forced | jq

Expected response when forced:

{"ephemeral_forced": true}

Verify Deployment is Ephemeral:

# List deployments and check is_ephemeral_session field
curl -s https://localhost/api/v1/apps/deployments | jq '.[].is_ephemeral_session'

All values should be true when KAMIWAZA_EPHEMERAL_EXTENSIONS=true.

Verify Cleanup on Logout:

  1. Deploy an ephemeral app from App Garden
  2. Note the container name: docker ps | grep kamiwaza-app
  3. Log out of Kamiwaza
  4. Verify container was removed: docker ps | grep kamiwaza-app

Verify Cleanup on Core Restart:

  1. Deploy an ephemeral app
  2. Restart Kamiwaza core: bash startup/kamiwazad.sh restart-core
  3. Check logs for cleanup: grep "ephemeral" $KAMIWAZA_LOG_DIR/kamiwaza.log
  4. Verify deployment was purged in UI or via API

Audit Ephemeral Deployments:

# Check for any non-ephemeral deployments (compliance check)
curl -s https://localhost/api/v1/apps/deployments | \
  jq '.[] | select(.is_ephemeral_session == false) | {id, name, is_ephemeral_session}'

If KAMIWAZA_EPHEMERAL_EXTENSIONS=true is set, this should return empty results.

Troubleshooting Ephemeral Sessions​

Issue: Deployments not being cleaned up on logout

  1. Check that the deployment was created as ephemeral:

    curl -s https://localhost/api/v1/apps/deployments/{id} | jq '.is_ephemeral_session'

  2. Verify the session cleanup service is running:

    grep "ephemeral cleanup" $KAMIWAZA_LOG_DIR/kamiwaza.log

  3. Check for errors during cleanup:

    grep -i "cleanup failed\|purge failed" $KAMIWAZA_LOG_DIR/kamiwaza.log

Issue: Checkbox not disabled when forcing ephemeral mode

  1. Verify environment variable is set and exported:

    env | grep KAMIWAZA_EPHEMERAL

  2. Restart Kamiwaza core to pick up the change:

    bash startup/kamiwazad.sh restart-core

  3. Hard refresh the browser (Ctrl+Shift+R or Cmd+Shift+R)

6. Monitoring & Troubleshooting​

6.1 Health Checks​

Auth Service Health Endpoint:

curl http://localhost:7777/health

Response:

{
  "status": "healthy",
  "version": "1.0.0",
  "uptime": 3600.5,
  "KAMIWAZA_USE_AUTH": true,
  "jwks_cache_status": "healthy"
}

Keycloak Health Check:

curl http://localhost:8080/health/ready

Response:

{"status":"UP"}

6.2 Log Monitoring​

Refer to the Observability Guide for end-to-end logging, OTEL, and SIEM integration. It covers how to tail auth logs, forward them to your enterprise collectors, and verify that allow/deny events appear in the standard dashboards.

6.3 Common Issues and Solutions​

Issue: 401 Unauthorized on All Requests​

Symptoms: All API requests return 401 even with valid tokens

Troubleshooting:

  1. Check if auth is enabled:

    echo $KAMIWAZA_USE_AUTH  # Should be 'true'

  2. Verify Keycloak is running:

    docker ps | grep keycloak
    curl http://localhost:8080/health/ready

    Expected docker ps output:

    default_kamiwaza-keycloak-web   Up 2 minutes (healthy)   0.0.0.0:8080->8080/tcp

  3. Check JWT issuer matches:

    # Decode your token
    echo $TOKEN | cut -d. -f2 | base64 -d | jq .iss

    # Compare with configuration
    echo $AUTH_GATEWAY_JWT_ISSUER

  4. Verify JWKS endpoint is accessible:

    curl $AUTH_GATEWAY_JWKS_URL

Solution:

  • Ensure AUTH_GATEWAY_JWT_ISSUER matches token issuer exactly
  • Verify Keycloak realm name is correct
  • Check network connectivity to Keycloak

Issue: 403 Forbidden (Valid Token)​

Symptoms: Token is valid but access denied

Troubleshooting:

  1. Check user roles in token:

    echo $TOKEN | cut -d. -f2 | base64 -d | jq .realm_access.roles

  2. Verify RBAC policy allows access:

    cat $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

  3. Check policy file syntax:

    # Invalid YAML prevents policy reload
    yamllint $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

Solution:

  • Add required roles to user in Keycloak
  • Update RBAC policy to allow endpoint/method/role combination
  • Fix YAML syntax errors and reload policy

Issue: Token Expired Too Quickly​

Symptoms: Tokens expire after minutes instead of expected duration

Troubleshooting:

  1. Check token lifespan in Keycloak:

    • Navigate to Realm Settings → Tokens
    • Verify Access Token Lifespan setting

  2. Check token claims:

    echo $TOKEN | cut -d. -f2 | base64 -d | jq '.exp - .iat'
    # Result is token lifetime in seconds

Solution:

  • Increase Access Token Lifespan in Keycloak (for development)
  • Implement token refresh in client applications
  • Use refresh tokens for long-lived sessions

Issue: Google/SSO Login Not Working​

Symptoms: SSO redirect fails or returns error

Troubleshooting:

  1. Check redirect URI configuration:

    • Verify redirect URI in Google/Azure console matches Keycloak exactly
    • Format: https://auth.yourdomain.com/realms/kamiwaza/broker/{provider}/endpoint

  2. Verify client secret is set:

    echo $GOOGLE_CLIENT_SECRET  # Should not be empty

  3. Check Keycloak identity provider logs:

    docker logs kamiwaza-keycloak -f | grep -i broker

Solution:

  • Update authorized redirect URIs in OAuth provider console
  • Ensure client secret is configured in Keycloak
  • Enable identity provider in Keycloak authentication flow

6.4 Diagnostic Commands​

Test Token Generation:

# Get token from Keycloak
TOKEN=$(curl -s -X POST http://localhost:8080/realms/kamiwaza/protocol/openid-connect/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=password" \
  -d "client_id=kamiwaza-platform" \
  -d "username=testuser" \
  -d "password=testpass" | jq -r .access_token)

# Decode token to inspect claims
echo $TOKEN | cut -d. -f2 | base64 -d | jq .

Test Token Validation:

# Test ForwardAuth validation endpoint directly
curl -v -H "Authorization: Bearer $TOKEN" \
  -H "X-Forwarded-Uri: /api/models" \
  -H "X-Forwarded-Method: GET" \
  http://localhost:7777/auth/validate

Verify JWKS Endpoint:

# Fetch public keys for signature validation
curl http://localhost:8080/realms/kamiwaza/protocol/openid-connect/certs | jq .

Expected response:

{
  "keys": [
    {
      "kid": "example-kid",
      "kty": "RSA",
      "alg": "RS256",
      "use": "sig"
    }
  ]
}

Check RBAC Policy:

# View current policy
cat $KAMIWAZA_ROOT/config/auth_gateway_policy.yaml

# Watch for policy reload events
tail -f $KAMIWAZA_LOG_DIR/kamiwaza.log | grep POLICY_RELOADED

Verify Keycloak login flows​

Use these checks after configuring SAML/OIDC to confirm the gateway and Keycloak agree on redirect URIs and credentials.

  1. OIDC loop 
    curl -I https://<gateway-host>/api/auth/login
    Expected result: HTTP 302 redirecting to https://<keycloak-host>/realms/<realm>/protocol/openid-connect/auth. After signing in, call: 
    curl -H "Authorization: Bearer $TOKEN" https://<gateway-host>/api/whoami
    Expected response: HTTP 200 JSON containing user_id, tenant_id, and roles.
  2. SAML provider
    • In Keycloak, open Identity Providers → SAML → Actions → Test.
    • Complete the upstream login.
    • Expected result: Keycloak displays Successfully authenticated and shows the mapped attributes (email, first name, last name). If the test fails, confirm the SAML IdP metadata matches the Keycloak binding URLs listed earlier in this guide.

The same curl commands can be scripted in CI to ensure future changes do not break either flow.

Appendix A: Environment Variable Reference​

Core Authentication​

VariableDescriptionDefaultRequired
KAMIWAZA_USE_AUTHEnable/disable authenticationtrueNo
AUTH_GATEWAY_JWT_ISSUERExpected JWT issuer URL-Yes
AUTH_GATEWAY_JWT_AUDIENCEExpected JWT audience claim-Recommended
AUTH_GATEWAY_JWKS_URLJWKS endpoint for key fetching-Yes
AUTH_GATEWAY_POLICY_FILEPath to RBAC policy file$KAMIWAZA_ROOT/config/auth_gateway_policy.yamlNo

Keycloak Configuration​

VariableDescriptionDefaultRequired
AUTH_GATEWAY_KEYCLOAK_URLKeycloak base URLhttp://localhost:8080Yes
AUTH_GATEWAY_KEYCLOAK_REALMKeycloak realm namekamiwazaYes
AUTH_GATEWAY_KEYCLOAK_CLIENT_IDOAuth client IDkamiwaza-platformYes
KEYCLOAK_ADMIN_PASSWORDKeycloak admin passwordadminYes

Security Hardening​

VariableDescriptionDefaultRequired
AUTH_REQUIRE_SUBRequire 'sub' claim in tokensfalseNo
AUTH_EXPOSE_TOKEN_HEADERExpose token in response headerstrueNo
AUTH_ALLOW_UNSIGNED_STATEAllow unsigned OIDC statetrue (dev only)No
AUTH_GATEWAY_TOKEN_LEEWAYClock skew tolerance (seconds)30No
AUTH_GATEWAY_JWKS_CACHE_TTLJWKS cache duration (seconds)300No

External Identity Providers​

VariableDescriptionDefaultRequired
GOOGLE_CLIENT_IDGoogle OAuth client ID-For Google SSO
GOOGLE_CLIENT_SECRETGoogle OAuth client secret-For Google SSO

Appendix B: RBAC Policy Examples​

Example 1: Tiered Access by Service​

version: 1
env: production
default_deny: true

roles:
  - id: admin
    description: "System administrators"
  - id: data_scientist
    description: "Data scientists and ML engineers"
  - id: analyst
    description: "Business analysts and viewers"

endpoints:
  # Model Management - Scientists can create/edit, analysts read-only
  - path: "/api/models*"
    methods: ["GET"]
    roles: ["admin", "data_scientist", "analyst"]

  - path: "/api/models*"
    methods: ["POST", "PUT", "DELETE"]
    roles: ["admin", "data_scientist"]

  # Model Serving - Scientists can deploy, analysts can query
  - path: "/api/serving/deployments*"
    methods: ["GET"]
    roles: ["admin", "data_scientist", "analyst"]

  - path: "/api/serving/deploy"
    methods: ["POST"]
    roles: ["admin", "data_scientist"]

  - path: "/api/serving/generate"
    methods: ["POST"]
    roles: ["admin", "data_scientist", "analyst"]

  # Cluster Management - Admin-only
  - path: "/api/cluster*"
    methods: ["*"]
    roles: ["admin"]

  # Public endpoints
  - path: "/health"
    methods: ["GET"]
    roles: ["*"]

Example 2: Read-Write Separation​

version: 1
env: production
default_deny: true

roles:
  - id: admin
  - id: editor
  - id: reader

endpoints:
  # Read endpoints - All authenticated users
  - path: "/api/models"
    methods: ["GET"]
    roles: ["admin", "editor", "reader"]

  - path: "/api/vectordb/collections"
    methods: ["GET"]
    roles: ["admin", "editor", "reader"]

  # Write endpoints - Editors and admins only
  - path: "/api/models"
    methods: ["POST", "PUT"]
    roles: ["admin", "editor"]

  - path: "/api/vectordb/collections"
    methods: ["POST", "PUT"]
    roles: ["admin", "editor"]

  # Delete endpoints - Admins only
  - path: "/api/models*"
    methods: ["DELETE"]
    roles: ["admin"]

  - path: "/api/vectordb/collections*"
    methods: ["DELETE"]
    roles: ["admin"]