Data Catalog and Secrets

Kamiwaza provides a centralized catalog for datasets, containers, and secrets. The catalog stores metadata, supports DataHub URNs, and is protected by access controls so only authorized users can view or change assets.

In current deployments, the catalog is the standard place to reference shared data assets and secret metadata across models, connectors, applications, and tools.

Core entities

Datasets

Datasets are the primary catalog entity. They are identified by DataHub URNs and can include schema metadata.

Containers

Containers are logical groupings of datasets (for example, a project or domain). They also use DataHub URNs and support dataset membership APIs. They also map well to things like S3 buckets, file folders, etc.

Secrets

Secrets are stored as catalog entities with encrypted values. Secret metadata is accessible via the API, but secret values are never returned through HTTP endpoints.

The current catalog backend encrypts secret values with a single Fernet key. It does not provide DEK/KEK envelope encryption or encryption-key rotation. Replacing a stored secret's value does not rotate that encryption key.

URN handling (DataHub)

Some DataHub URNs include forward slashes, which can conflict with path routing. Kamiwaza supports three approaches:

1. V2 regex path endpoints (recommended)
2. Query-parameter endpoints (/by-urn)
3. Legacy path endpoints (simple URNs only)

Dataset endpoints

POST   /api/catalog/datasets/

GET    /api/catalog/datasets/v2/{dataset_urn}
GET    /api/catalog/datasets/v2/{dataset_urn}/schema
PATCH  /api/catalog/datasets/v2/{dataset_urn}
DELETE /api/catalog/datasets/v2/{dataset_urn}
PUT    /api/catalog/datasets/v2/{dataset_urn}/schema

GET    /api/catalog/datasets/by-urn?urn={full_urn}
GET    /api/catalog/datasets/by-urn/schema?urn={full_urn}
PATCH  /api/catalog/datasets/by-urn?urn={full_urn}
DELETE /api/catalog/datasets/by-urn?urn={full_urn}
PUT    /api/catalog/datasets/by-urn/schema?urn={full_urn}

GET    /api/catalog/datasets/{encoded_urn}
PATCH  /api/catalog/datasets/{encoded_urn}
DELETE /api/catalog/datasets/{encoded_urn}

Container endpoints

GET    /api/catalog/containers/v2/{container_urn}
PATCH  /api/catalog/containers/v2/{container_urn}
DELETE /api/catalog/containers/v2/{container_urn}
POST   /api/catalog/containers/v2/{container_urn}/datasets
DELETE /api/catalog/containers/v2/{container_urn}/datasets/{dataset_urn}

GET    /api/catalog/containers/by-urn?urn={full_urn}
PATCH  /api/catalog/containers/by-urn?urn={full_urn}
DELETE /api/catalog/containers/by-urn?urn={full_urn}

Secret endpoints

POST   /api/catalog/secrets
GET    /api/catalog/secrets
GET    /api/catalog/secrets/v2/{secret_urn}
DELETE /api/catalog/secrets/v2/{secret_urn}

GET    /api/catalog/secrets/by-urn?urn={full_urn}
DELETE /api/catalog/secrets/by-urn?urn={full_urn}

Note: Secret values are never exposed in API responses. Use secrets in connectors and deployments by reference.

Register a dataset

This example registers one CSV object in S3, or an S3-compatible store, so the Retrieval Service can read it. The credential is stored as a catalog secret and referenced by URN; it never appears in the dataset.

The examples use API=https://kamiwaza.example.com/api and an access token in TOKEN.

1. Store the credential. The secret's owner is your user URN:

OWNER=$(curl --fail --silent --show-error -H "Authorization: Bearer $TOKEN" \
  "$API/auth/users/me" | jq -er '"urn:li:corpuser:" + .sub')

VALUE=$(jq -nc --arg id "$AWS_ACCESS_KEY_ID" --arg key "$AWS_SECRET_ACCESS_KEY" \
  '{aws_access_key_id: $id, aws_secret_access_key: $key}')

SECRET_URN=$(jq -n --arg owner "$OWNER" --arg value "$VALUE" \
    '{name: "sales-s3-reader", owner: $owner, value: $value}' \
  | curl --fail --silent --show-error -X POST "$API/catalog/secrets/" \
      -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
      --data-binary @- | jq -er .)

Use the URN the call returns. A secret with the same name returns 409. A wrong owner is not rejected here; it surfaces in step 2 as 403 dataset_secret_reference_denied, because only the owner can use the secret.

2. Register the dataset. path names one object. Set region for AWS; for an S3-compatible store, also set endpoint to its URL, including scheme and port:

jq -n --arg secret "$SECRET_URN" '{
  name: "my-bucket/data/sales.csv",
  platform: "s3",
  properties: {
    path: "s3://my-bucket/data/sales.csv",
    format: "csv",
    region: "us-east-1",
    # endpoint: "https://s3.example.com:9000",   # S3-compatible stores only
    credential_secret: $secret
  }
}' | curl --fail --silent --show-error -X POST "$API/catalog/datasets/" \
      -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
      --data-binary @- | jq -r .
# urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/data/sales.csv,PROD)

format is csv, parquet, json, text, or binary. The response is the dataset URN, in the form urn:li:dataset:(urn:li:dataPlatform:<platform>,<name>,<environment>); environment defaults to PROD. You become the dataset's owner.

3. Check it. A local retrieval job returns the object's rows:

curl --fail --silent --show-error -X POST "$API/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"dataset_urn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/data/sales.csv,PROD)", "transport": "inline", "limit_rows": 10}' | jq .

Kamiwaza reaches the store from inside the cluster, so pods must resolve its hostname (see DNS) and the cluster must allow egress to it. If the store's certificate is not issued by a publicly trusted CA, publish its CA in kamiwaza-edge-trust as for the edge certificate.

Access control

• Datasets: Protected by relationship-based access control (ReBAC). Owners/editors/viewers can access dataset metadata and schema.
• Containers: Container creation and membership changes require admin permissions.
• Secrets: Protected by ReBAC; only authorized users can view or delete secret metadata.

Best practices

• Prefer /v2/{urn} or /by-urn endpoints for any URN that might contain / characters.
• Use secrets for credentials instead of embedding them in connector definitions, templates, or external endpoint payloads.
• Align dataset URNs with your DataHub naming conventions so discovery, lineage, and governance are consistent across tools.