Data Catalog and Secrets

Kamiwaza provides a centralized catalog for datasets, containers, and secrets. The catalog stores metadata, supports DataHub URNs, and is protected by access controls so only authorized users can view or change assets.

In current deployments, the catalog is the standard place to reference shared data assets and secret metadata across models, connectors, applications, and tools.

Core entities

Datasets

Datasets are the primary catalog entity. They are identified by DataHub URNs and can include schema metadata.

Containers

Containers are logical groupings of datasets (for example, a project or domain). They also use DataHub URNs and support dataset membership APIs. They also map well to things like S3 buckets, file folders, etc.

Secrets

Secrets are stored as catalog entities with encrypted values. Secret metadata is accessible via the API, but secret values are never returned through HTTP endpoints.

The current catalog backend encrypts secret values with a single Fernet key. It does not provide DEK/KEK envelope encryption or encryption-key rotation. Replacing a stored secret's value does not rotate that encryption key.

URN handling (DataHub)

Some DataHub URNs include forward slashes, which can conflict with path routing. Kamiwaza supports three approaches:

V2 regex path endpoints (recommended)
Query-parameter endpoints (/by-urn)
Legacy path endpoints (simple URNs only)

Dataset endpoints

POST   /api/catalog/datasets/

GET    /api/catalog/datasets/v2/{dataset_urn}
GET    /api/catalog/datasets/v2/{dataset_urn}/schema
PATCH  /api/catalog/datasets/v2/{dataset_urn}
DELETE /api/catalog/datasets/v2/{dataset_urn}
PUT    /api/catalog/datasets/v2/{dataset_urn}/schema

GET    /api/catalog/datasets/by-urn?urn={full_urn}
GET    /api/catalog/datasets/by-urn/schema?urn={full_urn}
PATCH  /api/catalog/datasets/by-urn?urn={full_urn}
DELETE /api/catalog/datasets/by-urn?urn={full_urn}
PUT    /api/catalog/datasets/by-urn/schema?urn={full_urn}

GET    /api/catalog/datasets/{encoded_urn}
PATCH  /api/catalog/datasets/{encoded_urn}
DELETE /api/catalog/datasets/{encoded_urn}

Updating dataset metadata

PATCH is a partial update. Only the fields present in the request body change; every other field keeps its stored value. Sending only description leaves tags and properties untouched, and sending only properties leaves description untouched. Sending description, tags, or properties as null does not clear them; the field is treated as if it were absent.

properties, when sent, replaces the dataset's user-defined properties as a whole: send the complete map you want, and omit a key to remove it. Dataset gate properties (gate, gate_type, and any gate_* key) are the exception: they are kept even when you omit them, sending one back unchanged is accepted, and adding or changing one returns 400. Bind or remove a dataset gate with PUT or DELETE /api/catalog/datasets/{encoded_urn}/gate instead. These routes take the URN as a single path segment, so they cannot address a URN that contains /, even percent-encoded.

The platform reserves source_revision and every property key that starts with _kz_. A GET can return them, but a PATCH whose properties include one returns 400, even when its value is unchanged, so remove them from any properties you send back.

caution

Responses never include secret references (such as credential_secret) or an endpoint that embeds credentials, so the properties a GET returned are not always safe to send back as they are:

A dataset that no ingestion maintains. A properties replacement removes those hidden properties unless you send them again, so a dataset bound to a secret loses that binding. Sending a secret reference again requires permission to read that secret; without it, the update returns 403. To change only the description or tags, leave properties out of the request.
A dataset that an ingestion maintains (its properties include source_revision). The source fields, including secret references, are kept when you leave them out, and changing one returns 400.

Every dataset in a response carries an etag field: the entity-tag of the metadata you can see (description, tags, visible properties, and container). It changes whenever that metadata changes, including properties an ingestion maintains; properties hidden from responses do not affect it. Do not confuse it with properties.etag, which some datasets carry for the object they describe. Single-dataset GET and PATCH responses also return the tag in the ETag header.

Send the tag back in an If-Match header to make sure your update is based on the current metadata. The header takes the tag in double quotes, exactly as the ETag header returns it; the body's etag and a 412's current_etag are unquoted, so wrap them in double quotes first (an unquoted tag returns 400). The examples use API=https://kamiwaza.example.com/api and an access token in TOKEN:

URN='urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/sales,PROD)'

# Read the dataset and note its ETag
curl -si --get "$API/catalog/datasets/by-urn" --data-urlencode "urn=$URN" \
  -H "Authorization: Bearer $TOKEN" | grep -i '^etag'
# ETag: "sha256:9f2c..."

# Update only if nobody changed the metadata since
curl -X PATCH "$API/catalog/datasets/by-urn?urn=$(jq -rn --arg u "$URN" '$u | @uri')" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'If-Match: "sha256:9f2c..."' \
  -H "Content-Type: application/json" \
  --data '{"description": "Quarterly sales, cleaned"}'

Request	Result
`If-Match` lists the current `etag`	`200`, with the updated dataset and its new `ETag`
`If-Match` is stale	`412` with `{"detail": {"error": "stale_etag", "detail": "...", "current_etag": "..."}}`; nothing is written. Re-read, re-apply your change, and retry.
`If-Match` is malformed (not a quoted entity-tag)	`400` with `{"detail": {"error": "invalid_if_match", "detail": "..."}}`
No `If-Match`, or `If-Match: *`	The update applies unconditionally
Other writers kept changing the dataset during the update	`409` with `{"detail": {"error": "concurrent_update", "detail": "..."}}`; retry

If-Match is optional. Without it an update still changes only the fields it names, but it can overwrite a concurrent edit to those same fields.

In the Python SDK, pass the etag from an earlier read as it is; the SDK adds the double quotes: client.catalog.datasets.update(urn, DatasetUpdate(description="..."), if_match=dataset.etag). A stale tag raises APIError with status_code == 412.

Container endpoints

GET    /api/catalog/containers/v2/{container_urn}
PATCH  /api/catalog/containers/v2/{container_urn}
DELETE /api/catalog/containers/v2/{container_urn}
POST   /api/catalog/containers/v2/{container_urn}/datasets
DELETE /api/catalog/containers/v2/{container_urn}/datasets/{dataset_urn}

GET    /api/catalog/containers/by-urn?urn={full_urn}
PATCH  /api/catalog/containers/by-urn?urn={full_urn}
DELETE /api/catalog/containers/by-urn?urn={full_urn}

Secret endpoints

POST   /api/catalog/secrets
GET    /api/catalog/secrets
GET    /api/catalog/secrets/v2/{secret_urn}
DELETE /api/catalog/secrets/v2/{secret_urn}

GET    /api/catalog/secrets/by-urn?urn={full_urn}
DELETE /api/catalog/secrets/by-urn?urn={full_urn}

Note: Secret values are never exposed in API responses. Use secrets in connectors and deployments by reference.

Register a dataset

This example registers one CSV object in S3, or an S3-compatible store, so the Retrieval Service can read it. The credential is stored as a catalog secret and referenced by URN; it never appears in the dataset.

The examples use API=https://kamiwaza.example.com/api and an access token in TOKEN.

1. Store the credential. The secret's owner is your user URN:

OWNER=$(curl --fail --silent --show-error -H "Authorization: Bearer $TOKEN" \
  "$API/auth/users/me" | jq -er '"urn:li:corpuser:" + .sub')

VALUE=$(jq -nc --arg id "$AWS_ACCESS_KEY_ID" --arg key "$AWS_SECRET_ACCESS_KEY" \
  '{aws_access_key_id: $id, aws_secret_access_key: $key}')

SECRET_URN=$(jq -n --arg owner "$OWNER" --arg value "$VALUE" \
    '{name: "sales-s3-reader", owner: $owner, value: $value}' \
  | curl --fail --silent --show-error -X POST "$API/catalog/secrets/" \
      -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
      --data-binary @- | jq -er .)

Use the URN the call returns. A secret with the same name returns 409. A wrong owner is not rejected here; it surfaces in step 2 as 403 dataset_secret_reference_denied, because only the owner can use the secret.

2. Register the dataset. path names one object. Set region for AWS; for an S3-compatible store, also set endpoint to its URL, including scheme and port:

jq -n --arg secret "$SECRET_URN" '{
  name: "my-bucket/data/sales.csv",
  platform: "s3",
  properties: {
    path: "s3://my-bucket/data/sales.csv",
    format: "csv",
    region: "us-east-1",
    # endpoint: "https://s3.example.com:9000",   # S3-compatible stores only
    credential_secret: $secret
  }
}' | curl --fail --silent --show-error -X POST "$API/catalog/datasets/" \
      -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
      --data-binary @- | jq -r .
# urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/data/sales.csv,PROD)

format is csv, parquet, json, text, or binary. The response is the dataset URN, in the form urn:li:dataset:(urn:li:dataPlatform:<platform>,<name>,<environment>); environment defaults to PROD. You become the dataset's owner.

3. Check it. A local retrieval job returns the object's rows:

curl --fail --silent --show-error -X POST "$API/retrieval/jobs" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"dataset_urn": "urn:li:dataset:(urn:li:dataPlatform:s3,my-bucket/data/sales.csv,PROD)", "transport": "inline", "limit_rows": 10}' | jq .

Kamiwaza reaches the store from inside the cluster, so pods must resolve its hostname (see DNS) and the cluster must allow egress to it. If the store's certificate is not issued by a publicly trusted CA, add its CA to global.trust.extraAnchors; see TLS Trust for Private Certificate Authorities.

Access control

Datasets: Protected by relationship-based access control (ReBAC). Owners/editors/viewers can access dataset metadata and schema.
Containers: Container creation and membership changes require admin permissions.
Secrets: Protected by ReBAC; only authorized users can view or delete secret metadata.

Best practices

Prefer /v2/{urn} or /by-urn endpoints for any URN that might contain / characters.
Use secrets for credentials instead of embedding them in connector definitions, templates, or external endpoint payloads.
Align dataset URNs with your DataHub naming conventions so discovery, lineage, and governance are consistent across tools.

Core entities​

Datasets​

Containers​

Secrets​

URN handling (DataHub)​

Dataset endpoints​

Updating dataset metadata​

Container endpoints​

Secret endpoints​

Register a dataset​

Access control​

Best practices​