Catalog Service

Overview

The Catalog service exposes three focused clients that mirror the server-side routers:

Attribute	Description
client.catalog.datasets	Dataset CRUD, schema management, and URN helpers
client.catalog.containers	Container lifecycle and dataset membership
client.catalog.secrets	Secret registration/listing tied to corpuser owners

Each client accepts/returns the Pydantic models defined in kamiwaza_sdk/schemas/catalog.py, so responses are thread-safe and easy to validate.

Datasets

from kamiwaza_sdk import KamiwazaClient
from kamiwaza_sdk.schemas.catalog import DatasetCreate, DatasetUpdate

client = KamiwazaClient("https://localhost/api", api_key="...shh...")

# Create and immediately fetch the dataset metadata
payload = DatasetCreate(
    name="/var/data/visitors.parquet",
    platform="s3",
    environment="PROD",
    description="Daily visitor export",
    properties={"region": "us-east-1", "endpoint": "http://localhost:19100"},
    tags=["pii", "bronze"],
)
dataset_urn = client.catalog.datasets.create(payload)
dataset = client.catalog.datasets.get(dataset_urn)

# List only datasets the receiver authorizes for this federated identity.
remote_datasets = client.catalog.datasets.list(target_cluster="receiver-name")

# Update tags/properties via URN
client.catalog.datasets.update(
    dataset_urn,
    DatasetUpdate(tags=["pii", "silver"], properties={"region": "us-east-1"}),
)

# Fetch or update the logical schema
schema = client.catalog.datasets.get_schema(dataset_urn)
client.catalog.datasets.update_schema(dataset_urn, schema)

# Convenience wrappers for legacy code that previously called
# `client.catalog.create_dataset(...)` still work and delegate to the dataset client.

Containers

from kamiwaza_sdk.schemas.catalog import ContainerCreate, ContainerUpdate

containers = client.catalog.containers
container_urn = containers.create(ContainerCreate(name="rag-demo", platform="kamiwaza"))
containers.add_dataset(container_urn, dataset_urn)

# Read all members, or explicitly walk bounded pages. Cursors are opaque.
page = containers.get(container_urn, datasets_page_size=100)
members = list(page.datasets)
while page.datasets_next_cursor is not None:
    page = containers.get(
        container_urn,
        datasets_page_size=100,
        datasets_cursor=page.datasets_next_cursor,
    )
    members.extend(page.datasets)

# Update metadata or re-parent the container
containers.update(container_urn, ContainerUpdate(description="Demo assets"))

# Remove membership or delete the container
containers.remove_dataset(container_urn, dataset_urn)
containers.delete(container_urn)

get() without pagination arguments retains the server's default behavior. datasets_page_size accepts 1–1000. The returned datasets_next_cursor marks another page, and workroom_id identifies the container's Workroom. Removing a membership or deleting its container retains the dataset and its schema. Catalog metadata deletion does not establish physical data deletion.

All URNs can be percent-encoded for the /v2/{urn} endpoints with client.catalog.encode_urn(...) (or DatasetClient.encode_path_urn / ContainerClient.encode_path_urn).

Secrets

from kamiwaza_sdk.schemas.catalog import SecretCreate

secrets = client.catalog.secrets
secret_urn = secrets.create(
    SecretCreate(
        name="minio/rag-demo",
        value="{'aws_access_key_id':'minioadmin','aws_secret_access_key':'minioadmin'}",
        owner="urn:li:corpuser:demo",
        description="Demo MinIO credentials",
    ),
    clobber=True,  # overwrite existing entry if present
)

secret = secrets.get(secret_urn)
for visible in secrets.list():
    print(visible.urn, visible.owner)

secrets.delete(secret_urn)

Secret URNs are generated by the server (often DataHub-style such as urn:li:dataHubSecret:<name>). Treat them as opaque identifiers—capture the value returned by create(...) and pass it verbatim to subsequent get/delete calls rather than constructing prefixes client-side.

The SDK now calls the /catalog/secrets/v2/{secret_urn} path with the raw URN, so you shouldn’t need to percent-encode or reshape it yourself; keep the string exactly as the server returned.

Secret reads return metadata, not the stored value. Creating a duplicate name fails unless clobber=True is explicit. Clobber replaces the existing secret while preserving its URN; applications should treat that as a credential change, not as key rotation in the underlying storage encryption scheme.

test_catalog_secret_contract_live.py exercises those contracts through uv run pytest -m integration, including the actual v2, by-URN and list HTTP responses before SDK model parsing can hide unknown fields. It uses a unique synthetic secret, checks default duplicate refusal and explicit clobber, and removes the secret. It does not inspect encrypted storage or establish key custody, cryptographic rotation, or another user's value-read permissions.

Diagnostics

Both the catalog and auth services now expose lightweight health endpoints to simplify readiness checks:

client.auth.health()        # -> {"status": "ok", ...}
client.auth.metadata()      # -> {"version": "...", ...}
client.catalog.health()     # -> {"status": "ok"}
client.catalog.metadata()   # -> {... service banner ...}

These helpers pair with the improved VectorDBUnavailableError so SDK consumers can reason about platform readiness before kicking off ingestion or retrieval jobs.

Writable dataset content

Set DatasetCreate(writable=True, storage=DatasetStorageRequest(backend="object")) to ask the platform for managed storage in the selected Workroom. Omit both fields to preserve the existing metadata-only create request. Dataset responses expose workroom_id, storage_binding and content_revision.

client.catalog.datasets provides create_object, list_objects, get_object, get_object_content, replace_object, delete_object and purge_object. Uploads take a (filename, bytes_or_open_binary_stream, content_type) tuple; create_object optionally takes a distinct logical_path. The SDK does not close caller-owned upload streams. Downloads return bytes and close the HTTP response. Content metadata is typed by the models in kamiwaza_sdk.schemas.dataset_content and preserves unknown response fields.

from kamiwaza_sdk.schemas.catalog import DatasetCreate
from kamiwaza_sdk.schemas.dataset_content import DatasetStorageRequest

urn = client.catalog.datasets.create(DatasetCreate(
    name="notes", platform="kamiwaza", writable=True,
    storage=DatasetStorageRequest(backend="object"),
))
objects = client.catalog.datasets
item = objects.create_object(urn, ("note.txt", b"first", "text/plain"))
assert objects.get_object_content(urn, item.item_id) == b"first"
replacement = objects.replace_object(
    urn, item.item_id, ("note.txt", b"second", "text/plain"),
    if_match=f'"{item.etag}"',
)
tombstone = objects.delete_object(
    urn, replacement.item_id, if_match=f'"{replacement.etag}"',
)

POST refuses an existing live path. Replace/delete forward If-Match unchanged; Core requires a quoted ETag, *, or an RFC 9110 list and returns 412 for a missing or stale condition. Replacement may allocate a new item ID; use the returned item for later writes. Deletion leaves a tombstone and retained bytes. purge_object is a separate, explicit administrator operation; a response is not independent proof of physical removal from the backend.

The integration node tests/integration/test_writable_dataset_content_live.py::test_writable_dataset_content_lifecycle uses its own Workroom and object-backed dataset, exact bytes, precondition and revision checks, and strict cleanup. It is selected by pytest -m integration. Storage limits, non-admin/tenant permissions, file-backend variants, concurrent writers, retention/acknowledgements and independent purge/audit readback remain explicit residuals.

Writable datasets require the kamiwaza catalog platform, independently of the object/file backend choice. Byte downloads are buffered in client memory; this SDK method does not return an HTTP streaming handle or response headers.