Skip to main content

Ingestion Service

Client helpers live in kamiwaza_sdk/services/ingestion.py and wrap the /ingestion/ingest API surface exposed by Kamiwaza. DDE connector triggering uses the separate /dde/connectors route.

Usage​

from kamiwaza_sdk import KamiwazaClient

client = KamiwazaClient("https://localhost/api", api_key="...")  # or set authenticator

# Active ingestion (runs immediately)
resp = client.ingestion.run_active(
    "s3",
    bucket="kamiwaza-sdk-tests",
    prefix="sdk-integration",
    endpoint_url="http://localhost:19100",
    region="us-east-1",
    aws_access_key_id="minioadmin",
    aws_secret_access_key="minioadmin",
)
print(resp.urns)

# Emit a Metadata Change Proposal
client.ingestion.emit_mcp({"entityType": "dataset", "changeType": "UPSERT", "entityUrn": resp.urns[0]})

# Register a job for polling (registration does not execute the cron schedule)
from kamiwaza_sdk.schemas.ingestion import IngestJobCreate

job = IngestJobCreate(
    job_id="nightly-s3-sync",
    schedule="0 3 * * *",
    source_type="s3",
    conn_args={"bucket": "...", "recursive": True},
)
client.ingestion.schedule_job(job)
status = client.ingestion.get_job_status(job.job_id)
print(status.status, status.last_run)

# Trigger an approved DDE data-source connector (requires manage permission)
trigger = client.ingestion.trigger_dde_connector_ingest("9e7fc7b1-e817-47a3-b360-872f09f379bb")
print(trigger.status)

Routing note: the FastAPI router is included under /ingestion, so the fully-qualified paths are /ingestion/ingest/run, /ingestion/ingest/jobs, etc.

Live capability verification​

test_run_now_owner_job_and_dde_trigger requires a private JSON file named by KAMIWAZA_DDE_TRIGGER_FIXTURE. It must contain allow_repeat_trigger: true, the ID of a dedicated disposable DDE connector, its expected_source_ref seed document, and a dedicated non_manager_api_key. The non-manager must have read access to that same connector but not manage access. The requester needs platform-admin access to read back the dataset owner grant. Both clients require verified TLS; configure a trusted CA rather than disabling certificate checks. Keep the fixture file outside the repository. The test triggers ingestion and leaves the platform's job and document history in place; do not point it at a shared production connector. Without the approved fixture, the complete-claim test skips rather than emitting a passing capability record.