Run Your Own Connector

A connector is a small service that reads one kind of data source. Your developers build it as a container image (see Build a Connector Image). This page takes that image to a connection your analysts can use from Kaizen.

Kamiwaza stores the connection's credentials, such as a client certificate, and sends them to the connector with each call. The connector reaches the source itself.

Step	What
1	Get the image into your registry and register the connector type
2	Create the data source and its connection, and verify
3	Optional: share it with fewer people than everyone
4	Turn on its tools in Kaizen, and check

Before you start

You need:

A Kamiwaza administrator account, and a Kaizen administrator for step 4.
A registry the cluster can pull from. Keep connectors in your own registry: on an install without internet access, Kamiwaza redirects images on public registries such as ghcr.io or Docker Hub to its mirror.
The source reachable from the cluster's nodes. Open that route in your network first.
A machine with curl and jq that reaches Kamiwaza.

If Kamiwaza's certificate comes from your own CA, point curl and kz-ext at your CA file:

export CURL_CA_BUNDLE=<your CA file> REQUESTS_CA_BUNDLE=<your CA file>

If Kamiwaza uses a self-signed certificate, save it first, after checking with your administrator that it's the right one, and use "$PWD/kamiwaza-ca.crt" (the full path, so it still works from another directory) as the CA file above:

openssl s_client -connect <YOUR_KAMIWAZA_DOMAIN>:443 -servername <YOUR_KAMIWAZA_DOMAIN> </dev/null \
  | openssl x509 > kamiwaza-ca.crt

Get an API token, and define kapi, which calls Kamiwaza's API with it. The commands on this page use kapi <METHOD> <path> [curl options]:

export KAMIWAZA_URL=https://<YOUR_KAMIWAZA_DOMAIN>
ADMIN_USER=admin   # your administrator's username
printf 'password for %s: ' "$ADMIN_USER"; read -rs ADMIN_PASSWORD; echo
TOKEN=$(printf '%s' "$ADMIN_PASSWORD" | curl -sS -X POST "$KAMIWAZA_URL/api/auth/token" \
  -d grant_type=password --data-urlencode "username=$ADMIN_USER" --data-urlencode "password@-" \
  | jq -r '.access_token')
unset ADMIN_PASSWORD
[ -n "$TOKEN" ] && [ "$TOKEN" != null ] && echo "token ok" || echo "token failed"

kapi() { curl -sS -X "$1" "$KAMIWAZA_URL/api$2" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' "${@:3}"; }

The token lasts four hours; run this again when calls start returning 401.

Step 1: Get the image in and register it

Use A when one machine builds the connector and reaches both your registry and Kamiwaza. Use B when someone else builds the image, or builds it on another network, and you register it on the cluster.

A: With `kz-ext dev`

From the connector's project, with the Kamiwaza SDK installed:

docker login registry.acme.lan
export KAMIWAZA_REGISTRY=registry.acme.lan/acme   # where the cluster pulls from
kz-ext login --admin "$KAMIWAZA_URL/api"
kz-ext dev

kz-ext dev builds the image, pushes it to registry.acme.lan/acme/<name>-connector:<revision>, and registers the connector type. When it finishes, it prints the connector type. Set it for the next steps: CONNECTOR_TYPE=<that type>.

On a network without internet access, set up the project as in Without internet access first, and run kz-ext dev in a shell with BASE_IMAGE and PIP_INDEX_URL set. The SDK and its packages come from the field kit and your pip mirror.
Leave out docker login if your registry doesn't need one.
If this machine pushes to the registry under a different address than the cluster pulls from, set that address, with the same path, in KAMIWAZA_PUSH_REGISTRY too.
For a registry on plain HTTP, set KAMIWAZA_REGISTRY_INSECURE=1, and allow the registry in your engine: Docker's insecure-registries, or the Podman machine's registry settings. kz-ext dev checks this before it builds and says what's missing.
If you've logged in to another Kamiwaza before, check that kz-ext dev's first lines name the right one. Switch with kz-ext login --use <name> (kz-ext login --list shows them).

B: Register an image yourself

You also need docker or podman; the commands are the same for both. Name the image as the cluster pulls it, in three parts. If this machine reaches the registry at another address than the cluster's nodes, set that address in PUSH_REGISTRY; the commands below push and run the image through it:

IMAGE_REGISTRY=registry.acme.lan      # as the cluster's nodes reach it
IMAGE_REPOSITORY=acme/pg-connector
IMAGE_TAG=1.0.0
PUSH_REGISTRY=$IMAGE_REGISTRY         # as this machine reaches it
IMAGE="$PUSH_REGISTRY/$IMAGE_REPOSITORY:$IMAGE_TAG"

If the image is already in that registry, because your developers pushed it there, go on to reading the manifest below.

If you have the image as an archive (a .tar file from docker save, for example carried across to a network without internet access), load it:

docker load -i pg-connector.tar      # prints "Loaded image: <name>"

Then tag that name for your registry and push it:

docker tag <the name it printed> "$IMAGE"
docker login "$PUSH_REGISTRY"        # only if your registry needs a login
docker push "$IMAGE"                 # plain-HTTP registry: podman push --tls-verify=false

With Docker and a plain-HTTP registry, list the registry under insecure-registries in Docker's daemon settings before you push or run the image.

The connector describes itself. Run the image once, on any machine that can pull it, to read that description, its manifest:

docker run -d --name connector-manifest -p 127.0.0.1:18080:8080 "$IMAGE"
for i in $(seq 1 30); do curl -sf 127.0.0.1:18080/manifest > manifest.json && break; sleep 2; done
docker logs connector-manifest; docker rm -f connector-manifest
[ -s manifest.json ] && echo "manifest ok" || echo "no manifest: see the log above"

CONNECTOR_TYPE=$(jq -r .connector_type manifest.json)
jq --arg r "$IMAGE_REGISTRY" --arg p "$IMAGE_REPOSITORY" --arg t "$IMAGE_TAG" \
  '{manifest: (.deployment += {image_registry: $r, image_repository: $p, image_tag: $t}), version: $t}' \
  manifest.json \
  | kapi PUT "/connectors/catalog/$CONNECTOR_TYPE" -d @- | jq '.connector_type'

Check

List the connector's settings. You fill these in at step 2:

kapi GET "/connectors/catalog/$CONNECTOR_TYPE" | jq '.manifest.config_fields[] | {name, required, description}'

Step 2: Create the data source and connection

A data source is the catalog entry you share. The connection belongs to it and holds the settings and credentials.

DATA_SOURCE=$(kapi POST /catalog/containers/ \
  -d '{"name": "ACME operations incidents", "platform": "kamiwaza-connector"}' | jq -r .)
echo "$DATA_SOURCE"   # urn:li:container:(...), or an error
DS_ENC=$(jq -rn --arg ds "$DATA_SOURCE" '$ds|@uri')   # the same, for use in a URL

The data source is shared with the workroom it's created in. Without a workroom, that's the Global workroom, so every user can see it. To share it with only one workroom, enter that workroom before you create the data source, as in step 3.

Create the connection with the settings from step 1. The settings below are an example, for a PostgreSQL connector that takes a url and a client certificate:

jq -n --arg ds "$DATA_SOURCE" --arg type "$CONNECTOR_TYPE" \
  --rawfile cert client.crt --rawfile key client.key --rawfile ca source-ca.crt \
  '{name: "ACME ops database", connector_type: $type, container_urn: $ds,
    config: {url: "postgresql://10.20.0.15:5432/ops?user=svc_kamiwaza_reader",
             client_certificate: $cert, client_key: $key, ca_bundle: $ca}}' \
  | kapi POST /connectors/ -d @- > connection.json
CONNECTION=$(jq -r '.id // empty' connection.json)
[ -n "$CONNECTION" ] && echo "connection $CONNECTION" || cat connection.json

Kamiwaza starts the connector. The status goes from not_deployed to deploying to ready, usually within a minute:

for i in $(seq 1 30); do
  STATUS=$(kapi GET "/connectors/$CONNECTION" | jq -r .status)
  echo "$STATUS"; [ "$STATUS" = ready ] || [ "$STATUS" = degraded ] && break; sleep 10
done

Then have the connector check that it reaches the source, signs in and reads:

kapi POST "/connectors/$CONNECTION/verify" \
  | jq '{status, connection_status, checks: [.checks[] | {capability, status, message}]}'

Check: status is success and connection_status is connected. If not, see Troubleshooting.

Step 3 (optional): Choose who sees it

Skip this step to leave the data source shared with every user.

One workroom's members. Create the data source from inside that workroom. You must be a member of it. Before step 2, find its ID among the workrooms you belong to:

kapi GET /workrooms/ | jq -r '.items[] | "\(.id) \(.name)"'

Enter it. Some installs issue a new token for the workroom; this keeps it:

WORKROOM=<its ID>
kapi POST "/workrooms/$WORKROOM/enter" > enter.json
jq -r '.workroom_id // .detail' enter.json   # the workroom's ID, or why it failed
TOKEN=$(jq -r --arg t "$TOKEN" '.access_token // $t' enter.json)

If it doesn't print the workroom's ID, stop: step 2 would create the data source in the Global workroom.

Run step 2 as written, then return to the Global workroom the same way:

TOKEN=$(kapi POST /workrooms/leave | jq -r --arg t "$TOKEN" '.access_token // $t')

Or, a group or single users. For a data source created outside any workroom, grant them viewer, then remove the share with the Global workroom. A group is a Kamiwaza group; create one and add its members by username:

GROUP=$(kapi POST /authz/groups/ -d '{"name": "ops-analysts"}' | jq -r '.id // empty')
[ -n "$GROUP" ] && echo "group $GROUP" || echo "group not created"

for user in <USERNAME> <USERNAME>; do
  kapi POST "/authz/groups/$GROUP/members" -o /dev/null -w "$user %{http_code}\n" \
    -d "$(jq -n --arg u "$user" '{user_id: $u}')"
done

Each member prints 204; a 404 means no enabled user has that name. Then grant the group viewer and remove the Global share (GLOBAL is the Global workroom's ID):

GLOBAL=ffffffff-ffff-ffff-ffff-ffffffffffff
kapi POST /authz/resources/container/grants \
  -d "$(jq -n --arg ds "$DATA_SOURCE" --arg g "$GROUP" \
        '{object_id: $ds, subject: {namespace: "group", id: $g}, relation: "viewer"}')"
kapi DELETE "/authz/resources/container/grants?object_id=$DS_ENC&subject_namespace=workroom&subject_id=$GLOBAL&relation=shared_to_workroom"

To grant a single user instead, use {namespace: "user", id: <their linked_subject_id>}, with the ID from GET /api/auth/users/.

Check: the data source's grants are you as owner plus whoever you shared it with:

kapi GET "/authz/resources/container/grants?object_id=$DS_ENC" \
  | jq -r '.[] | "\(.relation) \(.subject.namespace):\(.subject.id)"'

For a workroom, that's shared_to_workroom workroom:<its ID>. A user can check for themselves with GET /api/connectors/available and their own token: it lists exactly the connections they can use.

Restrict what it returns. Sharing decides who can use a data source. To also drop records or redact fields based on the content and the user's attributes, see Restrict a Data Source.

Step 4: Turn it on in Kaizen

A Kaizen administrator opens Admin area → Connectors, finds the connection, and turns on Tools. Tools are turned on per connection, so do this for every new connection. RAG also needs an embedding model selected in Admin area → Models. On an install without internet access, Kaizen's chat model must be one you serve in the cluster or reach through a private endpoint.

Check: as a user who can see the data source, start a Kaizen conversation, open Chat actions, select the connection under Connectors, and ask something only the source can answer. The agent calls the connector's tools, named kz_conn_<connector type>_<last 10 characters of the connection ID>_<operation>, such as kz_conn_acme-postgres_90efdff5e1_query. A user who can't see the data source isn't offered the connection.

Roll out a new version

Give each new image a new tag; an image pushed again under the same tag doesn't roll out.

Route A: run kz-ext dev again. It moves every connection of this type to the new image, waits for them, and prints the version each one runs.
Route B: get the new image into your registry and register it again, as in step 1, with the new IMAGE_TAG. Replace the last jq '.connector_type' with jq '.rollout' to see each connection's progress: updated or recreating is under way, failed keeps the old version running and says why.

Then run verify on each connection, as in step 2. To confirm which image a connection runs, check its pod (find extension_name as in Status and logs):

kubectl -n kamiwaza get pods -l extensions.kamiwaza.io/deployment-id=<extension_name> \
  -o jsonpath='{.items[*].spec.containers[*].image}'

Remove

Delete the connection, which stops the connector, then the data source:

kapi DELETE "/connectors/$CONNECTION" -o /dev/null -w '%{http_code}\n'
kapi DELETE "/catalog/containers/$DS_ENC" -o /dev/null -w '%{http_code}\n'

Each prints 204. For a data source you created in a workroom, enter that workroom first, as in step 3, and leave it afterwards; otherwise the data source delete prints 404. Deleting the data source also removes who it was shared with. The connector type stays registered, and any group you created stays. To delete a group from step 3 as well:

kapi DELETE "/authz/groups/$GROUP" -o /dev/null -w '%{http_code}\n'

It prints 204. A 409 means the group still has access to something else.

On OpenShift with GreyMatter

The steps are the same, with these differences:

Certificate. The edge usually uses your own CA, so export it as in Before you start.

Registry. You can use the cluster's built-in registry. Push through its route, and have the cluster pull from its in-cluster address. With route A:

oc registry login
export KAMIWAZA_PUSH_REGISTRY=$(oc get route default-route -n openshift-image-registry -o jsonpath='{.spec.host}')/kamiwaza
export KAMIWAZA_REGISTRY=image-registry.openshift-image-registry.svc:5000/kamiwaza

With route B, run oc registry login, and in step 1's first block use these lines in place of the matching ones, keeping IMAGE_TAG and IMAGE. The image is pushed through the route and registered at the in-cluster address:

IMAGE_REGISTRY=image-registry.openshift-image-registry.svc:5000
IMAGE_REPOSITORY=kamiwaza/pg-connector
PUSH_REGISTRY=$(oc get route default-route -n openshift-image-registry -o jsonpath='{.spec.host}')

User ID. OpenShift runs the connector as a user ID from the namespace's range. A connector built from the SDK's scaffold runs unchanged.
Routing. Nothing to add in GreyMatter. Kamiwaza calls the connector inside the namespace; users still come in through the GreyMatter edge.

Troubleshooting

Symptom	Fix
`token failed`, or every call returns `401`	The token expires after four hours: get a new one, as in Before you start. Otherwise, check the CA file you exported there and the password.
The connection stays `deploying` or turns `degraded`	Usually the cluster can't pull the image. Check the pod's events (below): the image address, and the registry credential in the install's image pull Secret.
Verify fails to connect: refused or timed out	Open the route from the cluster's nodes to the source.
Verify fails with a certificate error	Check the certificate, key and CA files, then update the connection with `PUT /api/connectors/<id>` and `{"config": {...}}`.
Users outside the group still see the connection	Remove the Global share, as in step 3.
A new version doesn't roll out	Give the image a new tag.

Status and logs. status_reason says why a connection isn't ready, and extension_name finds its pod:

kapi GET "/connectors/$CONNECTION" | jq '{status, status_reason, extension_name}'

kubectl -n kamiwaza get pods -l extensions.kamiwaza.io/deployment-id=<extension_name>

Then, with the pod's name:

kubectl -n kamiwaza describe pod <pod>   # pull errors
kubectl -n kamiwaza logs <pod>           # the connector's own log

Use the namespace Kamiwaza is installed in if it isn't kamiwaza.

Audit. Kamiwaza records an audit event for every connector call, with the data source, the operation, the result and the number of rows; not the call's arguments. It goes to your OpenTelemetry collector, or to the API's log:

kubectl -n kamiwaza logs deployment/core-api --since=1h | grep kamiwaza.audit.sensitive_data.access

The API's log is a quick check only: it covers one pod and is lost when the API restarts. Keep audit events in your OpenTelemetry collector.

Before you start​

Step 1: Get the image in and register it​

A: With kz-ext dev​

B: Register an image yourself​

Check​

Step 2: Create the data source and connection​

Step 3 (optional): Choose who sees it​

Step 4: Turn it on in Kaizen​

Roll out a new version​

Remove​

On OpenShift with GreyMatter​

Troubleshooting​