Skip to main content

Build a Connector Image

This page is for the developers who write a connector. It goes from the SDK's scaffold to an image that an operator runs with Run Your Own Connector.

Before you start​

You need Python 3.12, Docker, and the Kamiwaza SDK 1.4.0 or later with its connector kit. Earlier SDKs have no connector kit.

pip install 'kamiwaza-sdk[connector]>=1.4.0'

On a network without internet access, install the SDK from the field kit instead, as in Without internet access, and follow that section through the steps below.

Without internet access​

The field kit carries the SDK in its sdk/ folder:

FileWhat it is
kamiwaza_sdk-1.4.0-py3-none-any.whlThe Kamiwaza SDK, with kz-ext
kamiwaza_extensions_lib-0.6.3-py3-none-any.whlA library the SDK needs
requirements-1.4.0.txtEvery other package the SDK and its connector kit need, each at an exact version

Copy the sdk/ folder to the machine you build on, and keep a copy where you fetch packages for your mirror. Then:

  1. Load your pip mirror with every package that requirements-1.4.0.txt lists, at the version it lists. pip picks each package's files for the Python that runs it, so fetch them once with each Python that installs from the mirror: one like your build machine's (the same Python 3.12 on the same operating system), and the image's, Python 3.12 in python:3.12-slim. On a machine with internet access and a container engine:

    SDK_DIR=<full path to the sdk folder>
    python3.12 -m venv /tmp/kz-download
    /tmp/kz-download/bin/pip download -r "$SDK_DIR/requirements-1.4.0.txt" -d mirror-files
    docker run --rm --platform linux/amd64 -v "$SDK_DIR:/sdk:ro,z" -v "$PWD/mirror-files:/out:z" \
      python:3.12-slim pip download -r /sdk/requirements-1.4.0.txt -d /out
    ls mirror-files | grep -v '\.whl$'   # expect no output: every file is a wheel

    Run the two host commands (python3.12 -m venv and the pip download after it) on a machine like your build machine. Then carry mirror-files across and load its files into your mirror. The two Kamiwaza wheels can stay out of the mirror: the commands below take them from the sdk/ folder.

  2. Install the SDK from your mirror and the sdk/ folder, into a Python virtual environment. The environment also gives you a pip command on systems that have only python3.12, such as RHEL:

    python3.12 -m venv ~/kzenv && source ~/kzenv/bin/activate
    export PIP_INDEX_URL=https://pypi.acme.lan/simple   # your pip mirror
    SDK_DIR=<full path to the sdk folder>
    pip install --find-links "$SDK_DIR" 'kamiwaza-sdk[connector]>=1.4.0'
    pip show kamiwaza-sdk | grep '^Version'             # expect 1.4.0

    In a new shell, run source ~/kzenv/bin/activate again before using kz-ext. pip reads PIP_INDEX_URL, and kz-ext passes it to the image build, so set it even when your pip configuration already names the mirror. Use a URL without a username or password: kz-ext refuses one, because Docker keeps build arguments in the image's history.

  3. Point the image build at your registry. The build starts from python:3.12-slim. Copy it into your registry:

    export BASE_IMAGE=registry.acme.lan/acme/python:3.12-slim
    export DOCKER_DEFAULT_PLATFORM=linux/amd64

    The mirror holds the image's packages for Linux x86_64, so the image is built for linux/amd64 on any build machine.

    If your registry uses plain HTTP, allow it in your container engine first: in Docker's insecure-registries, or for Podman with an insecure = true entry for the registry in registries.conf.

    If your pip mirror's certificate comes from your own CA, the image doesn't trust it, and the build fails with CERTIFICATE_VERIFY_FAILED. Build a base image that trusts your CA, once, and use it instead. In an empty directory, put your CA certificate in PEM format as site-ca.crt, and this Dockerfile:

    FROM registry.acme.lan/acme/python:3.12-slim
    COPY site-ca.crt /usr/local/share/ca-certificates/site-ca.crt
    RUN update-ca-certificates

    Then build it, push it, and use it:

    docker build --platform "$DOCKER_DEFAULT_PLATFORM" -t registry.acme.lan/acme/python:3.12-slim-site-ca .
    docker push registry.acme.lan/acme/python:3.12-slim-site-ca
    export BASE_IMAGE=registry.acme.lan/acme/python:3.12-slim-site-ca

    kz-ext dev local and kz-ext dev pass BASE_IMAGE and PIP_INDEX_URL to the image build. Without them, the build uses python:3.12-slim and PyPI, which it can't reach.

  4. Start from the scaffold as in step 1, then copy the two Kamiwaza wheels into the project's wheels/ folder:

    cp "$SDK_DIR"/*.whl wheels/

    The scaffold's requirements.txt starts with --find-links wheels, so the image build takes the Kamiwaza packages from wheels/ and every other package from your mirror.

Then test and build as in steps 2 and 3, in a shell with PIP_INDEX_URL, BASE_IMAGE and DOCKER_DEFAULT_PLATFORM set.

If you load the two Kamiwaza wheels into your mirror as well, you can leave out --find-links "$SDK_DIR" and leave wheels/ empty.

1. Start from the scaffold​

mkdir pg-connector && cd pg-connector
kz-ext create --type connector --name pg-connector

kz-ext create writes into the current directory when it's empty, and into a new pg-connector directory otherwise.

The scaffold has the plumbing in place: it runs, describes itself, and takes a client certificate for its source. Two files are yours to finish: src/provider.py describes what the connector offers, and src/dispatcher.py answers the calls. Fill in the TODOs in them to reach your source.

For a worked example, the SDK includes a SQLite connector that answers queries. This prints its folder; its README.md says how to run it:

python -c 'import kamiwaza_extensions, pathlib; print(pathlib.Path(kamiwaza_extensions.__file__).parent / "examples/connectors/sqlite")'

2. Test against a stand-in source​

Put the settings and credentials for a test copy of the source in .kz-ext/connection.json. Certificate fields take the PEM text or a file path in the project. The connector runs in a container, so localhost there is the container itself: name the source by an address the container can reach.

mkdir -p .kz-ext && cp connection.example.json .kz-ext/connection.json   # then edit it
kz-ext validate
kz-ext dev local --check

kz-ext dev local builds and runs the connector on your machine and checks that it starts, describes itself, and reaches the source. With --check it runs the checks once and exits; without it, the connector keeps running and reloads as you edit. It needs only Docker, not a Kamiwaza install.

3. Build the image​

Cluster nodes are usually linux/amd64. On an Apple Silicon Mac, set the platform, or the image won't run on them:

export DOCKER_DEFAULT_PLATFORM=linux/amd64

Then either:

  • Deploy it yourself with kz-ext dev, if you can reach Kamiwaza and its registry: see route A.

  • Hand it over to the operator who runs the cluster. Build the image and push it to a registry the cluster pulls from. Give every build a new tag; a rebuilt image under an old tag doesn't roll out:

    docker build -t registry.acme.lan/acme/pg-connector:1.0.0 .
    docker push registry.acme.lan/acme/pg-connector:1.0.0

    If no registry is reachable from both your network and the cluster's, save the image to a file instead and send that across. It's the same image, as one .tar file:

    docker build -t pg-connector:1.0.0 .
    docker save pg-connector:1.0.0 -o pg-connector.tar

    Either way, the operator takes it from there with route B.

Without internet access, docker build needs your base image and mirror too. Pass them as build arguments, after setting up the project as in Without internet access:

docker build --build-arg BASE_IMAGE="$BASE_IMAGE" --build-arg PIP_INDEX_URL="$PIP_INDEX_URL" \
  -t registry.acme.lan/acme/pg-connector:1.0.0 .