Ingestion & indexing

How content becomes searchable, and what to do when it does not.

The pipeline

source ──▶ fetch ──▶ extract ──▶ chunk ──▶ lexical index ──▶ searchable
           │          │                          │
        connector  document-                     └─▶ embed ──▶ dense index
        or upload  extractor                         (opt-in, member-initiated)

For uploads, the two halves are deliberately separate. Extraction, chunking, and the lexical index run automatically, so a document is previewable and keyword-searchable as soon as it lands. Dense semantic indexing is opt-in: a member selects Index on a ready document when they want it available for retrieval-augmented answers.

When hybrid retrieval is disabled or pg_textsearch is unavailable, Search still includes keyword matches alongside available vector matches. Restart the API after installing pg_textsearch to enable hybrid retrieval; existing uploads do not need to be imported again.

Indexing runs in bounded batches and resumes over the document's existing chunks, so a large document is never re-uploaded or duplicated to finish it.

Indexing covers a chunk completely, whatever wrote it. The streaming lexical path writes rows far larger than one embedding window, so those rows are embedded as a series of windows spanning the whole chunk and stored as one pooled vector for it. A large XML, NDJSON, JSON, CSV, or text upload is therefore semantically searchable across all of its content rather than by its opening tokens; the cost of indexing scales with bytes, exactly as it does for a PDF of the same size.

The request that starts it is a conditional claim: it transitions the document into dense_indexing only when there is still work to do. When the claim matches nothing the request is refused — 409 when the document is already indexed or indexing, 422 when it holds nothing left to embed — and never accepted. A document sitting at lexical_ready after an accepted index request is therefore a real fault to investigate, not a state the member could have reached by clicking Index.

Ready chunks also provide the exact-version read substrate for corpus Tasks. Corpus enumeration does not require dense indexing and never uses top-k search as its coverage authority. Folder and Collection membership is snapshotted at run start; each chunk page then re-checks current lifecycle and ACL before use. Deleting or revoking a source stops later partition reads and redacts member projections without rewriting historical lineage.

Upload phases

queued · extracting · chunking · embedding · dense_indexing · semantic_degraded · lexical_ready · ready · failed

lexical_ready is the steady state for a document that has been extracted and lexically indexed but not semantically indexed — it is not a failure. Document chat still works against it, falling back to keyword matching over its chunks. semantic_degraded means dense indexing was attempted and could not complete; check the embedding role first.

Two entry points feed it:

• Uploads — a member adds a file; bytes go to the object store, metadata to PostgreSQL, and the document worker takes it from there.
• Connector sync — a member starts a sync; Kaizen walks the connector's generic surfaces and streams each discovered document through the same pipeline.

The durable upload row is committed before queue delivery. If Valkey or a worker disappears in that handoff window, the recovery coordinator finds documents still in the queued phase and re-enqueues each with a deterministic job identity. This repairs broker loss without duplicating the upload or retaining a Valkey backup.

Incomplete upload recovery

Multipart completion has one durable owner for at most ten minutes. The browser waits up to ten minutes and thirty seconds for a confirmed result, including bounded requests and polls. upload_in_progress means that completion is still owned; an unknown result at the browser deadline does not authorize deleting or restarting the staged object. The Documents transfer row retains its original session and offers Check upload, which reads only that session’s committed outcome through authenticated GET on its original /complete path. It performs no claim, provider I/O, renewal, takeover, hash, cleanup or journal write. The original POST alone may publish. An older backend GET 405 leaves the handle unresolved; it cannot accidentally initiate completion. It blocks visible, grouped and programmatic Retry, Cancel and Dismiss while unresolved. Neither the filename nor another library row establishes origin. Another unknown result, conflict or missing result stays unresolved without a new upload or cleanup; inspect the exact original session if repeated checks cannot settle it. The chat composer also stops after an unconfirmed completion; it does not start another transfer. Its attachment message directs the member to Documents and Check upload, where the retained original-session handle is restored. A completed session whose document was deleted returns not_found and cannot recreate the document.

The 10 GiB file limit is an admission limit, not a guarantee that every storage deployment can finish a maximum-size file within the completion budget. Reading and hashing the staged object, plus publication, must fit the server's 600-second owner window; the browser's 630-second limit supplies bounded response grace. Repeated timeouts require investigation of object-storage throughput, hashing capacity and competing load. Check the original session before recovery; starting another transfer or deleting its staged object is not a timeout remedy.

Deploy the current frontend and backend together for this recovery contract. Before admitting current upload cleanup, drain or stop older backend upload, part and completion handlers. The current reference fence protects retained registered originals, including another owner's or a soft-deleted row's, but does not make publication atomic with concurrently running older handlers. An uploading session left after an older registration commit refuses abort with upload_abort_uncertain; its original completion can reconcile the saved document without deleting its bytes. No deployed writer drain is attested by the local source tests. Only an acknowledged exact-session completion_outcome="rejected" response permits fresh Retry; absent proof, including raw rejection codes from older backends, stays unresolved. Rejection proof is not cleanup proof. Browser sessionStorage must retain and read back the minimal original-session/app-binding handle before the first completion request; if unavailable or over quota, publication is blocked while precompletion cleanup remains safe. That known preflight failure offers ordinary retained-file Retry after storage restoration, not Check/replay. A potentially-written handle clears only after exact owner abort headers confirm fenced precompletion cleanup and acknowledged row deletion; raw204 also covers foreign/missing sessions and never proves that boundary. Absent proof or a lost acknowledgement retains an explicit cleanup gap. A reload or normal 401 sign-in navigation restores a Pending upload row without file or private metadata. A lost storage acknowledgement can leave a handle after zero completion POSTs; it does not prove saving started. The restored no-File row may only inspect the read-only result, preserving missing-result cleanup gaps without replaying publication. Check still requires the exact current app/API binding and server member ownership. Missing, authentication-denied or unknown results keep the handle; there is no expiry-based deletion or inferred restart permission.

Check can expose a full original manifest as explicitly resumable with upload_resume_available; Resume saving rechecks and uses the original POST. It does not override a current owner or restart the file transfer. Cancel stays recorded until exact original-session logical discard is acknowledged; Check can reconcile a lost acknowledgment from committed origin plus an absent or tombstoned owned row. This disposition is not provider-purge success. Missing, foreign, malformed or authentication failures retain the handle. Pair current frontend/backend contracts; do not replace this recovery with guessed IDs.

Abort of a completed multipart session returns non-retryable 409 upload_abort_completed; interrupted or expired unknown publication returns non-retryable 409 upload_abort_uncertain. A 503 upload_in_progress abort response is reserved for a currently active completion lease. These refusals leave the receipt and staged bytes intact; they never report successful abort.

A definite quota, size, corrupt-file, or unsupported-format rejection records a permanent server-owned receipt before reclaiming its staged object. Cleanup checks every document reference, including another member's or a deleted row's, and performs provider I/O after releasing the database session. A stale claim, unknown provider result, or unacknowledged registration commit never grants cleanup permission.

Provider deletion can fail or outlive a canceled request. These receipts retain cleanup: pending until deletion is acknowledged and a new journal checkpoint records cleanup: done. Initial rejection cleanup and subsequent recovery each have a two-second total budget. Exhausting that budget keeps the original upload result and pending receipt; it does not prove that a remote operation stopped. The owner can replay or abort the rejected multipart session to retry cleanup. Automatic stream discovery starts from indexed rejected/canceled pending events and excludes any event with a newer head at the same owner, session and branch. Successful upload history is outside that candidate index. Historical pending events remain append-only even after later settlement; the index is not a fixed size live-pending queue, and absent canceled objects remain pending for late writes. Migration 0148_stream_recovery_candidates builds an ordinary index over the journal. Qualify a maintenance window and append-writer blocking at the installed scale; no zero-downtime construction or deployed latency bound is attested. A subsequent authenticated stream upload attempts up to ten distinct eligible stream receipts for that owner within the same total budget, before accepting the new file. A durable owner-scoped cursor rotates through immutable receipt IDs and commits each next position before provider I/O. Missing, failing or slow provider results therefore cannot monopolize every later recovery call. The cursor grants no cleanup authority; terminal receipt and global-reference checks still decide each attempt. Other owners cannot select those receipts.

Large stream uploads commit a server-owned intent before writing any object. Its owner, document ID, derived key, immutable metadata and token also fence the publication transaction. If initial intent persistence fails or its acknowledgement is lost, no provider write starts. If a later definitive-rejection transaction fails, the intent remains discoverable as settlement_pending; the original typed upload rejection is preserved. An intent never grants automatic cleanup authority. Successful registration and its terminal receipt commit together.

In the application container environment, list at most ten pending stream or rejected multipart receipt IDs for an inactive owner with python -m kaizen.documents.rejection_recovery_cli --owner-sub <stored-owner> --list-pending. Inspect one exact receipt using --receipt multipart:<session-uuid> or --receipt stream-rejection:<document-uuid>. The inventory includes next_receipt when another page exists. Continue with --list-pending --after-receipt <next-receipt> using the same exact owner; the cursor must identify that owner's exact stored stream or multipart receipt, even if its latest cleanup has finished since the previous page. Each page remains limited to ten IDs and two seconds. A null next_receipt ends that scan; start a new scan to include later arrivals. Listing never deletes, cancels, or advances the automatic stream-only recovery cursor. Automatic retries of one already durable cursor position still inspect the provider, but do not append a redundant cursor event. An absent object remains pending; it does not acquire cleanup proof. Only latest multipart heads with a definitive rejection and pending cleanup appear; active, interrupted, completed or unstarted multipart sessions grant no listing or cleanup authority. Older pending events do not make a latest cleanup-done receipt pending again.

The default is dry-run. Add --apply to retry an acknowledged rejection or cancellation cleanup. These commands use the same immutable receipt and global reference fences; shell access to the configured application DB/storage is the operator boundary. No new member endpoint or arbitrary blob-key deletion exists.

Before consequential --apply or --cancel-pending work, a short transaction commits document.upload.recovery.requested through AuditService. It targets the exact owner-bound stored journal event UUID and carries an opaque attempt UUID plus closed action, receipt kind, state, cleanup and reason fields. An unknown or unavailable audit acknowledgement starts no cancellation, journal mutation or provider I/O. document.upload.recovery.finished reports success only when cleanup is actually acknowledged; pending or unknown effects are non-success. An outcome audit failure reports audit: outcome_unconfirmed with exit code 2 even if cleanup completed; it does not claim rollback. A failed cleanup retry after a late writer recreated bytes reports cleanup: unconfirmed and a failed finished outcome instead of treating a prior cleanup: done receipt as proof of this attempt. Intent failures report audit: intent_unconfirmed and exit 2. Inspect again rather than assuming success.

The fixed code-owned actor is system:document-upload-recovery, with SYSTEM kind. This identifies the tool component, not an authenticated human operator or runtime execution instance. Owner selectors and environment/CLI person claims cannot change it. No human attribution is provided. Read-only inventory, inspect and inapplicable/foreign receipts emit no destructive-action audit. Audit never stores raw blob keys, owner text, credentials or private errors.

A stranded stream intent with valid bytes may have lost a mutable quota rejection; recovery cannot reconstruct that original decision or publish it as an accepted upload. An operator can instead explicitly cancel that intent by selecting its receipt ID and adding --cancel-pending. This records operator_explicit cancellation under the publication fence. It refuses any complete/canonical outcome and any exact-key reference, including another owner or a deleted row. Later publishers are fenced. Cleanup only attempts a known existing atomic object; missing objects and ambiguous provider responses retain cleanup: pending. Cancellation does not prove that a remote writer stopped. The same canceled receipt can be recovered again if a late writer recreates orphan bytes, without registering a document or inventing the original failure.

Each command has a two-second total budget. Timeout/unavailable outcomes and applied cleanup still pending exit with code 2; inspect again rather than assuming deletion. Foreign, malformed, and multipart active/unknown receipts grant no cleanup authority. Only the latest journal head determines pending state; historical pending events remain after recovery. Keep the bounded receipt/status output as operational evidence. These commands have not been run against live operator resources as part of the local controlled regression checks.

Extraction

document-extractor turns PDFs, Office formats, plain text, Markdown, and more into text, including OCR for scanned or image-only pages.

ZIP, TAR, TAR.GZ, and TGZ uploads take a separate, read-only path in the document worker. It indexes UTF-8 text entries with their relative paths as section labels; it does not write archive members to disk, execute content, or expand nested archives. Non-regular entries and unsafe paths fail the import. Sources are limited to 64 MiB compressed, 1,000 entries, 64 MiB of declared uncompressed data, and 8 MiB of readable text. A limit failure leaves an actionable failed status; binary entries are not indexed. XML is indexed as UTF-8 markup without entity expansion; files of at least 8 MiB use the streaming lexical path. These paths do not call Tika. Raw original projection to an agent's code sandbox retains its separate 8 MiB total limit.

For self-contained HTML applications, the worker combines Tika's visible-text projection with valid JSON found in JSON script elements and static JavaScript assignments. The JSON is decoded without executing the page and rendered as record-oriented text, so embedded graph nodes, links, and similar datasets can be chunked and indexed. DOCUMENT_HTML_STRUCTURED_DATA_MAX_BYTES limits both the script material inspected and the structured source admitted per document; the default is 64 MiB. Exceeding the bound keeps the visible text, records an extraction warning, and omits the excess structured data.

It is stateless and replaceable. A failure here delays indexing without affecting anything already indexed.

OCR is the expensive path. A batch of scanned PDFs will take substantially longer than the same page count of native text.

Pictures inside a document

A picture embedded in a document that also has text is extracted in its own right: Tika returns each embedded image as a further resource, Tesseract reads the words in it, and the worker folds that text into the document at the end of the page the picture sits on, under a member-legible marker such as [Figure 1 (page 2): chart.png]. The page separators and page count are untouched, so page citations still resolve to the page a member would turn to. A picture that yields no readable words is still recorded as a figure, which is what lets an assistant say a file contains a picture instead of denying it.

PDFs need one extra thing that the other formats do not. Office, OpenDocument and email attachments are embedded resources already, but a picture drawn onto a PDF page only becomes one when PDFParser is told to extract inline images. The shipped document-extractor image carries that setting in /opt/tika-server/tika-config.json, so an operator running a substituted or older extractor image gets figure extraction for Office formats and nothing for PDFs. Tika 4 removed the per-request X-Tika-* parser headers, so this is a property of the image rather than of a request.

Two bounds keep an image-heavy upload from dominating a worker. Both are per-document, applied in document order, and what they drop is reported as an extraction warning on the document rather than silently omitted.

Setting	Default	Purpose
DOCUMENT_EMBEDDED_FIGURE_MAX_COUNT	20	Pictures read per document
DOCUMENT_EMBEDDED_FIGURE_TEXT_MAX_CHARS	20000	Recovered text per document

Either setting at 0 turns figure extraction off without changing the extractor image. Identical repeats of one image — a logo on every page — are collapsed by the extractor before Kaizen sees them.

Cost is bounded in practice by the extraction deadline rather than by page count: the words in each picture are read once, so a text PDF with no pictures costs exactly what it did before, while a PDF carrying many distinct photographs adds one OCR pass per picture. Uploads above DOCUMENT_INGEST_ASYNC_BYTES run on the worker tier with DOCUMENT_LARGE_EXTRACTION_TIMEOUT_MS, which is where a picture-heavy file lands; a document that exceeds its deadline fails with a reason, as before.

Tesseract in the shipped image carries eng, deu, fra, ita, jpn and spa. Words in a picture in another script are not recovered, and the figure is recorded without text.

Chunking

Setting	Default	Purpose
INGEST_MINI_CHUNKS_ENABLED	on	Fine-grained chunks alongside passages
INGEST_MINI_CHUNK_TARGET_TOKENS	64	Mini-chunk size
INGEST_MINI_CHUNK_OVERLAP_TOKENS	16	Overlap
INGEST_EMBED_MAX_INPUT_TOKENS	240	Per embedding input
DOCUMENT_HTML_STRUCTURED_DATA_MAX_BYTES	67108864	Embedded JSON admitted from HTML

Mini-chunks improve precision on short factual lookups; passages preserve context for synthesis. Both are indexed.

Embedding

Embeddings are produced only through Kaizen's model gateway, on the worker's bulk budget.

Setting	Default
INGEST_EMBED_BATCH_SIZE	16
INGEST_EMBED_MAX_BATCH_TOKENS	4096
KAMIWAZA_CONNECTOR_SYNC_INGEST_CONCURRENCY	2 in the packaged Kamiwaza topology
DOCUMENT_DENSE_INDEX_BATCH_SIZE	4 in the packaged Kamiwaza topology
INGEST_EMBED_LATENCY_BUDGET_SECONDS	120.0

The bulk budget is deliberately separate from the interactive query-embedding budget (SEARCH_EMBED_LATENCY_BUDGET_SECONDS, 5.0). That separation is what stops a large reindex from making interactive search feel slow. Uploaded-document reindexing also caps each physical platform request at four inputs so a CPU embedding deployment completes within Kamiwaza's upstream request deadline; later batches continue under the same 120-second Kaizen budget.

Interactive connector sync keeps two document-embedding lanes in flight in the packaged Kamiwaza topology. That preserves useful parallelism without saturating the reference single-node platform's external-authorization hop, which can otherwise time out and surface an empty 403. Larger installations may raise the setting after measuring authorization and model latency. Kaizen does not retry or bypass a 403 authorization result.

Contextual RAG

INGEST_CONTEXTUAL_RAG_ENABLED (off by default) generates a short summary per chunk to improve recall on terse content.

It calls a model per chunk. On a large corpus that is a substantial ingest cost. Measure it on a sample before enabling it broadly.

Progress reporting

A connector sync shows an indeterminate count while discovery, fetch, chunking, and embedding run concurrently. Once the walk establishes the final total, the same run switches to exact percentage progress.

A sync sitting without a percentage is working, not stuck.

Worker capacity

document-worker is deliberately larger than the general worker (4 CPU / 4 GB versus 2 / 2) because extraction and embedding dominate.

Setting	Default
WORKER_CONCURRENCY	4
WORKER_JOB_TIMEOUT_SECONDS	43200
DOCUMENT_WORKER_SEPARATE_QUEUE_ENABLED	off

Enable the separate document queue when a large ingestion starts delaying task dispatch. That is precisely the symptom it exists to fix.

Conversation and Project projections

Chat and Project search projections are maintained differently from documents.

The same transaction that stores an index-eligible journal event writes a projection outbox row. Background workers then update or tombstone the passage and Project indexes.

Those indexes are disposable projections. They never become access capabilities — a stale row cannot resurrect a revoked grant, because opening an anchor re-authorizes.

Setting	Purpose
CHAT_HISTORY_PROJECTION_BATCH_SIZE	Rows per pass
CHAT_HISTORY_PROJECTION_LEASE_SECONDS	Worker lease
CHAT_HISTORY_PROJECTION_MAX_ATTEMPTS	Retry ceiling
CHAT_HISTORY_PROJECTION_RETRY_BASE_SECONDS	Backoff base

When to reindex

• After changing the embedding model — vectors from a different model are not comparable. This is mandatory, not optional.
• After a failed or partial sync.
• After a source changes substantially and freshness matters.
• After changing chunking settings, if you want them applied to existing content.
• After an upgrade that changes how text is chunked. This one will not announce itself, so it is the one to watch for.

Expect a reindex to be the heaviest embedding load the deployment ever sees. Schedule it, and watch model capacity while it runs.

info

a sync skips a source whose content has not changed, and it decides that from the source's own revision combined with the selected embedding model — not from the version of Kaizen that chunked it. So an upgrade that improves chunking leaves already-indexed content exactly as it was, indefinitely. New and changed documents get the better chunking while older ones keep the old, and the difference shows up as uneven retrieval quality across a corpus rather than as an error anywhere.

Connector and filesystem sources are recoverable: force a reindex, and the next sync rebuilds them. Uploaded documents are not. An upload is chunked once when it is ingested, and re-ingestion can only be claimed for a document in the failed state — a ready upload has no re-processing path. Removing and re-uploading the file is the only way to re-chunk it.

When release notes call out a chunking or extraction change, treat a reindex as part of that upgrade rather than as optional tuning.

Diagnosing "nothing is searchable"

Work down this list:

1. Is an embedding model assigned, and does it actually serve embeddings? The dominant cause. Look for 404 on the embeddings path in worker logs and "embedding failed; hybrid unavailable" in search. See Models & embeddings.
2. Is the document worker running and Ready?
3. Is the extractor healthy on GET /version?
4. Is the object store reachable? Bytes live there; the backend checks it at startup.
5. Did extraction succeed for the specific document? Status is visible per upload, and a failed import can be retried without re-uploading.
6. Is the member looking at content they can actually see? Zero results can be a correct authorization outcome.

Diagnosing "results are poor"

1. Confirm hybrid retrieval is on — lexical-only recall on paraphrased queries is noticeably worse.
2. Check whether a reranker is selected. With none, Kaizen keeps the fused order and makes no reranking call.
3. Consider raising SEARCH_POOL_SIZE (50) so fusion has more to work with.
4. Consider HNSW_EF_SEARCH (80) — higher recall, more latency.
5. Consider enabling SEARCH_EXPANSION_ENABLED for short queries.
6. Check SEARCH_MAX_PER_DOCUMENT (3) if one large document dominates.

Related

• Retrieval & ranking
• Models & embeddings
• Scaling & performance

Global reference-index schema dependency

Upload intent and recovery check exact object references across all owners, including soft-deleted documents. Deploy this candidate only after the static shared native/terminal migration prefix through 0140_multipart_terminal_claims and 0141_uploaded_blob_key, followed by the shared static merge revisions 0142_attachment_task_merge, 0143_multipart_task_merge, and 0144_uploaded_blob_task_merge. These additive no-op revisions compose the current task-grant branch; they do not rewrite any applied migration or change stored rows. Continue through 0145_profile_lineage_repair, 0146_multipart_history_merge, and 0147_uploaded_history_merge, then the single current head 0148_stream_recovery_candidates. The guarded forward repair supplies required capture-generation and retrieval-only columns when an original attachment-input stamp skipped their earlier DDL; incompatible schemas fail closed. Its downgrade retains those required columns. The two later merge revisions perform no schema or data changes. Revision 0148 adds the partial stream-recovery candidate index described above; it does not change document rows. See Observability for the historical-schema upgrade conditions. The four prerequisite revisions are unchanged from the native/terminal backlog; this upload branch does not activate the corresponding native product features.

Revision 0141 adds a nonunique, nonpartial PostgreSQL hash index on the existing TEXT blob key. Equality retains the exact raw-key predicate and PostgreSQL rechecks index matches; foreign and deleted references remain protected. Hash indexing also accepts opaque long legacy keys without imposing a new text-length restriction. Schedule the normal schema upgrade with an appropriate maintenance window: index creation can hold a table lock and scan existing rows, and local tests do not establish deployed migration timing or zero downtime. Downgrading only 0141 removes the index and preserves document rows and the shared prerequisite schema.