Skip to main content

Retrieval & ranking

Kaizen's answers are only as good as what it can retrieve. This page covers the matching pipeline itself: where knowledge comes from, how candidates are found, and why authorization is part of the query rather than a filter over its results.

Trace matching branches through current authorization, deterministic fusion, optional reranking, and bounded evidence projection.

Two kinds of knowledge​

  • The shared corpus — your organization's collective knowledge, ingested from connected sources. Visibility is governed by each source's own permissions.
  • Your documents — files a member uploaded. Private to that member, indexed separately, and never mixed into the shared corpus.

In Chat, a selected uploaded file or folder is an exact set of document identifiers, not an inline dump of extracted text. The shared turn resolver checks current visibility for every identifier in one database query and registers that visible set with query_document. The tool searches ranked chunks across the entire set and returns citable passages. The model prompt shows only a bounded sample of file pointers; files omitted from that sample remain searchable. Querying a folder does not enumerate all its contents as evidence.

The conversation reference records whether it came from a library selection. That marker persists across turns and prevents selected image bytes from entering the vision prompt. Directly uploaded images keep their separate vision path. Standalone library images without extracted passages cannot be visually inspected through document retrieval.

One pipeline, several providers​

The Search page, the Projects and Chats library filters, and eligible agents all use one server-owned matching pipeline. Providers contribute documents, uploads, Projects, and chat history; one deterministic merge ranks the final authorized projections.

This prevents a title-only browser search from contradicting the main Search page — a disagreement that would be an access-control defect, not a relevance one.

Home, Chat, and Search suggestions sample titles through the same document ACL predicate. Repository-level connected knowledge is eligible once its active sync projection is readable; item-level tickets, pull requests, discussions, findings, and compliance records are excluded to avoid turning suggestions into an operational queue.

The matching branches​

Kaizen does not pick one strategy. It combines:

BranchRole
BM25 lexicalStrong term relevance, when the PostgreSQL text-search substrate is available
GIN full-textThe built-in lexical fallback
HNSW vectorBounded semantic candidate discovery, in indexed mode
Exact vector distanceThe authorization oracle and the indexed-retrieval rollback mode
FuzzyOptional near-miss fallback (SEARCH_FUZZY_FALLBACK_ENABLED, on by default)

Authorized lexical and semantic survivors are combined with deterministic reciprocal-rank fusion, then exact-reranked where configured.

Uploaded-file Search keeps native PostgreSQL keyword matches alongside vector matches when hybrid retrieval is disabled or pg_textsearch is unavailable. Files at lexical_ready remain discoverable before optional dense indexing. Both branches check current document visibility before ranking results: the viewer owns the file, it is shared globally, or an owner-authored, still-shared Project pointer lends it through the viewer's active membership. Direct passage reads, representative passages, conversation attachments and citation resolution apply the same rule, so a file the Library shows a viewer is never silently missing from its passages.

An administrator can optionally select a Kamiwaza reranker deployment. With none selected, Kaizen keeps the fused order and makes no reranking call.

Ranked evidence is not corpus enumeration​

Search and RAG return a bounded, ranked evidence set. They are appropriate for answering questions, reviewing likely-relevant passages, and discovering an exact file for a later authorized tool call. A successful search does not prove that every file, CSV row, or chunk in a folder was examined.

General-purpose corpus transformations — for example, building and reconciling a 38,000-node ontology from mixed documents — require deterministic input enumeration, bounded partitions, checkpoints, retries, and a coverage manifest. Those workloads run as durable Tasks and persist intermediate state outside model context. RAG remains useful inside the workflow for evidence selection and review; it does not replace the exhaustive enumeration layer.

The trusted PostgreSQL boundary​

Current workroom, owner, direct, and Project access, and Project lifecycle checks live inside the repository query. Every lexical, semantic, exact, and fuzzy branch must consume that relation before branch output, fusion, snippets, facets, counts, cache writes, API projection, audit detail, or model context.

Raw candidate IDs and text never cross into application code for post-filtering. This is not a stylistic preference: post-filtering leaks through result counts, facet totals, pagination, and error messages even when the rows themselves are withheld.

The HNSW nuance, stated plainly​

PostgreSQL's HNSW access method can inspect index rows before relational filters are fully applied. Those rows may influence internal index traversal, so Kaizen does not claim the method is authorization-prefiltered.

What Kaizen does instead:

  • each indexed branch joins the full current authorization relation inside PostgreSQL,
  • transaction-local iterative-scan settings are bounded,
  • only authorized survivors are exposed, and exact reranking consumes only those survivors.

With CHAT_HISTORY_INDEXED_RETRIEVAL_ENABLED off, Kaizen first materializes the complete eligible relation and then performs lexical scoring and exact vector distance — it does not traverse the protected HNSW, BM25, or GIN indexes in this strict exact mode.

Current viewer projection​

Search passages and Project documents are disposable projections, not authority. A result is admitted only when the viewer has a current room-access path. Direct access to a chat does not reveal a linked Project's metadata unless the viewer is also an active Project member.

Opening an anchor re-authorizes it. A stale card or a copied URL cannot preserve a revoked grant. Other independent providers may still return results, which the Search page labels as partial.

Typed authority​

Result kindAuthorityMeaning
Document, uploaded documentEvidenceMay be cited as fact
Chat, ProjectNavigationMay be linked to; never enters the citation ledger

This distinction is what stops a historical conversation excerpt from becoming grounding for a new answer merely because it matched a query.

Shared-chat room content​

Project or direct membership grants the complete retained room: messages, artifacts, citations, and linked resources. It does not transfer unrelated private libraries, live connector credentials, tools, sandboxes, packages, roles, confirmation state, MFA, IT approval, or another person's agent authority.

Each turn executes under its initiating participant and records the agent, model, and effort used.

Keeping knowledge fresh​

  • Live connector retrieval is evaluated again on each agent turn.
  • Privately synced connector chunks carry an opaque member/workroom/ connector grant that Kaizen revalidates against Kamiwaza on every retrieval.
  • Conversation and Project search projections are updated or tombstoned through a durable transactional outbox — the same transaction that stores an index-eligible journal event writes the outbox row.
  • Passage and project embeddings are produced only through Kaizen's model gateway, on the worker's bulk budget. Interactive query embedding keeps its own separate member-API latency budget.

Uploaded HTML follows the same retrieval pipeline after extraction. Tika provides the visible plain-text projection; Kaizen additionally parses valid JSON from JSON script elements and static const, let, var, window, or globalThis assignments. It renders nested objects and arrays as bounded, record-oriented text before chunking. No browser or JavaScript evaluator runs during ingestion. Data assembled only by executing application code remains outside the index unless the source also carries it as a valid JSON literal.

Text recovered from a picture inside a document is indexed as part of that document rather than as a separate unit. The extractor appends it at the end of the page the picture sits on, under a marker naming the figure, its page, and the picture's own name, so it chunks, ranks, and cites exactly like the prose around it — a hit on wording that appears only inside a photograph resolves to the page a member would turn to. Page separators are untouched by that fold, which is what keeps the page arithmetic behind a citation correct.

An uploaded ZIP or TAR remains one owner-scoped document. The worker projects bounded UTF-8 member text into separate Markdown sections headed by each relative entry path. Archive sections are kept separate during chunking so a search hit carries the entry path in source_loc.section; citations still resolve to the authorized original archive. Binary members and nested archives never become chunks. XML markup is indexed without resolving external entities.

Those indexes remain disposable projections. They never become access capabilities.

Ingestion knobs​

SettingDefaultPurpose
INGEST_MINI_CHUNKS_ENABLEDonFine-grained chunks alongside passages
INGEST_MINI_CHUNK_TARGET_TOKENS64Mini-chunk size
INGEST_MINI_CHUNK_OVERLAP_TOKENS16Overlap between mini-chunks
INGEST_EMBED_BATCH_SIZE16Embedding batch
INGEST_EMBED_MAX_BATCH_TOKENS4096Token ceiling per batch
INGEST_EMBED_LATENCY_BUDGET_SECONDS120Bulk embedding ceiling
INGEST_CONTEXTUAL_RAG_ENABLEDoffAdds a generated summary per chunk
EMBEDDING_CACHE_TTL_SECONDS604800Cached query embeddings
DOCUMENT_EMBEDDED_FIGURE_MAX_COUNT20Pictures read per document
DOCUMENT_EMBEDDED_FIGURE_TEXT_MAX_CHARS20000Recovered picture text per document

Contextual RAG improves recall on terse chunks at real ingest cost — it calls a model per chunk. Measure before enabling it on a large corpus.