RAG & Context Assembly (how context gets into the AI)

The pipeline that turns "a question" into "a grounded, citation-checked answer." This is the RAG file system and the context window builder. It is the single most important AI file — read with ai/30 and 32.

Two phases: index-time and query-time

M0 store (D1): the corpus lives in Postgres pgvector — a rag_chunks table (chunk text, embedding vector(1024), entity_refs[], obs_refs[], classification, position) with an HNSW/IVFFlat index, in the same canonical DB. So retrieval is RLS-scoped like everything else, vector + structured search are one transaction, and there's no cross-store consistency to manage. The Mongo/$vectorSearch form in 22 is the post-M0 target; swap the vector retriever behind the same packages/rag interface when you migrate.

INDEX-TIME (ingest)                          QUERY-TIME (a chat turn)
─────────────────────                        ───────────────────────
document/observation                         user question + scope (tenant, role, case, entity, AsOf)
  → extract text (PDF/HTML→text)               → understand intent (classify: lookup vs analysis vs draft)
  → CHUNK (~500 tok, ~15% overlap)             → RETRIEVE candidates (hybrid)
  → EMBED each chunk                           → FILTER by tenant + classification + cell
  → store in Mongo rag_chunks (+vector)        → RERANK + dedupe
  → keep obs_refs/entity_refs on each          → BUDGET into the context window
                                               → ASSEMBLE prompt (system + grounding block + tools)
                                               → CALL Claude (stream)
                                               → VALIDATE citations → return / regenerate

Index-time: building the corpus

1. Extract. Normalize source → text (PDFs via a text extractor, HTML stripped, structured feeds rendered to a canonical sentence form). Store the original in GCS, the text in Mongo documents.
2. Chunk. ~500-token chunks with ~15% overlap, split on semantic boundaries (headings/paragraphs) not mid-sentence. Carry entity_refs, obs_refs, classification, position.
3. Embed. Call the embedding model (see 32); cache by content-hash (embeddings_cache) to avoid re-embedding identical text.
4. Store. Write rag_chunks with the vector → Atlas Vector Search index (22).

Chunks always retain their canonical links, so any retrieved snippet can be re-grounded to a real observation/entity in Postgres.

Query-time: retrieval (hybrid)

All three retrievers normalize results to one context-item shape before fusion — { id, kind: OBS|DOC|EVT, text, score, refs } — so heterogeneous candidates (structured rows, vector chunks, keyword hits) can be ranked together. Fusion is RRF-only for v0 (defer the cross-encoder reranker). The assembler runs three retrievers and fuses them (Reciprocal Rank Fusion):

async function retrieve(q, scope) {
  const [vec, kw, structured] = await Promise.all([
    vectorSearch(q.embedding, { tenant_id: scope.tid, classification: scope.allowedClasses, entity_refs: scope.entity, k: 40 }),
    textSearch(q.text,        { tenant_id: scope.tid, classification: scope.allowedClasses, k: 40 }),   // Mongo text / BM25
    structuredFetch({ tenant_id: scope.tid, entity: scope.entity, case: scope.case, since: scope.since, until: scope.asof }) // Postgres: recent observations/events
  ]);
  return rrfFuse([vec, kw, structured]);   // one ranked list of candidates
}

• Filters are mandatory and security-bearing: tenant_id always; classification/cell from the caller's clearance (so wargame cell isolation and OPEN/secret boundaries hold in retrieval, see security/41, 43).
• Structured retrieval pulls the facts (recent observations, events, the case's bound items) directly from Postgres — these are first-class context, not just vector hits, so the model always has the authoritative current state, not only documents.
• AsOf caps ingested_at/observed_at so a historical question retrieves only what was known then.

Query-time: assembling the context window

A token budget allocates the window deliberately (don't just stuff top-k):

SYSTEM PROMPT            (fixed)   — role, grounding contract, output schema, doctrine
TASK + SCOPE            (small)    — the question, the active case/entity, AsOf, the user's role
STRUCTURED FACTS        (~30%)     — current entities/events/observations as compact rows w/ [OBS-…] ids
RETRIEVED EVIDENCE      (~45%)     — top reranked chunks, each prefixed with its [OBS-…]/[DOC-…] id
CONVERSATION           (~15%)     — prior turns (summarized if long)
TOOL DEFINITIONS        (fixed)    — the typed tools the agent may call (ai/30)
RESERVE FOR OUTPUT      (~10%)     — leave room for the answer

Rules the assembler enforces:

• Every evidence block is labeled with its canonical id so the model can cite it and the validator can resolve it.
• Dedupe near-identical chunks; diversify across sources (don't let one document dominate).
• Budget defaults (v0): structured facts ~30% · retrieved evidence ~45% · conversation ~15% · reserve ~10%. Compress older conversation with a rolling summary once it exceeds the conversation budget; never silently truncate mid-evidence.
• If retrieval returns nothing in scope, the model is instructed to say so — never to invent.

Query-time: generation + citation validation

1. Stream the Claude call with tools (32). The agent may call query_observations/search_corpus mid-turn to pull more — those calls re-enter retrieval with the same scope/filters.
2. On completion, the citation validator parses [OBS-####]/[DOC-…] tokens, resolves each against Postgres/Mongo for the tenant, and:
   • all resolve → return (citations become clickable provenance links in the UI);
   • any dangling/out-of-scope → regenerate once with a corrective note; if still bad, return a "could not ground" state rather than an unverified claim.

Where context comes from, in one sentence

The model's context is authoritative facts from Postgres (structured) + semantically-retrieved evidence from Mongo/Atlas (vectors) + the conversation, all tenant- and clearance-filtered, budgeted into the window with every block carrying a canonical id, and every output claim re-checked against the stores before it ships.

Gaps to close

• Lock chunk size/overlap and the embedding model/dims (D5). Reranker decided: RRF-only v0, cross-encoder later.
• Tune budget percentages per agent (v0 defaults above) and the summarization trigger threshold.
• Specify the citation grammar + validator regex shared with ai/30.
• Decide caching of retrieval results and the latency budget per turn (p95 target).