MongoDB & the Document / RAG Store

⚠ Deferred to post-M0 (D1, see 00 + 02). This file describes the target document/RAG store once Mongo is introduced. For M0–M1 the RAG corpus lives in Postgres pgvector (single-store) — see 31 for how chunks/embeddings map onto pgvector tables instead of the collections below. Build this only when document ingestion outgrows the single store; everything here (collections, Atlas Vector Search, the outbox consistency in 11) is the migration plan, not day-one work.

The non-relational half of the polyglot. Mongo holds what Postgres shouldn't: raw heterogeneous payloads, large documents, the RAG corpus (chunks + embeddings), and agent transcripts. Atlas Vector Search powers semantic retrieval.

Database & collections (per tenant)

One Mongo database per tenant: insigz_<slug>.

raw_payloads        # exactly what a source returned, keyed to the canonical observation
documents           # ingested long-form docs (articles, filings, PDFs' text)
rag_chunks          # chunked text + embedding vector (Atlas Vector Search index)
agent_runs          # full agent transcripts: tool calls, retrieved context, output, citations
embeddings_cache    # content-hash → vector, to avoid re-embedding identical text

raw_payloads

{
  "_id": "obs:50412",                 // matches Postgres observation id
  "tenant_id": "…",
  "source_id": "ais.spire",
  "source_hash": "sha256:…",          // == provenance.source_hash in Postgres
  "received_at": "2026-06-03T14:02:11Z",
  "raw": { /* verbatim payload as received */ }
}

Verbatim raw is kept so the signed source_hash is verifiable and re-normalization is possible.

documents + rag_chunks

// documents
{ "_id":"doc:9931", "tenant_id":"…", "source_id":"reuters", "entity_refs":["VESSEL-SF12"],
  "title":"…", "published_at":"…", "text":"…", "gcs_uri":"gs://…/doc9931.pdf" }

// rag_chunks  (one per chunk)
{ "_id":"doc:9931#0007", "tenant_id":"…", "doc_id":"doc:9931",
  "entity_refs":["VESSEL-SF12"], "obs_refs":[50412],
  "classification":"OPEN",
  "text":"… ~500 token chunk …",
  "embedding": [0.0123, -0.044, …],   // dims match your model
  "tokens": 480, "position": 7 }

Atlas Vector Search index

{
  "fields": [
    { "type": "vector", "path": "embedding", "numDimensions": 1024, "similarity": "cosine" },
    { "type": "filter", "path": "tenant_id" },
    { "type": "filter", "path": "classification" },
    { "type": "filter", "path": "entity_refs" }
  ]
}

Retrieval is a $vectorSearch with mandatory filters on tenant_id and the caller's allowed classification/cell — so semantic search can never cross tenant or clearance boundaries (the same RLS spirit as Postgres, enforced in the query). See ai/31.

Indexes

• rag_chunks: the vector index above; plus { tenant_id:1, entity_refs:1 } and { doc_id:1, position:1 }.
• raw_payloads: { _id } (obs id) and { source_hash:1 }.
• documents: { entity_refs:1, published_at:-1 }, text index on title,text for keyword/hybrid search.
• agent_runs: { tenant_id:1, created_at:-1 }, TTL index if transcripts expire.

Isolation & access

• DB-per-tenant with a Mongo user scoped to only that database; connection string + creds in Secret Manager (per tenant).
• Atlas network: private endpoint / VPC peering to the tenant's VPC; no public listener.
• Encryption: Atlas encryption-at-rest with customer key (KMS) where the tier supports it; TLS in transit.
• Every query in packages/rag injects tenant_id as a filter even though the DB is already tenant-scoped (defense-in-depth).

Consistency with Postgres

Mongo is a projection (see architecture/11): written post-commit via the outbox, always carrying the canonical obs_refs/entity_refs. If Mongo is lost, it is rebuilt by replaying observations/documents from Postgres + GCS. Nothing in Mongo is authoritative.

Hybrid retrieval (recommended)

Combine vector (semantic) + keyword/text + structured (Postgres filters: this entity, this time window, this case) and fuse with Reciprocal Rank Fusion. Pure vector search misses exact ids/codes; pure keyword misses paraphrase. The assembler in ai/31 does this fusion.

Gaps to close

• Pick the embedding model + dimensions (locks the Vector Search index) and the chunking strategy (size/overlap).
• Confirm Atlas tier per plan (Vector Search availability, CMEK support) and project-per-tenant vs DB-per-tenant.
• Decide agent_runs retention (TTL) and whether transcripts are class-sensitive.
• If you choose single-store (pgvector) instead, this whole file collapses into Postgres tables — revisit architecture/11 first.