Data-Source Integration & the Responsibility Boundary
insigz integrates any source through the canonical contract, but the customer owns the data rights, cost, credentials, and lawful authority.
insigz can integrate any source a customer needs — it all normalises through the canonical contract (
20) via the Ontology Agent (product06, engineering30). But the sources themselves are the customer's responsibility: they choose, license, pay for, and lawfully authorise the feeds and supply the access; insigz builds the pipeline and hosts compliant to their needs (54). This is a load-bearing commercial + legal boundary — encoded, not just stated.
The principle
We integrate the data; the customer owns the data rights.
Any feed — public API, commercial licence, file drop, message stream, the customer's own database, a sensor, or a classified feed in an enclave (54) — enters as canonical Observations with signed provenance and a classification. insigz provides the connector, the normalisation, the provenance, the isolation, and the hosting. The customer provides the lawful right to use the data, the subscription/cost, and the access credentials.
Responsibility split (who owns what)
| Concern | Customer | insigz |
|---|---|---|
| Choosing the sources | ✔ | advises |
| Paying for the data licence / subscription | ✔ | — (not resold unless an explicit reseller deal) |
| Legal right & authority to ingest and have insigz process it | ✔ (warranted in contract) | acts as processor |
| Compliance of the source use with its terms, export control, sanctions, privacy law | ✔ | screens + flags; refuses unlawful asks |
| Supplying credentials / endpoints / sample payloads | ✔ (out-of-band, 42) | operates them; never owns the root credential |
| Building & operating the connector | — | ✔ |
Normalising to the canonical model + observed_property typing | — | ✔ (agent drafts, Steward commits — Q9) |
Provenance signing, source_native retention, lineage | — | ✔ |
| Classification / RLS / isolation enforcement | sets the marking | ✔ enforces (43, 82) |
| Hosting to the customer's residency / compliance posture | states the requirement | ✔ delivers (54) |
| Renewing / maintaining the subscription & rotating its key | ✔ | reminds; operates rotation |
"Any source" — why the universality actually holds
The canonical contract is the universal adapter: every source collapses to the same Observation shape (20), so there is no bespoke pipeline per feed — one ingestor pattern (fetch → normalise → geolocate → sign → write), one schema downstream. The only thing a new feed can touch in the shared model is a new observed_property or entity type, and that goes through the Q9 Steward gate as a versioned ontology_diff. That is what lets insigz say "yes" to an arbitrary customer feed without forking the platform.
BYO-source onboarding (extends the Ontology Agent 8-stage flow, product 06)
A customer-brought source adds two things to the standard wizard — an attestation at intake and an ownership flag at catalog:
0 Intake + LICENCE & AUTHORITY ATTESTATION ← customer attests: we hold the right to use this
feed and to have insigz process it; the cost is ours
1 Profile detect protocol / fields / PII / classification
2 Connector draft endpoint + auth (secret REFERENCED, never embedded — 42)
3 Mapping field → Observation; observed_property; UTC / WGS84 normalise
4 Ontology fit reuse existing types > propose new (anti-fork)
5 Dry-run replay sample, no commit; show resolved / new / dedup / conflicts
6 Steward gate human commits the diff (Q9) — logged
7 Catalog register with entitlement, classification, residency, OWNERSHIP = customer
(8 Connect) Project Lead connects it into a projectStep 0's attestation and step 7's ownership = customer flag are the responsibility boundary made concrete in the audit log.
Credentials & secrets (the access the customer gives us)
- The customer supplies credentials out-of-band; they land in the tenant's secret store / KMS (
42, and in BYOK/HYOK profiles the customer's own KMS —54). They are referenced by the connector, never embedded in config or code. - insigz operates the credential (calls the feed) but never takes ownership of the customer's licence; rotation is automated, but the obligation to keep the subscription valid stays with the customer.
- A customer's licensed feed is never shared across tenants — isolation (
43) applies to brought data exactly as to everything else; nothing a customer paid for leaks into the shared connector catalog as reusable.
Compliance posture (contract + technical)
- The customer warrants in the agreement that it holds the lawful right to ingest each source and to have insigz process it; insigz is the processor, bound by the DPA, hosting to the stated residency / classification (
54). - insigz runs sanctions / export-control screening on sources and refuses feeds that are unlawful, breach a source's terms, or require classified / illegitimate access insigz isn't authorised to touch. This is a hard product boundary, not legal advice.
- Every observation keeps
source_native+ signature + classification, so the customer always has a defensible, exportable lineage of exactly what was ingested and from where (20).
What insigz will NOT do
- Procure or pay for the customer's data licences (unless an explicit reseller arrangement).
- Assume liability for the customer's source licensing or legality.
- Bypass a source's terms of use or its access controls.
- Reuse one tenant's brought / licensed feed for another tenant.
- Acquire classified data on the customer's behalf — in enclave profiles the customer brings it (
54).
Decisions to ratify (proposed — extend the register in 00)
- Responsibility boundary: customer owns source rights / cost / credentials; insigz owns integration / normalisation / provenance / hosting — encoded as the intake attestation + the
ownershipcatalog flag. - Brought-source isolation: customer-licensed feeds never enter the shared catalog as reusable; they are tenant-private.
- Refusal boundary: sanctions / export / terms screening can block a source at the Steward gate.
Gaps to close
- Draft the licence-&-authority attestation wording (legal) and where it's captured (intake step 0 + audit row).
- Define the catalog metadata for a brought source:
ownership,licence_ref,classification,residency,cost_owner, renewal date. - Specify the screening checklist (sanctions / export / terms) the Steward sees at the gate.
- Decide the reseller exception path (when insigz does provide / meter a feed) and how it's billed.
MongoDB & the Document / RAG Store
The non-relational half of the polyglot — raw payloads, documents, the RAG corpus, and agent transcripts, with Atlas Vector Search for semantic retrieval.
Agent Architecture
How the AI agents are built server-side - the roster, the grounding contract, tool use, and the human-commit gates.