insigz docs
Engineering

Data-Source Integration & the Responsibility Boundary

insigz integrates any source through the canonical contract, but the customer owns the data rights, cost, credentials, and lawful authority.

insigz can integrate any source a customer needs — it all normalises through the canonical contract (20) via the Ontology Agent (product 06, engineering 30). But the sources themselves are the customer's responsibility: they choose, license, pay for, and lawfully authorise the feeds and supply the access; insigz builds the pipeline and hosts compliant to their needs (54). This is a load-bearing commercial + legal boundary — encoded, not just stated.

The principle

We integrate the data; the customer owns the data rights.

Any feed — public API, commercial licence, file drop, message stream, the customer's own database, a sensor, or a classified feed in an enclave (54) — enters as canonical Observations with signed provenance and a classification. insigz provides the connector, the normalisation, the provenance, the isolation, and the hosting. The customer provides the lawful right to use the data, the subscription/cost, and the access credentials.

Responsibility split (who owns what)

ConcernCustomerinsigz
Choosing the sourcesadvises
Paying for the data licence / subscription— (not resold unless an explicit reseller deal)
Legal right & authority to ingest and have insigz process it✔ (warranted in contract)acts as processor
Compliance of the source use with its terms, export control, sanctions, privacy lawscreens + flags; refuses unlawful asks
Supplying credentials / endpoints / sample payloads✔ (out-of-band, 42)operates them; never owns the root credential
Building & operating the connector
Normalising to the canonical model + observed_property typing✔ (agent drafts, Steward commits — Q9)
Provenance signing, source_native retention, lineage
Classification / RLS / isolation enforcementsets the marking✔ enforces (43, 82)
Hosting to the customer's residency / compliance posturestates the requirement✔ delivers (54)
Renewing / maintaining the subscription & rotating its keyreminds; operates rotation

"Any source" — why the universality actually holds

The canonical contract is the universal adapter: every source collapses to the same Observation shape (20), so there is no bespoke pipeline per feed — one ingestor pattern (fetch → normalise → geolocate → sign → write), one schema downstream. The only thing a new feed can touch in the shared model is a new observed_property or entity type, and that goes through the Q9 Steward gate as a versioned ontology_diff. That is what lets insigz say "yes" to an arbitrary customer feed without forking the platform.

BYO-source onboarding (extends the Ontology Agent 8-stage flow, product 06)

A customer-brought source adds two things to the standard wizard — an attestation at intake and an ownership flag at catalog:

0  Intake + LICENCE & AUTHORITY ATTESTATION   ← customer attests: we hold the right to use this
                                                 feed and to have insigz process it; the cost is ours
1  Profile        detect protocol / fields / PII / classification
2  Connector      draft endpoint + auth (secret REFERENCED, never embedded — 42)
3  Mapping        field → Observation; observed_property; UTC / WGS84 normalise
4  Ontology fit   reuse existing types > propose new (anti-fork)
5  Dry-run        replay sample, no commit; show resolved / new / dedup / conflicts
6  Steward gate   human commits the diff (Q9) — logged
7  Catalog        register with entitlement, classification, residency, OWNERSHIP = customer
(8 Connect)       Project Lead connects it into a project

Step 0's attestation and step 7's ownership = customer flag are the responsibility boundary made concrete in the audit log.

Credentials & secrets (the access the customer gives us)

  • The customer supplies credentials out-of-band; they land in the tenant's secret store / KMS (42, and in BYOK/HYOK profiles the customer's own KMS — 54). They are referenced by the connector, never embedded in config or code.
  • insigz operates the credential (calls the feed) but never takes ownership of the customer's licence; rotation is automated, but the obligation to keep the subscription valid stays with the customer.
  • A customer's licensed feed is never shared across tenants — isolation (43) applies to brought data exactly as to everything else; nothing a customer paid for leaks into the shared connector catalog as reusable.

Compliance posture (contract + technical)

  • The customer warrants in the agreement that it holds the lawful right to ingest each source and to have insigz process it; insigz is the processor, bound by the DPA, hosting to the stated residency / classification (54).
  • insigz runs sanctions / export-control screening on sources and refuses feeds that are unlawful, breach a source's terms, or require classified / illegitimate access insigz isn't authorised to touch. This is a hard product boundary, not legal advice.
  • Every observation keeps source_native + signature + classification, so the customer always has a defensible, exportable lineage of exactly what was ingested and from where (20).

What insigz will NOT do

  • Procure or pay for the customer's data licences (unless an explicit reseller arrangement).
  • Assume liability for the customer's source licensing or legality.
  • Bypass a source's terms of use or its access controls.
  • Reuse one tenant's brought / licensed feed for another tenant.
  • Acquire classified data on the customer's behalf — in enclave profiles the customer brings it (54).

Decisions to ratify (proposed — extend the register in 00)

  • Responsibility boundary: customer owns source rights / cost / credentials; insigz owns integration / normalisation / provenance / hosting — encoded as the intake attestation + the ownership catalog flag.
  • Brought-source isolation: customer-licensed feeds never enter the shared catalog as reusable; they are tenant-private.
  • Refusal boundary: sanctions / export / terms screening can block a source at the Steward gate.

Gaps to close

  • Draft the licence-&-authority attestation wording (legal) and where it's captured (intake step 0 + audit row).
  • Define the catalog metadata for a brought source: ownership, licence_ref, classification, residency, cost_owner, renewal date.
  • Specify the screening checklist (sanctions / export / terms) the Steward sees at the gate.
  • Decide the reseller exception path (when insigz does provide / meter a feed) and how it's billed.