The Canonical Model

The single most important concept in insigz. Every project, whatever its domain, runs the same fusion pipeline; every surface, agent and report reads from it.

SOURCE → OBSERVATION → ENTITY → EVENT → CASE → REPORT

The AI analyst is grounded at the Observation layer — it can only cite real observations, never invent facts. The canonical invariant: every table in every tenant is one of a source registry, observation log, entity table, event log, case binding, or report manifest. No exceptions, no escape hatches.

The Observation schema (concrete contract)

One shape for every fact, from every source. This is the atomic unit the AI is grounded on and lineage is tracked at.

{
  "id": "OBS-50412",                      // canonical id
  "source_id": "spire-ais",               // FK to catalog connector
  "observed_property": "vessel_position", // typed enum; governs `value` shape
  "observed_at": "2027-02-17T14:33:07Z",  // EVENT time (when the fact was true), UTC
  "ingested_at": "2027-02-17T14:33:11Z",  // SYSTEM time (when we received it), UTC
  "entity_ref": "VESSEL-SF12",            // resolved entity, or null
  "entity_candidate": null,               // candidate id when unresolved / low-confidence
  "geo": { "lat": 59.81, "lon": 24.77, "alt_m": null, "geohash": "ud9…" }, // WGS84 | null
  "value": { "sog_kn": 0.2, "cog_deg": 184 },        // typed payload; schema per observed_property
  "source_native": { "mmsi": 209123456, "name": "SHADOW STAR" }, // raw fields, preserved for replay
  "provenance": {
    "connector_type": "ws",               // ws | http | file | manual
    "signature": "ed25519:…",            // Ed25519 signature over the canonical record
    "classification": "OPEN",             // handling caveat → drives entitlement/visibility
    "resolution": { "method": "deterministic", "confidence": 1.0, "keys_used": ["mmsi"] }
  }
}

Backend design points:

• Bitemporal (observed_at vs ingested_at). This is what powers the AsOf control (doctrine Q2): "the world as it looked at time T, given what we knew at T" is a query over these two axes. Never collapse them.
• source_native is never discarded. Normalization is additive, not destructive — lineage and replay require the raw fields.
• signature + classification on every record. A degraded/untrusted feed can be down-weighted or quarantined without corrupting the picture; access is scoped at the row.
• value is polymorphic by observed_property. Maintain a registry of property types (vessel_position, throughput_reading, sanction_designation, flight_track, literature_assertion, …), each with its own value sub-schema. A genuinely new property type is part of a Q9 ontology diff.

The reference SQL (from the public Docs page) keys observations on a source_id FK, a nullable entity_id, observed_at timestamptz, geom geometry(Point,4326), a jsonb payload, and a bytea signature (an Ed25519 signature over the canonical record — the normalized tuple (source_id, observed_property, observed_at, entity_ref, geom, payload, classification), signed with a per-tenant key; see engineering 20/42). The signature is what makes the audit log defensible.

Companion shapes (abbreviated)

// Entity — a thing observations are about
{ "id":"VESSEL-SF12", "type":"VESSEL",
  "resolution_keys": { "mmsi":"209123456", "imo":"9…", "callsign":null },
  "tombstone": { "display_name":"SHADOW STAR", "first_seen":"…", "last_seen":"…", "freshness":"fresh" },
  "attributes": { "flag": { "value":"PA", "source_ref":"OBS-…" } },   // conflicts kept, not overwritten
  "links": [ { "type":"owned_by", "target_id":"ENTITY-Q42-12", "source_ref":"OBS-…" } ],
  "flags": { "sanctions_exposure": true } }

// Event — fused from observations
{ "id":"EVENT-7720", "type":"ais_dark_loiter",
  "window": { "from":"…", "to":"…" },
  "bound_observations": ["OBS-50412","OBS-50413"],
  "bound_entities": ["VESSEL-SF12","VESSEL-CAND-8841"],
  "detected_by":"rule", "confidence":0.81, "status":"proposed" }  // proposed | confirmed | dismissed

Entity resolution (where fusion creates — or destroys — value)

Rules are declared per entity type, proposed by the Ontology Agent at onboarding, approved by the Steward.

Declared keys per type

• Deterministic identity keys — exact-match, globally unique, ordered by priority (e.g. VESSEL: imo > mmsi > callsign; ORG: lei > ofac_id; CABLE: asset_id).
• Probabilistic signals — used only when no deterministic key hits (name similarity, attribute overlap, geo+time proximity, kinematic plausibility).

Runtime algorithm (per incoming Observation)

1. try deterministic keys in priority order
     unique hit                 → attach, confidence 1.0, method = deterministic
2. else block + score (probabilistic)
     block on a coarse key (geo-cell + time window, or normalized name prefix)
     score candidates on weighted signals → s ∈ [0,1]
3. decide by band:
     s ≥ AUTO   (≈0.92)         → attach, method = probabilistic
     REVIEW ≤ s < AUTO (≈0.65)  → create candidate + MERGE-REVIEW task   (no silent merge · Q9)
     s < REVIEW                 → create NEW candidate entity

Lifecycle, conflicts, reversibility

• Candidate → confirmed. Probabilistic / unresolved attaches create candidate entities (e.g. VESSEL-CAND-8841); promotion is an analyst/Steward action, logged.
• Merge & split are first-class, audited, reversible. Resolution is never destructive; a merge can be undone, restoring both lineages.
• Attribute conflicts are kept, not overwritten. Disagreeing sources both retained with source_ref; the entity surfaces the freshest value and flags the conflict. No last-writer-wins; truth is adjudicated by a human.
• Thresholds are per-type and tunable. AUTO/REVIEW bands live on the entity type, version-controlled with the ontology.

In-product, review-band decisions surface as a merge-review alert in the 🔔 inbox (see 09-notifications-and-ux.md) — no silent merges.

Framework lineage (positioning, for pitches & research)

The chain is a deliberate synthesis of the two dominant traditions, not a bespoke invention. (Documented publicly at knowledge.html#lineage and docs.html#lineage.)

Defense / IC data fusion — the JDL/DFIG model. The reference framework for intelligence data fusion. Our stages map almost one-to-one:

JDL level                      insigz stage
L0  source / signal refinement  Source, Observation
L1  object refinement           Entity
L2  situation refinement        Event
L2/L3 situation & threat        Case
L3  impact assessment           Report
L4  process refinement          the platform itself (ingestion, tasking, audit)

Entity-centric ontology — the commercial tradition. Object types (entity/event), properties, link types, mapped from source data. insigz grounds its AI analyst at the observation/entity layer — the same move ontology platforms made when the ontology became the backbone for AI agents (~2023). The analyst cites [OBS-####] rather than asserting unsupported facts.

Why "Case" signals heritage. In pure sensor-fusion vocabulary "Case" isn't a stage — it belongs to investigative / compliance / OSINT practice (case management, sanctions work, link analysis). A deliberate, recognizable choice for our maritime/sanctions/academic users; for a hard-defense audience the same stage reads as "investigation."

Honest caveat. There is no single canonical industry pipeline; the value is the clean synthesis. Fusion isn't strictly linear (any level can run on another's output) — the diagram is a communication device. What earns expert credibility is the properties around the chain: provenance/lineage, entity-resolution quality, grounding, auditability, and time-travel reconstruction (AsOf).