Observability & Operations

Per-tenant telemetry, because a healthy fleet average hides one customer in trouble (product §17 Observability). Logs, metrics, traces, SLOs, incidents.

The three signals (all tenant-labeled)

• Logs — structured JSON, every line carries tenant_id, user_id, request_id. Ship to Cloud Logging; export to a tamper-evident sink for audit. Redact secrets/PII.
• Metrics — Cloud Monitoring + OpenTelemetry: request rate, p50/p95/p99 latency, error rate, Cloud Run instance count, Cloud SQL connections/CPU, Mongo ops, agent tokens, ingest lag — all per tenant.
• Traces — OpenTelemetry spans across api → DB → Mongo → Claude, so a slow chat turn is attributable (retrieval vs. model vs. DB).

SLOs & alerting

• Define SLOs per plan tier (e.g. enterprise 24×7: 99.9% uptime, p99 < 250ms). The control plane surfaces uptime/p99/error per tenant.
• Alert on per-tenant breach (not just fleet aggregate): error-rate spike, p99 over SLO, Cloud SQL connection saturation (the demoed Helvetia incident), ingest stalls, auth anomalies.
• Alerts open incidents with severity (SEV-1..4), shown in the control plane and routed (PagerDuty/Opsgenie).

Audit vs. observability

Keep them separate: observability = operational health (may sample, may expire); audit = security/compliance record (append-only, complete, retained, exportable). Break-glass and privileged actions go to audit, not just logs.

Cost & capacity

Per-tenant cloud cost feeds the §17 margin view. Usage signals (API/ingest/storage/seats) drive capacity planning and the "this tenant will need a bigger tier" signal.

Runbooks

Connection saturation, ingest backlog, region outage, failover test, noisy-neighbor (shouldn't happen in silos but check shared infra). Each: detect → diagnose → mitigate → verify → post-mortem.

Decided

• Cloud Ops + OpenTelemetry (D10).
• SLO targets per plan (pinned, tune later): Starter 99.5%/p99<400ms · Standard 99.9%/p99<300ms · Pro & Enterprise 99.9%/p99<250ms (Enterprise 24×7 P1 response).

Gaps to close

• Pick the incident router (PagerDuty/Opsgenie).
• Set log/metric retention vs. audit retention windows.