Engineering
Secrets & Key Management
Per-tenant secrets and envelope encryption on Google Cloud - no secret ever lives in code, the app database, or the browser.
Per-tenant secrets and envelope encryption on Google Cloud. The rule: no secret ever lives in application code, the app database, or the browser.
What's a secret
Per tenant: Postgres password, Mongo connection string, JWT signing key (Ed25519), the per-tenant Ed25519 provenance signing key (data/20), the Claude API key, embedding-provider key, OAuth/SAML client secrets, webhook signing secrets.
Where they live
- Secret Manager holds each value, namespaced per tenant:
<slug>-db-password,<slug>-jwt-signing-key,<slug>-claude-api-key,<slug>-provenance-key, … - Cloud KMS provides a per-tenant key ring used for envelope encryption (CMEK on Cloud SQL, GCS, and Atlas where supported), so each tenant's data-at-rest is encrypted under its own key.
- Cloud Run mounts secrets as injected env/secret volumes at deploy (
--set-secrets), resolved at boot via the service account — values never appear in images, config files, or logs.
Access control
- Each tenant's Cloud Run service runs as a dedicated service account with
secretAccessoronly on that tenant's secrets — a compromised service can't read another tenant's secrets. - The control plane has no access to tenant DB/Claude secrets; it can create/rotate them during provisioning but the provisioning runner uses a scoped, audited service account, and break-glass into data is separate and logged (§17).
- Humans never read secrets directly; rotation is automated.
Rotation
| Secret | Cadence | Mechanism |
|---|---|---|
| DB password | 90d / on-incident | new version in Secret Manager → rolling Cloud Run restart |
| JWT signing key | 90d | dual-key window (verify old+new) then retire |
| Provenance key | yearly / on-incident | new key; old kept for verification of historical signatures |
| Claude/embedding keys | on-incident / vendor rotation | new version → restart |
Rotation is a control-plane action (the §17 "Rotate secrets" / tenant danger-zone), gated and audited. Keep historical verification keys so old signatures/JWTs still validate during the window.
Hygiene
- Secret scanning in CI (gitleaks/trufflehog) blocks commits that contain credentials (
security/44). - No secrets in logs: redact known keys; structured logging with allowlisted fields.
- Local dev uses a separate secret namespace + dummy keys; never prod secrets on laptops.
.envfiles are git-ignored and never the source of truth in prod.
Gaps to close
- Confirm CMEK coverage (Cloud SQL + GCS + Atlas tier) and the KMS key-ring layout (per-tenant vs. per-region).
- Automate rotation jobs (Cloud Scheduler → control-plane endpoint) and the dual-key JWT window length.
- Decide service-account-per-tenant vs. per-service granularity.