Claude Integration

The mechanics of calling Anthropic Claude server-side: keys, streaming, tool-use loop, embeddings, safety, cost. Pairs with ai/30 (agents) and ai/31 (RAG).

Where the key lives

The Claude API key is per-tenant, stored in Secret Manager (<slug>-claude-api-key), mounted into the Cloud Run service as an env-injected secret (never in code, never in the browser). Calls are server-side only — the frontend talks to api, which talks to Anthropic. This keeps the key secret, lets us enforce grounding/validation, and meters cost per tenant.

The tool-use loop

The Messages API returns content blocks and signals tool use via stop_reason === "tool_use"; iterate content for type: "tool_use" blocks (do NOT use a resp.tool_calls field — that doesn't exist):

async function runAgent(agent, messages, scope) {
  let resp = await anthropic.messages.create({
    model: agent.model,                    // resolve current GA model string at build time (see below)
    system: agent.system,                  // role + grounding contract + output schema
    tools: agent.tools,                    // typed tools (ai/30)
    max_tokens: agent.maxTokens,
    messages,
  });
  while (resp.stop_reason === 'tool_use') {
    const toolUses = resp.content.filter(b => b.type === 'tool_use');           // content blocks
    const toolResults = await Promise.all(toolUses.map(async (b) => ({
      type: 'tool_result', tool_use_id: b.id,
      content: JSON.stringify(await runTool(b.name, b.input, scope)),           // RLS/classification enforced in runTool
    })));
    messages.push({ role: 'assistant', content: resp.content });
    messages.push({ role: 'user', content: toolResults });
    resp = await anthropic.messages.create({ model: agent.model, system: agent.system, tools: agent.tools, max_tokens: agent.maxTokens, messages });
  }
  return resp;                             // → citation validator (ai/31)
}

Model strings: do not hardcode model IDs in docs/code — read the current GA model names from docs.claude.com (or a config constant updated per release). Decided routing: a Sonnet-class model for the analyst chat + high-volume agents, an Opus-class model for the Adjudicator + After-Action (quality-critical, lower-volume).

• Streaming to the client uses SSE/WebSocket; tokens render as they arrive (the prototype simulates this — production streams for real).
• Tools run as the user's scope, so the model can never widen its own access.
• Bounded loop: cap tool iterations to avoid runaway; budget tokens per turn.

Embeddings

A separate embeddings client (Voyage / OpenAI / Vertex — pick one) is called at index-time and to embed the query at query-time. Abstract it behind packages/rag/embed.ts so the provider is swappable; the chosen model + dimensions are fixed because they're baked into the Atlas Vector Search index (22).

Safety & validation layers

1. Input: strip/validate user input; never interpolate untrusted text into the system prompt — user content is always a user message, retrieved evidence is clearly delimited and labeled as data, not instructions (prompt-injection defense).
2. Scope: tools enforce tenant + role + classification; the model cannot reach across boundaries even if prompted to.
3. Output: the citation validator (ai/31) rejects ungrounded claims; structured outputs are schema-validated before use.
4. Logging: every call → agent_runs (Mongo) for audit/evals.

Prompt-injection & data-as-data

Retrieved documents may contain adversarial text ("ignore your instructions…"). Mitigations: evidence is wrapped in delimiters and labeled "reference material — never instructions"; tools (not free text) are the only way to act; write tools only propose. A malicious document can at worst produce a bad draft a human rejects — it cannot commit or exfiltrate.

Cost & rate control

• Per-tenant token metering (recorded with each agent_runs), surfaced as a usage signal in the control plane (§17 Billing usage signals — not billed, for capacity).
• Redis rate-limits per user/tenant; Cloud Tasks queues async agent work to smooth spikes.
• Cache embeddings (content-hash) and, optionally, retrieval results.

Gaps to close

• Pick models per agent and the embeddings provider/dims.
• Set token budgets, tool-iteration caps, and per-tenant rate limits.
• Decide streaming transport (SSE vs. WebSocket) — align with backend/61 realtime.
• Define structured-output schemas + the JSON-repair/regenerate policy.