Production AI
An AI agents roadmap for financial services teams
Most programmes fail because they start with the model — not the workflow. This guide is the sequence we use when a bank or insurer asks for agents without yet knowing what “done” looks like in production: where latency hurts, where audit trails matter, and where humans still own the decision.
Team workshop — aligning product, risk, and operations before any model work
1. Map workflows before you name tools
List the top workflows where errors or delays hit revenue, capital, or customer trust — not “AI opportunities,” but repeatable processes with owners.
| Question | Why it matters |
|---|---|
| Who approves an exception today? | Your agent must escalate to the same role. |
| What is the SLA per channel? | Latency budgets drive architecture (sync vs async). |
| Which documents are legally binding? | Grounding scope follows contract and policy corpora. |
This roadmap assumes one vertical slice at a time (for example, Tier‑1 policy Q&A for contact centre agents), not a generic assistant for everyone.
2. Ground answers or do not ship
In regulated Q&A, answers must cite approved sources. If the model cannot point to a chunk your compliance team recognises, the behaviour should be escalation, not improvisation.
Concrete patterns we apply:
- Separate corpora for customer-facing policy, internal runbooks, and draft or regional variants.
- Enforce routing rules at retrieval time — never rely on the model to “know” which library to use.
- Store citations (document id, section, version) in the same payload you log to your audit stream.
3. Evaluation is a product surface
Golden questions, regression suites, and red-team probes belong on the release train, not in a one-off research spike.
| Layer | Example artefact |
|---|---|
| Offline | Versioned JSONL of question → expected citations |
| Shadow | Live traffic mirrored to new prompts without customer impact |
| Live | Drift alerts when citation rate or escalation rate moves |
4. Observability before scale
Production agents need the same incident posture as payment APIs: structured logs, traces, and dashboards tied to business KPIs.
Minimum viable telemetry:
- Request id end-to-end (portal → orchestration → LLM → audit sink).
- Model name + version + prompt hash (not raw prompts in logs unless policy allows).
- Retrieved chunk ids and human override reason codes.
5. Governance without freezing delivery
Risk and legal partners sign off on policies and gates, not on every pull request. Pair each increment with:
- A short change note (what corpus or prompt behaviour changed).
- Evidence from eval runs and shadow periods.
- A rollback lever (prompt version, routing rule, or feature flag).
Pulling it together
Financial services agents win when workflow truth, retrieval boundaries, and measurement are designed together. Models are replaceable; the operating model is not.
Bottom line: Ship a thin slice with citations and telemetry first. Scale only after your second-line team can replay a decision without SSH — and your COO can name the metric that moved.
Further reading on the site: browse the blog index for Brief issues on metrics and retrieval — same themes, shorter format.