How to Build a transaction monitoring Agent Using AutoGen in TypeScript for retail banking

A transaction monitoring agent watches incoming retail banking transactions, scores them against policy and behavioral patterns, and escalates suspicious activity for review. It matters because you need fast detection of fraud, AML triggers, and unusual customer behavior without burying analysts in false positives.

Architecture

• •

  Transaction ingestion layer

  • •Pulls card payments, ACH, wire transfers, and internal ledger events from Kafka, SQS, or a database stream.
  • •Normalizes raw events into a strict transaction schema before the agent sees them.
• •

  Policy and rules engine

  • •Applies deterministic checks first: velocity limits, structuring patterns, sanctioned geographies, high-risk merchant categories.
  • •Keeps obvious cases out of the model path.
• •

  AutoGen orchestrator

  • •Coordinates specialized agents using AssistantAgent and UserProxyAgent.
  • •Routes a transaction to fraud analysis, AML analysis, or case summarization.
• •

  Risk scoring and decision layer

  • •Combines rules output, model reasoning, and customer context into a final risk score.
  • •Produces one of: allow, review, or escalate.
• •

  Audit and case storage

  • •Stores every input, intermediate decision, prompt, model output, and final action.
  • •This is mandatory for compliance review and examiner requests.
• •

  Analyst handoff API

  • •Sends suspicious cases to the case management system with a structured summary.
  • •Includes evidence fields that an investigator can verify quickly.

Implementation

1. •

   Define the transaction schema and the agent roles

   Keep the schema strict. In retail banking you do not want free-form JSON drifting across services because auditability breaks fast.

   import { config } from "dotenv";
   import { AssistantAgent, UserProxyAgent } from "@autogen/agentchat";
   import OpenAI from "openai";
   
   config();
   
   type Transaction = {
     transactionId: string;
     accountId: string;
     customerId: string;
     amount: number;
     currency: string;
     channel: "card" | "ach" | "wire" | "cash";
     merchantCategory?: string;
     country: string;
     timestamp: string;
     description?: string;
   };
   
   type MonitoringResult = {
     riskScore: number;
     decision: "allow" | "review" | "escalate";
     reasons: string[];
     auditSummary: string;
   };
   
   const llmConfig = {
     apiKey: process.env.OPENAI_API_KEY!,
     model: "gpt-4o-mini",
   };
   
   const fraudAnalyst = new AssistantAgent({
     name: "fraud_analyst",
     systemMessage:
       "You analyze retail banking transactions for fraud indicators. Return concise JSON only.",
     llmConfig,
   });
   
   const amlAnalyst = new AssistantAgent({
     name: "aml_analyst",
     systemMessage:
       "You analyze retail banking transactions for AML red flags. Return concise JSON only.",
     llmConfig,
   });
   
   const supervisor = new UserProxyAgent({
     name: "supervisor",
     humanInputMode: "NEVER",
     codeExecutionMode: undefined,
   });
   

2. •

   Add deterministic pre-checks before AutoGen runs

   Retail banking teams should not ask an LLM to rediscover basic policy. Use rules first, then let AutoGen handle ambiguous cases.

   function precheck(txn: Transaction): string[] {
     const flags: string[] = [];
   
     if (txn.amount >= 10000) flags.push("large_value");
     if (txn.channel === "wire" && txn.country !== "US") flags.push("cross_border_wire");
     if (txn.merchantCategory === "money_services") flags.push("high_risk_mcc");
   
     return flags;
   }
   

3. •

   Run the multi-agent analysis and force structured output

   The pattern below sends the same transaction to two specialists and then asks a supervisor to consolidate the result. This is the part where AutoGen pays off: you get role separation without building your own orchestration framework.

   export async function monitorTransaction(txn: Transaction): Promise<MonitoringResult> {
     const flags = precheck(txn);
   
     const prompt = `
   

Transaction: ${JSON.stringify(txn, null, 2)}

Precheck flags: ${JSON.stringify(flags)}

Tasks:

1. •

   Fraud analyst returns JSON with keys riskScore (0-100), reasons (string[]), decision ("allow"|"review"|"escalate")

2. •

   AML analyst returns JSON with keys riskScore (0-100), reasons (string[]), decision ("allow"|"review"|"escalate")

3. •

   Supervisor consolidates both into final JSON with keys riskScore, decision, reasons, auditSummary `;

   const fraudReply = await supervisor.initiateChat(fraudAnalyst, { message: ${prompt}\nFocus on fraud only., maxTurns: 1, clearHistory: true, });

   const amlReply = await supervisor.initiateChat(amlAnalyst, { message: ${prompt}\nFocus on AML only., maxTurns: 1, clearHistory: true, });

   const consolidationPrompt = ` Fraud reply: ${JSON.stringify(fraudReply.chatHistory.at(-1)?.content)}

AML reply: ${JSON.stringify(amlReply.chatHistory.at(-1)?.content)}

Return final JSON only. `;

const finalReply = await supervisor.initiateChat(fraudAnalyst, {
  message: consolidationPrompt,
  maxTurns: 1,
  clearHistory: true,
});

const content = String(finalReply.chatHistory.at(-1)?.content ?? "{}");
return JSON.parse(content) as MonitoringResult;

}

4. **Wrap it in a service boundary with audit logging**

You need durable logs for every decision path. In production this means writing prompts and outputs to immutable storage with retention aligned to your bank’s policy.

```typescript
async function handleIncomingTransaction(txn: Transaction) {
  const result = await monitorTransaction(txn);

  console.log(JSON.stringify({
    transactionId: txn.transactionId,
    customerIdMask: txn.customerId.slice(0, 4) + "***",
    result,
    timestamp: new Date().toISOString(),
  }));

  return result.decision;
}

Production Considerations

• •

  Data residency

  • •Keep customer PII in-region.
  • •If your bank requires EU or US-only processing, pin your model endpoint and storage accordingly.
  • •Do not send raw account numbers or full names into prompts unless your legal team has approved that flow.
• •

  Auditability

  • •Persist input payloads, rule hits, agent messages, final decisions, and versioned prompts.
  • •Examiners will ask why a case was escalated; “the model said so” is not acceptable.
• •

  Guardrails

  • •Force structured JSON output and reject malformed responses.
  • •Add schema validation before any downstream action.
  • •Keep human review mandatory for high-value wires and sanctions-adjacent cases.
• •

  Monitoring

  • •Track false positive rate by channel and merchant category.
  • •Watch latency separately for card auth flows versus batch AML jobs.
  • •Alert on prompt drift when outputs start producing vague explanations instead of concrete reasons.

Common Pitfalls

• •

  Letting the LLM replace hard rules

  • •Mistake: asking AutoGen to detect everything from scratch.
  • •Fix: run deterministic checks first for thresholds, geography blocks, sanctions filters, and velocity rules.
• •

  Skipping schema validation

  • •Mistake: trusting raw model text as if it were machine-safe output.
  • •Fix: parse JSON strictly and reject anything that does not match MonitoringResult.
• •

  Ignoring compliance boundaries

  • •Mistake: sending full PII or cross-border data to an external model endpoint without controls.
  • •Fix:
    • •redact sensitive fields before prompting
    • •keep logs in approved regions
    • •version prompts for audit review
    • •involve compliance early when changing thresholds or escalation logic

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.