AI Agents for payments: How to Automate fraud detection (multi-agent with AutoGen)

Payments fraud teams are drowning in alerts, manual case reviews, and false positives. The real problem is not detecting every suspicious transaction; it’s triaging millions of events fast enough to stop losses without freezing legitimate card-not-present payments, ACH transfers, and wallet top-ups.

Multi-agent systems built with AutoGen fit this problem well because fraud detection is not one decision. It’s a chain of decisions: enrich the transaction, score risk, compare against historical patterns, check policy, and decide whether to approve, step-up authenticate, or route to manual review.

The Business Case

• •

  Reduce manual review volume by 30-50%

  • •A mid-size payments processor handling 5-10 million transactions/day can often cut analyst queue load by filtering obvious low-risk cases and prioritizing only the highest-risk disputes, chargebacks, and authorization anomalies.
  • •That usually saves 3-6 FTEs per 10 million monthly transactions in operations and fraud ops.
• •

  Cut fraud decision latency from minutes to seconds

  • •Today, many teams batch enrich alerts or wait for downstream rules engines.
  • •An agent workflow can get enrichment, network signals, device reputation, velocity checks, and policy lookup done in 1-3 seconds, which matters for auth-time decisions on card-present and card-not-present flows.
• •

  Lower false positives by 10-20%

  • •In payments, false positives are expensive: lost interchange revenue, abandoned checkout, support calls, and merchant churn.
  • •A multi-agent setup can reduce over-blocking by combining rules with contextual analysis instead of relying on a single threshold score.
• •

  Reduce investigation cost per case by 25-40%

  • •If an analyst spends 8-12 minutes per alert today, agent-generated summaries and evidence bundles can bring that down to 5-7 minutes.
  • •At scale, that’s meaningful for PSPs and acquirers running lean fraud teams under tight SLA pressure.

Architecture

A production setup should not be “LLM decides fraud.” It should be a controlled workflow with clear ownership per step.

• •

  1) Ingestion and feature layer

  • •Stream auth events, chargeback signals, device fingerprints, merchant metadata, BIN data, IP intelligence, and customer history into Kafka or Kinesis.
  • •Store structured features in Postgres or a feature store like Feast.
  • •Use pgvector for retrieval of prior case notes, similar fraud patterns, and merchant-specific playbooks.
• •

  2) Multi-agent orchestration

  • •Use AutoGen for agent-to-agent coordination.
  • •Keep the workflow explicit:
    • •Enrichment Agent pulls context
    • •Risk Analyst Agent evaluates pattern similarity
    • •Policy Agent checks internal rules and regulatory constraints
    • •Decision Agent recommends approve/step-up/review
  • •If you want stricter control flow and retries, wrap the agents in LangGraph rather than letting them free-run.
• •

  3) Deterministic scoring + LLM reasoning

  • •Do not replace your existing fraud engine.
  • •Combine traditional models such as XGBoost or isolation forests with agent reasoning over unstructured evidence like dispute notes or merchant complaint text.
  • •The LLM should explain why a transaction looks risky; it should not be the only scorer.
• •

  4) Audit and governance layer

  • •Log every prompt, tool call, retrieved record ID, model version, and final action.
  • •Store immutable audit trails in S3 + DynamoDB/Postgres with retention policies aligned to SOC 2 controls.
  • •For regulated markets, keep model outputs explainable enough for internal audit and external review under GDPR access requests or banking model risk management expectations tied to Basel-style governance.

Reference stack

What Can Go Wrong

Regulatory risk

If an agent makes or influences adverse decisions without traceability, you create problems under GDPR data rights obligations and internal model governance standards. In payments adjacent environments that touch healthcare reimbursement cards or benefit-linked payment flows, HIPAA-adjacent handling expectations can also show up if protected data leaks into prompts.

Mitigation:

• •Redact PII before sending data to the LLM.
• •Keep the final approval/decline rule deterministic for high-value transactions.
• •Maintain model cards, prompt logs, and decision rationale for audit.

Reputation risk

Blocking legitimate customers at checkout is how you lose merchants. One bad rollout can spike decline rates on a major merchant account and trigger support escalations within hours.

Mitigation:

• •Start in “shadow mode” where agents recommend but do not act.
• •Put hard caps on auto-decline rates by merchant segment.
• •Require human approval for high-value or first-party fraud edge cases until precision is proven.

Operational risk

Agent chains can fail in messy ways: stale retrieval data, tool timeouts, hallucinated explanations that sound plausible but are wrong. In payments operations this becomes expensive fast because auth windows are short.

Mitigation:

• •Add timeouts and fallback rules at every step.
• •Use confidence thresholds plus deterministic fallback logic.
• •Run chaos testing on missing data scenarios: absent BIN intel, delayed chargeback feeds, incomplete device fingerprints.

Getting Started

1. •

   Pick one narrow use case

   • •Start with card-not-present authorization reviews for a single merchant vertical or region.
   • •Avoid chargebacks plus AML plus account takeover in the same pilot. That’s three programs pretending to be one.
2. •

   Build a small team

   • •You need:
     • •1 product owner from fraud/risk
     • •1 ML engineer
     • •1 backend engineer
     • •1 data engineer
     • •part-time compliance/legal reviewer
   • •That is enough to ship a pilot in 6-10 weeks if your event pipeline already exists.
3. •

   Run shadow mode for 2-4 weeks

   • •Compare agent recommendations against current rules engine outcomes.
   • •Measure precision at top-k alerts, false positive reduction, analyst time saved per case, and merchant-level approval rate impact.
   • •Do not optimize for model cleverness; optimize for fewer bad escalations.
4. •

   Move to controlled automation

   • •Let the system auto-close low-risk cases first.
   • •Then allow step-up actions like OTP verification or manual review routing.
   • •Only after you prove stability should you let agents influence declines on higher-risk segments.

The right goal here is not “AI replaces fraud ops.” The goal is tighter control over payment risk with less manual work and better customer experience. If you implement AutoGen with strong guardrails, you get a system that behaves like a disciplined analyst team instead of a black box.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.