How to Build a policy Q&A Agent Using AutoGen in Python for fintech

A policy Q&A agent for fintech answers internal questions about policies, controls, procedures, and regulatory rules without making people hunt through PDFs or wiki pages. It matters because support, operations, risk, and engineering teams need fast answers that are consistent, auditable, and grounded in approved source material, not model guesswork.

Architecture

• •User interface layer
  • •Slack bot, web app, or internal API that receives policy questions.
• •Retriever
  • •Pulls approved policy chunks from a controlled knowledge base.
  • •Use only curated documents: AML, KYC, PCI DSS, SOC 2, retention, incident response.
• •Assistant agent
  • •Generates the answer using retrieved context.
  • •Must be constrained to cite sources and refuse unsupported claims.
• •Reviewer / compliance agent
  • •Checks the draft answer for policy drift, missing citations, or risky language.
  • •Useful for fintech controls where auditability matters.
• •Human escalation path
  • •Routes ambiguous or high-risk questions to compliance or legal.
• •Audit logger
  • •Stores question, retrieved sources, final answer, model version, and timestamps for review.

Implementation

1) Install AutoGen and set up a controlled model client

For production work, use AutoGen’s AssistantAgent with an explicit model client configuration. Keep the model choice centralized so you can pin versions and route traffic by environment.

import os
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

model_client = OpenAIChatCompletionClient(
    model=os.environ["OPENAI_MODEL"],
    api_key=os.environ["OPENAI_API_KEY"],
)

policy_agent = AssistantAgent(
    name="policy_qna_assistant",
    model_client=model_client,
    system_message=(
        "You answer fintech policy questions using only provided context. "
        "If the context is insufficient, say you don't know and recommend escalation. "
        "Always cite the source titles in your answer."
    ),
)

That system message is doing real work. In fintech, you want refusal behavior by default because a confident wrong answer about retention periods or KYC exceptions is a control failure.

2) Add retrieval before generation

AutoGen does not magically solve retrieval. You still need a controlled retrieval layer that returns only approved policy text.

from dataclasses import dataclass
from typing import List

@dataclass
class PolicyChunk:
    title: str
    text: str

POLICY_CORPUS: List[PolicyChunk] = [
    PolicyChunk("AML Policy v4", "Suspicious activity must be escalated within 24 hours..."),
    PolicyChunk("Data Retention Standard", "Customer records must be retained for 7 years..."),
    PolicyChunk("PCI DSS Access Control", "Access to cardholder data is restricted to least privilege..."),
]

def retrieve_policy_context(question: str) -> str:
    hits = []
    q = question.lower()
    for chunk in POLICY_CORPUS:
        if any(term in q for term in chunk.text.lower().split()[:8]):
            hits.append(f"[{chunk.title}] {chunk.text}")
    return "\n\n".join(hits) if hits else ""

question = "How long do we retain customer records?"
context = retrieve_policy_context(question)

In a real system this function should query your vector store or search index with document-level ACLs. Do not retrieve from raw drive folders or uncontrolled Confluence spaces.

3) Run the agent with grounded context and an explicit fallback

Use run() on the assistant with the retrieved context embedded into the prompt. If nothing relevant comes back, return an escalation response instead of letting the model improvise.

from autogen_agentchat.messages import TextMessage

def answer_policy_question(question: str) -> str:
    context = retrieve_policy_context(question)

    if not context:
        return (
            "I don't have enough approved policy context to answer this safely. "
            "Please escalate to Compliance or Risk."
        )

    prompt = f"""
Question:
{question}

Approved policy context:
{context}

Instructions:
- Answer only from the approved policy context.
- Cite the source titles inline.
- If there is any ambiguity, say so clearly.
"""

    result = policy_agent.run(task=prompt)
    return result.messages[-1].content

print(answer_policy_question("How long do we retain customer records?"))

This pattern keeps the assistant bounded by your internal sources. For fintech teams, that boundary is what makes the difference between an assistant and an uncontrolled liability.

4) Add a reviewer agent for a second pass

A second agent can inspect answers for missing citations or unsafe statements before release. This is especially useful when you need stronger controls around compliance wording.

reviewer = AssistantAgent(
    name="policy_reviewer",
    model_client=model_client,
    system_message=(
        "Review answers to fintech policy questions. "
        "Flag unsupported claims, missing citations, regulatory overreach, "
        "and anything that sounds like legal advice."
    ),
)

review_prompt = """
Review this draft answer for compliance risk:

Draft answer:
Customer records must be retained for seven years according to our Data Retention Standard.

Check whether it includes citations and whether it overstates certainty.
"""

review_result = reviewer.run(task=review_prompt)
print(review_result.messages[-1].content)

A reviewer agent is not a substitute for legal review. It is a guardrail that catches obvious misses before they reach users.

Production Considerations

• •Deployment
  • •Keep retrieval services inside your tenant boundary.
  • •Enforce region-specific hosting if data residency rules apply.
  • •Separate dev/test/prod models and keys; never share endpoints across environments.
• •Monitoring
  • •Log question text, retrieved chunks, final answer, latency, refusal rate, and escalation rate.
  • •Track hallucination indicators such as answers without citations or answers that mention absent policies.
• •Guardrails
  • •Require source citations in every response.
  • •Block responses when retrieval returns no approved context.
  • •Add allowlists for document types; exclude drafts and personal notes.
• •Fintech controls
  • •Store audit logs immutably for review by compliance and internal audit.
  • •Redact PII before sending prompts to external models where possible.
  • •Map each policy domain to an owner so updates are reviewed before indexing.

Common Pitfalls

1. •

   Letting the model answer without retrieval

   • •This turns your Q&A bot into a generic chatbot with no control boundary.
   • •Fix it by refusing unanswered queries unless approved source chunks are present.
2. •

   Indexing unapproved content

   • •Draft policies, old SOPs, Slack threads, and personal docs create conflicting answers.
   • •Fix it with strict ingestion rules and document ownership checks.
3. •

   Skipping auditability

   • •If you cannot show which source informed an answer, you cannot defend it during review.
   • •Fix it by logging source titles, chunk IDs, model version, prompt hash, and timestamp.
4. •

   Ignoring data residency and PII

   • •Fintech teams often handle regulated customer data even in internal workflows.
   • •Fix it by masking sensitive fields before prompting and keeping regional processing aligned with policy requirements.

A good fintech policy Q&A agent is boring in the right way: grounded answers, clear refusals, traceable sources. Build it with retrieval first, generation second, and compliance hooks everywhere else.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.