How to Build a policy Q&A Agent Using AutoGen in Python for lending

A policy Q&A agent for lending answers questions like “Can this borrower qualify under our current debt-to-income policy?” or “What documents are required for a self-employed applicant?” It matters because loan ops teams waste time searching policy manuals, while inconsistent answers create compliance risk, bad customer experiences, and avoidable escalations.

Architecture

• •User interface layer
  • •Web app, internal chat tool, or loan officer portal that sends policy questions and displays answers.
• •Retriever
  • •Pulls relevant policy snippets from a controlled source such as PDF manuals, SharePoint exports, or a vector index.
• •AutoGen assistant agent
  • •Generates the response using retrieved policy context and a strict lending policy prompt.
• •Validator / reviewer agent
  • •Checks for unsupported claims, missing citations, and risky language before the answer is returned.
• •Audit logger
  • •Stores question, retrieved passages, answer, model version, timestamps, and reviewer outcome for compliance review.
• •Policy store
  • •Versioned repository of lending policies with effective dates and jurisdiction tags.

Implementation

1) Install AutoGen and define the agents

Use the current AutoGen API from autogen-agentchat. The pattern below uses AssistantAgent, UserProxyAgent, and a simple retrieval function you control.

import os
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.messages import TextMessage
from autogen_ext.models.openai import OpenAIChatCompletionClient

model_client = OpenAIChatCompletionClient(
    model="gpt-4o-mini",
    api_key=os.environ["OPENAI_API_KEY"],
)

policy_agent = AssistantAgent(
    name="policy_agent",
    model_client=model_client,
    system_message=(
        "You answer lending policy questions using only the provided context. "
        "If the context is insufficient, say what is missing. "
        "Do not invent policy. Include citations to the provided excerpts."
    ),
)

reviewer_agent = AssistantAgent(
    name="reviewer_agent",
    model_client=model_client,
    system_message=(
        "You are a compliance reviewer for lending policy answers. "
        "Check whether the answer is supported by context, mentions uncertainty when needed, "
        "and avoids regulatory or underwriting claims not present in the source text."
    ),
)

2) Retrieve policy context before calling the agent

In production, replace this stub with your vector search or document store lookup. The key point is that AutoGen should receive grounded context, not raw user prompts alone.

def retrieve_policy_context(question: str) -> str:
    # Replace with vector DB / keyword search / document retrieval.
    if "self-employed" in question.lower():
        return (
            "[POLICY-2024-07 §3.2] Self-employed borrowers must provide two years of "
            "personal tax returns and year-to-date profit-and-loss statements.\n"
            "[POLICY-2024-07 §3.2] Alternative documentation is not permitted unless approved "
            "by Credit Policy."
        )
    if "debt-to-income" in question.lower():
        return (
            "[POLICY-2024-07 §4.1] Maximum back-end DTI is 43% for standard conventional loans.\n"
            "[POLICY-2024-07 §4.1] Exceptions require documented compensating factors and manager approval."
        )
    return "[POLICY-2024-07] No matching excerpt found."

3) Generate an answer, then run a reviewer pass

This is the pattern you want: retrieve first, answer second, review third. That keeps the assistant anchored to source text and gives you a place to enforce lending-specific guardrails.

import asyncio

async def answer_policy_question(question: str) -> str:
    context = retrieve_policy_context(question)

    prompt = f"""
Question:
{question}

Policy excerpts:
{context}

Instructions:
- Answer only from the excerpts.
- Cite each factual claim using the excerpt labels.
- If the excerpts do not contain enough information, say so explicitly.
"""

    draft = await policy_agent.on_messages(
        [TextMessage(content=prompt, source="user")],
        cancellation_token=None,
    )

    review_prompt = f"""
Question:
{question}

Draft answer:
{draft.chat_message.content}

Policy excerpts:
{context}

Task:
Return either APPROVED or REJECTED with a short reason.
Reject if any claim is unsupported or if the answer omits necessary uncertainty.
"""
    review = await reviewer_agent.on_messages(
        [TextMessage(content=review_prompt, source="user")],
        cancellation_token=None,
    )

    if "REJECTED" in review.chat_message.content.upper():
        return (
            "I can't return this answer as-is because it failed compliance review.\n"
            f"Review note: {review.chat_message.content}"
        )

    return draft.chat_message.content


if __name__ == "__main__":
    result = asyncio.run(answer_policy_question(
        "What documents do we need for a self-employed borrower?"
    ))
    print(result)

4) Add audit logging around every response

For lending workflows, you need to reconstruct why an answer was given. Log both inputs and outputs with versioned metadata.

from datetime import datetime
import json

def audit_event(question: str, context: str, answer: str, model_name: str):
    record = {
        "timestamp_utc": datetime.utcnow().isoformat(),
        "question": question,
        "retrieved_context": context,
        "answer": answer,
        "model": model_name,
        "policy_version": "POLICY-2024-07",
    }
    with open("policy_audit_log.jsonl", "a", encoding="utf-8") as f:
        f.write(json.dumps(record) + "\n")

Production Considerations

• •Keep data residency explicit
  • •If your lending policies or borrower data must stay in-region, pin your model endpoint and retrieval infrastructure to approved regions only.
• •Log every decision path
  • •Store question text, retrieved passages, final answer, reviewer result, and policy version. Auditors will ask how an answer was produced.
• •Add hard guardrails for regulated content
  • •Block advice on credit decisions unless it comes from approved policy text. For anything ambiguous, force escalation to a human underwriter or compliance analyst.
• •Monitor drift in both retrieval and generation
  • •Track unanswered questions, rejected reviews, citation coverage, and top failing queries. Policy changes often break retrieval before they break generation.

Common Pitfalls

1. •

   Letting the model answer without retrieved policy text

   • •Fix: always pass source excerpts into AssistantAgent and reject answers that lack citations.
2. •

   Using one generic prompt for all lending questions

   • •Fix: separate prompts by domain such as underwriting rules, documentation requirements, exceptions handling, and adverse action language.
3. •

   Skipping version control on policies

   • •Fix: tag every excerpt with effective date and version ID so you can prove which rule set was used at response time.
4. •

   Treating review as optional

   • •Fix: make the reviewer agent or rule-based validator mandatory before any customer-facing response leaves the system.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.