How to Build a policy Q&A Agent Using AutoGen in Python for pension funds

A policy Q&A agent for a pension fund answers staff, trustees, and operations teams with grounded responses from internal policy documents, scheme rules, and regulatory guidance. It matters because pension operations are full of edge cases: eligibility, contribution limits, benefit options, disclosure rules, and jurisdiction-specific compliance requirements all need consistent answers with an audit trail.

Architecture

• •User interface layer
  • •A chat API or internal portal where staff ask questions like “Can a deferred member transfer out after age 55?”
• •Policy retrieval layer
  • •A document store plus retrieval tool that pulls the relevant policy sections, scheme rules, and circulars.
• •AutoGen agent layer
  • •A primary assistant agent that reasons over retrieved context and produces the answer.
• •Compliance checker agent
  • •A second agent that verifies the answer against pension rules, internal policy constraints, and forbidden advice boundaries.
• •Audit logging
  • •Store the question, retrieved sources, model response, and final decision for review by compliance teams.
• •Guardrail layer
  • •PII redaction, prompt-injection filtering, jurisdiction checks, and “answer only from sources” enforcement.

Implementation

1. •Install AutoGen and define your agents

For a production-style setup, use AssistantAgent for answering and UserProxyAgent for orchestration. In AutoGen’s current Python API, you can wire agents together with a group chat or direct conversation pattern.

from autogen import AssistantAgent, UserProxyAgent

llm_config = {
    "model": "gpt-4o-mini",
    "api_key": "YOUR_OPENAI_API_KEY",
    "temperature": 0,
}

policy_agent = AssistantAgent(
    name="policy_agent",
    llm_config=llm_config,
    system_message=(
        "You answer pension fund policy questions using only provided context. "
        "If the context is insufficient, say what is missing. "
        "Do not provide financial advice. Cite the policy section used."
    ),
)

user_proxy = UserProxyAgent(
    name="user_proxy",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=1,
)

2. •Add a retrieval function that injects policy context

AutoGen agents work best when you give them narrow context. For pension funds, that usually means retrieving the exact scheme rule or policy memo before generating any answer.

POLICY_DB = {
    "transfer_out_age": (
        "Scheme Rule 7.2: Deferred members may request a transfer value "
        "before retirement age subject to statutory checks and trustee approval."
    ),
    "benefit_payment": (
        "Operations Policy 4.1: Monthly pension payments are processed on the "
        "last business day of each month."
    ),
}

def retrieve_policy_context(question: str) -> str:
    q = question.lower()
    if "transfer" in q or "age" in q:
        return POLICY_DB["transfer_out_age"]
    if "payment" in q or "pension" in q:
        return POLICY_DB["benefit_payment"]
    return "No matching policy found."

question = "Can a deferred member transfer out after age 55?"
context = retrieve_policy_context(question)

prompt = f"""
Question: {question}

Policy context:
{context}

Answer using only the policy context.
"""

3. •Run the answer flow and capture an audit record

For a real system, log the input question, retrieved policy text, model output, timestamp, and source identifiers. Pension funds need this for complaint handling, trustee review, and regulator queries.

import json
from datetime import datetime

chat_result = user_proxy.initiate_chat(
    policy_agent,
    message=prompt,
)

answer_text = chat_result.chat_history[-1]["content"]

audit_record = {
    "timestamp": datetime.utcnow().isoformat(),
    "question": question,
    "retrieved_context": context,
    "agent_name": policy_agent.name,
    "answer": answer_text,
}

print(json.dumps(audit_record, indent=2))

4. •Add a compliance reviewer agent for high-risk answers

For pension funds, not every answer should go straight to users. Questions about transfers, tax treatment, retirement options, protected benefits, or jurisdictional differences should pass through a second agent that checks whether the draft answer stays inside policy.

compliance_agent = AssistantAgent(
    name="compliance_agent",
    llm_config=llm_config,
    system_message=(
        "You are a compliance reviewer for a pension fund. "
        "Check whether the draft answer is consistent with the supplied policy context. "
        "Flag missing citations, unsupported claims, advice language, or ambiguity."
    ),
)

draft_answer = answer_text
review_prompt = f"""
Question: {question}

Policy context:
{context}

Draft answer:
{draft_answer}

Review this answer for compliance issues.
"""

review_result = user_proxy.initiate_chat(
    compliance_agent,
    message=review_prompt,
)

print(review_result.chat_history[-1]["content"])

Production Considerations

• •

  Data residency

  • •Keep member data and policy documents in-region if your fund operates under local residency rules.
  • •If you use hosted LLM APIs, confirm where prompts and logs are processed and stored.
• •

  Auditability

  • •Store every retrieved document ID and version number alongside the final answer.
  • •Trustees will want to know which rule set was used when an employee asked about eligibility or transfer rights.
• •

  Guardrails

  • •Block answers when retrieval confidence is low or no authoritative source is found.
  • •Add redaction for names, member numbers, tax IDs, and medical information before sending prompts to any model.
• •

  Monitoring

  • •Track unanswered questions, compliance-review failures, escalation rates, and source mismatch frequency.
  • •In pension environments these metrics matter more than raw latency.

Common Pitfalls

1. •

   Letting the model answer without source grounding

   • •This is how you get plausible but wrong guidance on vesting dates or transfer conditions.
   • •Fix it by requiring retrieved context in every prompt and refusing to answer when no source is found.
2. •

   Using one agent for both answering and compliance

   • •A single agent will often rationalize its own mistakes.
   • •Fix it by separating draft generation from review using distinct AssistantAgent roles.
3. •

   Ignoring jurisdiction and scheme-specific rules

   • •Pension policies differ by country, employer plan design, union agreements, and tax regime.
   • •Fix it by tagging every document with jurisdiction metadata and filtering retrieval before prompting.
4. •

   Skipping operational logging

   • •Without logs you cannot reconstruct why an employee received a particular response.
   • •Fix it by persisting question text, retrieved snippets, model output, reviewer output, and timestamps on every interaction.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.