How to Build a policy Q&A Agent Using CrewAI in Python for pension funds

A policy Q&A agent for pension funds answers member, trustee, and operations questions from approved policy documents, scheme rules, investment mandates, and internal procedures. It matters because the wrong answer can create compliance risk, breach fiduciary obligations, or trigger an inconsistent member experience across call centers, portals, and back-office teams.

Architecture

• •

  Policy ingestion layer

  • •Pulls PDFs, Word docs, HTML policy pages, and internal memos into a controlled corpus.
  • •Normalizes document text and metadata like effective date, jurisdiction, scheme name, and version.
• •

  Retrieval layer

  • •Finds the most relevant policy sections for each question.
  • •Uses chunking plus metadata filters so the agent does not mix old scheme rules with current ones.
• •

  Answering agent

  • •Generates a grounded response from retrieved policy excerpts only.
  • •Must cite source documents and refuse to answer when evidence is missing.
• •

  Compliance guardrail

  • •Checks for restricted topics like financial advice, benefit guarantees, or legal interpretation beyond policy text.
  • •Forces escalation to a human reviewer when confidence is low or the question touches regulated advice.
• •

  Audit trail

  • •Stores the user question, retrieved chunks, final answer, model version, and timestamps.
  • •Needed for trustee reporting, complaint handling, and regulator review.
• •

  Deployment boundary

  • •Runs in a region approved for pension data residency.
  • •Keeps member data out of public tools and enforces tenant-level isolation.

Implementation

1. •Install CrewAI and define the document-backed tools

For a pension fund use case, don’t let the agent “know” policy from memory. Give it explicit retrieval tools that search your approved corpus.

from crewai import Agent, Task, Crew
from crewai_tools import tool

POLICY_DOCS = [
    {
        "title": "Member Benefit Policy",
        "section": "Early retirement",
        "text": "Members may retire from age 55 subject to trustee approval and actuarial reduction."
    },
    {
        "title": "Contribution Policy",
        "section": "Employer contributions",
        "text": "Employer contributions are paid monthly by bank transfer on the last business day."
    },
]

@tool("search_policies")
def search_policies(query: str) -> str:
    """Return matching policy excerpts for a pension fund question."""
    matches = []
    q = query.lower()
    for doc in POLICY_DOCS:
        haystack = f"{doc['title']} {doc['section']} {doc['text']}".lower()
        if any(term in haystack for term in q.split()):
            matches.append(f"{doc['title']}::{doc['section']} -> {doc['text']}")
    return "\n".join(matches) if matches else "NO_MATCH"

2. •Create a compliance-first answering agent

Use Agent with strict instructions. The key pattern is: answer only from retrieved evidence, cite sources, and escalate when the policy text does not support a direct answer.

policy_qa_agent = Agent(
    role="Pension Fund Policy Q&A Specialist",
    goal="Answer pension fund policy questions using only approved source documents.",
    backstory=(
        "You work for a pension administrator. "
        "You must avoid financial advice, legal interpretation beyond the source text, "
        "and any unsupported claims."
    ),
    tools=[search_policies],
    verbose=True,
    allow_delegation=False,
)

3. •Define the task with explicit refusal rules

The task should force grounded output. For pension funds this is non-negotiable: if there is no supporting excerpt, the agent should say so instead of guessing.

question = "Can a member retire at age 54?"

policy_task = Task(
    description=(
        f"Answer this pension policy question: {question}\n"
        "Use only information returned by search_policies.\n"
        "If there is no supporting evidence or the question requires advice outside policy text, "
        "say you cannot confirm from current policies and recommend human review.\n"
        "Include cited source snippets in your response."
    ),
    expected_output=(
        "A concise answer with citations to source snippets or an escalation note if unsupported."
    ),
    agent=policy_qa_agent,
)

4. •Run the crew and wrap it with audit logging

This is where you make it production-shaped. Capture inputs and outputs so every answer can be traced back during complaints or audits.

import json
from datetime import datetime
from pathlib import Path

audit_log = Path("audit_log.jsonl")

crew = Crew(
    agents=[policy_qa_agent],
    tasks=[policy_task],
    verbose=True,
)

result = crew.kickoff()

record = {
    "timestamp": datetime.utcnow().isoformat(),
    "question": question,
    "answer": str(result),
}
with audit_log.open("a", encoding="utf-8") as f:
    f.write(json.dumps(record) + "\n")

print(result)

Production Considerations

• •

  Deploy inside your approved data boundary

  • •Pension records often fall under strict residency rules.
  • •Keep document stores, logs, and model calls in-region where required by your regulatory framework.
• •

  Add deterministic guardrails before generation

  • •Block questions that request personalized retirement advice unless a licensed workflow handles them.
  • •Use rule-based checks for terms like “should I transfer,” “best fund,” or “guaranteed payout.”
• •

  Store full audit context

  • •Log retrieved passages, document versions, prompt template version, model name, and final output.
  • •For trustees and regulators, you need to show why an answer was produced.
• •

  Monitor refusal rates and unsupported queries

  • •A rising refusal rate can mean stale policies or broken retrieval.
  • •Track which schemes or topics fail most often so you can fix coverage gaps.

Common Pitfalls

1. •

   Letting the agent answer from general model knowledge

   • •This creates hallucinated policy answers.
   • •Avoid it by forcing retrieval-first prompts and refusing any answer without source snippets.
2. •

   Mixing old and current scheme rules

   • •Pension funds often have legacy sections that still look valid but are superseded.
   • •Add metadata filters for effective date, scheme code, jurisdiction, and document status before retrieval.
3. •

   Treating compliance as a prompt-only problem

   • •Prompt wording alone will not stop risky outputs.
   • •Add hard controls: topic classification, human escalation paths, immutable logs, and approval workflows for sensitive queries.
4. •

   Ignoring data residency in tool design

   • •If your retriever or vector store sits in the wrong region, you can create a regulatory issue even if the LLM is compliant.
   • •Keep embeddings stores and audit logs aligned with your pension fund’s hosting requirements.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.