How to Build a policy Q&A Agent Using LlamaIndex in Python for pension funds

A policy Q&A agent for pension funds answers staff and member questions against internal policy documents, scheme rules, investment guidelines, and compliance manuals. It matters because pension operations are document-heavy, regulated, and sensitive to error; a bad answer can create compliance breaches, member harm, or audit issues.

Architecture

• •

  Document ingestion layer

  • •Pulls PDFs, Word docs, and policy handbooks from approved storage.
  • •Normalizes content into text chunks with source metadata.
• •

  Indexing layer

  • •Uses LlamaIndex to build a searchable knowledge base.
  • •Stores chunk-level citations so every answer can point back to the source document.
• •

  Retrieval layer

  • •Retrieves only the most relevant policy sections for each question.
  • •Applies filters for scheme, jurisdiction, policy version, and effective date.
• •

  Answer synthesis layer

  • •Uses an LLM through LlamaIndex to generate grounded answers.
  • •Forces responses to stay within retrieved context.
• •

  Audit and logging layer

  • •Records question, retrieved nodes, answer, timestamps, and document versions.
  • •Supports compliance review and incident investigation.
• •

  Guardrail layer

  • •Blocks unsupported advice, speculative language, and requests outside policy scope.
  • •Routes high-risk queries to human review.

Implementation

1) Install dependencies and load documents

Use SimpleDirectoryReader for a local proof of concept. In production, replace this with a controlled loader from your DMS or object store.

from llama_index.core import SimpleDirectoryReader

docs = SimpleDirectoryReader(
    input_dir="./pension_policies",
    recursive=True
).load_data()

print(f"Loaded {len(docs)} documents")

Each document should carry metadata like scheme_name, jurisdiction, policy_version, and effective_date. That metadata becomes important when you need to prove which version informed an answer.

2) Build the index with citation-friendly chunks

For pension funds, chunking matters. Policies often have definitions, exceptions, and thresholds that should stay together. VectorStoreIndex.from_documents() is the standard entry point for a retrieval-first agent.

from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(chunk_size=800, chunk_overlap=120)

index = VectorStoreIndex.from_documents(
    docs,
    transformations=[splitter]
)

query_engine = index.as_query_engine(similarity_top_k=4)

This gives you a basic retrieval pipeline. If your corpus is large or regulated across multiple jurisdictions, back the index with a persistent vector store rather than keeping it in memory.

3) Add metadata-aware retrieval for policy scope

A pension fund agent should not mix policies from different schemes or outdated versions. Use metadata filters so the retriever only searches the right subset of documents.

from llama_index.core.vector_stores import MetadataFilters, ExactMatchFilter
from llama_index.core.indices.vector_store.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine

filters = MetadataFilters(filters=[
    ExactMatchFilter(key="scheme_name", value="UK Defined Benefit"),
    ExactMatchFilter(key="jurisdiction", value="UK"),
])

retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=4,
    filters=filters,
)

query_engine = RetrieverQueryEngine.from_args(retriever)

response = query_engine.query(
    "What is the pension commencement lump sum limit?"
)

print(response)

This pattern is useful when one fund has multiple schemes or when policy differs by country. It also reduces cross-contamination between legacy rules and current rules.

4) Return answers with citations and keep an audit trail

For regulated environments, plain answers are not enough. You need source-backed responses and logs that show what was asked and what evidence was used.

from llama_index.core.response.pprint_utils import pprint_response

response = query_engine.query(
    "Can members transfer out before normal retirement age?"
)

pprint_response(response)

for i, node in enumerate(response.source_nodes):
    print(f"\nSource {i+1}")
    print(node.node.metadata)
    print(node.node.get_content()[:500])

In production, write these artifacts to an immutable audit store:

• •user question
• •response text
• •retrieved node IDs
• •document names and versions
• •timestamp
• •model name and prompt version

Production Considerations

• •

  Data residency

  • •Keep embeddings, indexes, and raw documents in-region if your pension fund operates under local residency rules.
  • •Don’t send member data or restricted policy content to unmanaged external endpoints.
• •

  Compliance controls

  • •Restrict the agent to informational Q&A; do not let it give legal advice or override policy wording.
  • •Add escalation paths for questions involving benefit disputes, protected characteristics, or exceptions.
• •

  Monitoring

  • •Track retrieval quality: low-confidence queries, empty retrievals, and repeated fallback responses.
  • •Review sample outputs weekly against source documents to catch drift after policy updates.
• •

  Auditability

  • •Version every document ingestion run.
  • •Store the exact prompt template and model configuration used for each response so compliance can reproduce outcomes later.

Common Pitfalls

1. •

   Mixing outdated policies with current ones

   • •This happens when you ingest everything into one index without metadata filters.
   • •Fix it by filtering on effective date, scheme name, jurisdiction, and version before retrieval.
2. •

   Letting the model answer outside the retrieved context

   • •Pension users will ask edge-case questions that invite speculation.
   • •Fix it by using retrieval-first prompts and rejecting answers when supporting nodes are weak or missing.
3. •

   Skipping audit logging

   • •In regulated environments, “the model said so” is useless during review.
   • •Fix it by storing question text, sources cited, response text, timestamps, and document hashes in an immutable log.

A solid pension policy Q&A agent is not just a chatbot over PDFs. It is a controlled retrieval system with traceability baked in from day one. If you get indexing discipline, metadata filtering, and audit logging right, you can ship something useful without creating compliance debt.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.