How to Build a policy Q&A Agent Using LangGraph in Python for banking

A policy Q&A agent answers employee or customer questions against bank-approved policy documents, then returns a grounded response with traceable evidence. In banking, that matters because the difference between “what the policy says” and “what the model guessed” is compliance risk, audit failure, and inconsistent customer handling.

Architecture

• •User input node
  • •Accepts the question and any context like product line, region, or customer segment.
• •Policy retrieval layer
  • •Pulls the most relevant policy excerpts from an indexed knowledge base.
  • •In banking, this should be scoped by jurisdiction and business unit.
• •Answer generation node
  • •Uses an LLM to draft a response only from retrieved policy text.
• •Verification / guardrail node
  • •Checks that the answer is supported by citations and does not violate banking rules.
• •Audit logging node
  • •Stores the question, retrieved sources, final answer, model version, and timestamps.
• •Fallback / escalation path
  • •Routes low-confidence or high-risk questions to a human reviewer.

Implementation

1) Define the graph state and build retrieval helpers

For a policy agent, keep state explicit. You want the question, retrieved policy chunks, draft answer, and final answer all visible in the graph so you can inspect each step during audits.

from typing import TypedDict, List
from langchain_core.documents import Document
from langchain_openai import ChatOpenAI
from langgraph.graph import StateGraph, START, END

class PolicyState(TypedDict):
    question: str
    docs: List[Document]
    draft_answer: str
    final_answer: str

policy_corpus = [
    Document(
        page_content="Mortgage applications require income verification and KYC completion before approval.",
        metadata={"policy_id": "MORT-001", "jurisdiction": "US"}
    ),
    Document(
        page_content="Card disputes must be reported within 60 days of statement date.",
        metadata={"policy_id": "CARD-014", "jurisdiction": "US"}
    ),
]

def retrieve_policy(state: PolicyState) -> dict:
    q = state["question"].lower()
    matched = [d for d in policy_corpus if any(word in d.page_content.lower() for word in q.split())]
    return {"docs": matched[:3]}

This example uses in-memory documents so you can see the pattern. In production, replace it with a vector store or search engine that filters by jurisdiction, product type, and permission scope.

2) Add answer generation with LangGraph nodes

LangGraph nodes are just Python callables returning partial state updates. Use ChatOpenAI or your approved internal model endpoint to generate a grounded answer from retrieved documents only.

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def generate_answer(state: PolicyState) -> dict:
    context = "\n\n".join(
        f"[{d.metadata['policy_id']}] {d.page_content}" for d in state["docs"]
    )
    prompt = f"""
You are a banking policy assistant.
Answer only from the provided policy excerpts.
If the policy does not contain enough information, say you need escalation.

Question: {state['question']}

Policy excerpts:
{context}
"""
    response = llm.invoke(prompt)
    return {"draft_answer": response.content}

This is where most teams make mistakes. They let the model improvise; don’t do that. The prompt should force grounding and make “insufficient information” an acceptable outcome.

3) Verify support and build the graph flow

A banking agent needs a check before returning anything to users. At minimum, validate that retrieved sources exist and that high-risk questions are escalated.

def verify_answer(state: PolicyState) -> dict:
    if not state["docs"]:
        return {"final_answer": "I could not find a relevant policy excerpt. Escalating to compliance review."}

    risky_terms = ["override", "bypass", "exception", "waive fee", "approve anyway"]
    if any(term in state["question"].lower() for term in risky_terms):
        return {"final_answer": "This request requires manual review due to potential policy exception handling."}

    citations = ", ".join(d.metadata["policy_id"] for d in state["docs"])
    final = f"{state['draft_answer']}\n\nSources: {citations}"
    return {"final_answer": final}

graph = StateGraph(PolicyState)
graph.add_node("retrieve_policy", retrieve_policy)
graph.add_node("generate_answer", generate_answer)
graph.add_node("verify_answer", verify_answer)

graph.add_edge(START, "retrieve_policy")
graph.add_edge("retrieve_policy", "generate_answer")
graph.add_edge("generate_answer", "verify_answer")
graph.add_edge("verify_answer", END)

app = graph.compile()

result = app.invoke({"question": "What is required before approving a mortgage application?"})
print(result["final_answer"])

This uses StateGraph, add_node, add_edge, compile, invoke, START, and END from LangGraph’s actual API. That’s the core pattern you will use even when you add routing, retries, or human review.

4) Add conditional escalation for banking risk

For real banking workflows, route certain questions to humans based on content or confidence. LangGraph supports conditional edges through add_conditional_edges, which gives you clean branching without stuffing logic into one node.

def route_next(state: PolicyState) -> str:
    q = state["question"].lower()
    if any(term in q for term in ["exception", "waive", "override", "manual approval"]):
        return "escalate"
    return "answer"

def escalate(state: PolicyState) -> dict:
    return {"final_answer": "Escalated to compliance operations for manual review."}

graph = StateGraph(PolicyState)
graph.add_node("retrieve_policy", retrieve_policy)
graph.add_node("generate_answer", generate_answer)
graph.add_node("verify_answer", verify_answer)
graph.add_node("escalate", escalate)

graph.add_edge(START, "retrieve_policy")
graph.add_conditional_edges("retrieve_policy", route_next, {
    "answer": "generate_answer",
    "escalate": "escalate",
})
graph.add_edge("generate_answer", "verify_answer")
graph.add_edge("verify_answer", END)
graph.add_edge("escalate", END)

That branching matters because not every banking question should be answered automatically. Exceptions, waivers, suspicious activity guidance, and regulatory interpretation should go through controlled review.

Production Considerations

• •Use jurisdiction-aware retrieval
  • •Filter policies by country, business line, and customer type before generation.
  • •A US card dispute rule should never bleed into EU servicing guidance.
• •Log everything for audit
  • •Persist input question, retrieved document IDs, final answer, model name/version, latency, and reviewer actions.
  • •Banks need reproducibility when compliance asks why an answer was returned.
• •Add hard guardrails
  • •Block advice on legal interpretation beyond approved policy text.
  • •Escalate requests involving exceptions, fraud handling, sanctions screening, or PII disclosure.
• •Control data residency
  • •Keep embeddings, vector stores, logs, and model calls inside approved regions.
  • •If your bank has residency constraints, your LangGraph app must respect them end-to-end.

Common Pitfalls

• •Using the LLM as the source of truth
  • •Don’t let it answer from memory.
  • •Always retrieve policy text first and require citations in the output.
• •Ignoring document scope
  • •A single global index is not enough for banking.
  • •Partition by jurisdiction and product so answers stay compliant with local rules.
• •Skipping escalation paths
  • •If every question gets an answer string back, you will eventually ship bad guidance.
  • •Add explicit branches for exceptions and low-confidence cases using conditional routing.
• •Not versioning policies
  • •Policies change often in banks.
  • •Store document version IDs alongside answers so audits can reconstruct exactly what was used.

A good banking policy agent is not just “chat over docs.” It is a controlled workflow with retrieval discipline, deterministic routing where possible, human escalation where necessary, and logs that stand up in an audit review. LangGraph fits this problem because it makes those controls explicit instead of hiding them inside one opaque chain.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.