How to Build a policy Q&A Agent Using LangChain in TypeScript for payments

A policy Q&A agent for payments answers internal questions like “Can we store this cardholder data?” or “Is this payout route allowed for this country?” by retrieving approved policy text and generating a grounded answer. It matters because payments teams need fast decisions without letting engineers, ops, or support staff improvise around compliance rules, audit requirements, and regional restrictions.

Architecture

• •

  Policy source loader

  • •Pulls policy docs from approved sources: Confluence exports, PDFs, Markdown in Git, or a controlled S3 bucket.
  • •Keep the source set small and versioned.
• •

  Text splitter

  • •Breaks policies into retrieval-sized chunks.
  • •Use chunk boundaries that preserve legal meaning, not arbitrary token cuts.
• •

  Vector store retriever

  • •Indexes policy chunks in a vector database.
  • •Returns the most relevant passages for each question.
• •

  LLM answer chain

  • •Uses retrieved policy context to answer with citations.
  • •Must refuse when the policy corpus does not support a confident answer.
• •

  Guardrails layer

  • •Blocks requests involving secrets, PCI data, or unsupported advice.
  • •Forces escalation when the query is ambiguous or high-risk.
• •

  Audit logging

  • •Stores question, retrieved sources, answer, model version, and timestamp.
  • •Required for compliance review and incident investigation.

Implementation

1) Load policy documents and build an index

For payments, keep policy documents in a controlled repository and re-index on every approved change. The example below loads Markdown files from disk, splits them into chunks, and stores embeddings in memory for simplicity.

import { DirectoryLoader } from "langchain/document_loaders/fs/directory";
import { TextLoader } from "langchain/document_loaders/fs/text";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { OpenAIEmbeddings } from "@langchain/openai";

async function buildVectorStore() {
  const loader = new DirectoryLoader("./policy-docs", {
    ".md": (path) => new TextLoader(path),
    ".txt": (path) => new TextLoader(path),
  });

  const docs = await loader.load();

  const splitter = new RecursiveCharacterTextSplitter({
    chunkSize: 900,
    chunkOverlap: 150,
  });

  const splitDocs = await splitter.splitDocuments(docs);

  const embeddings = new OpenAIEmbeddings({
    apiKey: process.env.OPENAI_API_KEY,
    model: "text-embedding-3-small",
  });

  return await MemoryVectorStore.fromDocuments(splitDocs, embeddings);
}

2) Create a retriever with strict scope

Do not let the agent search arbitrary company content. For payments policy Q&A, constrain retrieval to approved policy files and use metadata filters if you separate regions like EU, UK, or US.

const vectorStore = await buildVectorStore();

const retriever = vectorStore.asRetriever(4);

If you need region-aware behavior later, move to a store that supports metadata filtering and tag each document with jurisdiction, policy_type, and version.

3) Build the question-answer chain

Use ChatOpenAI plus createRetrievalChain so answers are grounded in retrieved policy text. The prompt should tell the model to cite only supplied context and to escalate if the context is insufficient.

import { ChatOpenAI } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { createStuffDocumentsChain } from "langchain/chains/combine_documents";
import { createRetrievalChain } from "langchain/chains/retrieval";

const llm = new ChatOpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  model: "gpt-4o-mini",
  temperature: 0,
});

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are a payments policy assistant.
Answer only using the provided context.
If the context does not contain enough information, say you cannot confirm and recommend escalation.
Cite the exact policy section names when possible.`,
  ],
  ["human", "Question: {input}\n\nContext:\n{context}"],
]);

const combineDocsChain = await createStuffDocumentsChain({
  llm,
  prompt,
});

const qaChain = await createRetrievalChain({
  retriever,
  combineDocsChain,
});

export async function answerPolicyQuestion(question: string) {
  const result = await qaChain.invoke({ input: question });
  return result.answer;
}

4) Add a payment-specific guardrail before answering

A policy agent should refuse questions that try to extract secrets or operationalize prohibited behavior. In payments environments, this includes PANs, CVVs, auth credentials, webhook secrets, settlement account details, and anything that looks like PCI scope leakage.

function isHighRiskPaymentQuery(question: string): boolean {
  const patterns = [
    /\b\d{13,19}\b/, // possible PAN
    /\bcvv\b/i,
    /\bsecret\b/i,
    /\bapi key\b/i,
    /\bwebhook\b.*\bsecret\b/i,
    /\bcardholder data\b/i,
    /\bstore.*card\b/i,
  ];

  return patterns.some((re) => re.test(question));
}

export async function safeAnswerPolicyQuestion(question: string) {
  if (isHighRiskPaymentQuery(question)) {
    return {
      answer:
        "I can't help with requests involving payment secrets or cardholder data. Escalate to security/compliance.",
      source: "guardrail",
    };
  }

  return {
    answer: await answerPolicyQuestion(question),
    source: "policy-retrieval",
   };
}

Production Considerations

• •

  Deploy with versioned policies

  • •Tie every response to a specific policy version and document hash.
  • •When compliance updates land, re-index before exposing the new version.
• •

  Log for auditability

  • •Store query text, retrieved chunks, final answer, model name, latency, and user identity.
  • •Payments teams need traceability when a decision is challenged later.
• •

  Keep data residency explicit

  • •If policies include region-specific rules or regulated data references, keep embeddings and logs in the correct region.
  • •Do not send restricted content to an LLM endpoint outside approved jurisdictions.
• •

  Add escalation paths

  • •If retrieval confidence is low or policies conflict across regions, return “needs review.”
  • •Route those cases to compliance or operations instead of guessing.

Common Pitfalls

• •

  Using general web knowledge instead of approved policies

  This is how agents hallucinate around PCI DSS, chargeback handling, or sanctions rules. Always ground answers in your own controlled document set.

• •

  Chunking policies too aggressively

  Splitting legal text into tiny fragments breaks meaning. Use chunk sizes that preserve definitions, exceptions, and cross-references so retrieval returns usable context.

• •

  Skipping refusal behavior

  A payments agent must say “I can’t confirm” more often than a generic chatbot. If the retrieved text does not explicitly support the answer, escalate rather than infer.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.