How to Build a policy Q&A Agent Using AutoGen in TypeScript for healthcare

A policy Q&A agent for healthcare answers staff questions about coverage, prior authorization, eligibility, benefits, and internal policy rules. The point is not just speed; it is reducing bad decisions caused by stale PDFs, inconsistent interpretations, and missing audit trails.

Architecture

• •User interface layer
  • •A chat app, internal portal, or support tool where clinicians, care coordinators, or ops staff ask policy questions.
• •Policy retrieval layer
  • •A document index over plan documents, SOPs, CMS guidance, and internal policy memos.
  • •This should return only approved sources with version metadata.
• •AutoGen agent layer
  • •A ConversableAgent that handles the conversation and a second agent that acts as a policy checker or reviewer.
  • •In healthcare, you want at least one verification step before returning an answer.
• •Tooling layer
  • •Functions for search, citation lookup, and policy version checks.
  • •Keep tools narrow: retrieve facts, don’t let the model invent them.
• •Audit and logging layer
  • •Store prompts, retrieved passages, outputs, source IDs, timestamps, and user identity.
  • •This is what lets compliance teams review why the agent answered a certain way.
• •Safety and governance layer
  • •PHI redaction, access control, residency-aware storage, and refusal behavior when evidence is weak.

Implementation

1) Install AutoGen and define your policy tools

Use the TypeScript package from AutoGen’s agentchat stack. The pattern below assumes you are wiring the agent into a backend service that already has authenticated users.

npm install @autogen/agentchat @autogen/core zod

Create a strict retrieval tool that only returns approved policy snippets. Do not pass raw document blobs to the model.

import { z } from "zod";

type PolicyChunk = {
  id: string;
  title: string;
  version: string;
  text: string;
};

const querySchema = z.object({
  question: z.string().min(5),
});

const policyIndex: PolicyChunk[] = [
  {
    id: "pol-102",
    title: "Prior Authorization Rules",
    version: "2025.01",
    text: "MRI lumbar spine requires prior authorization unless emergency criteria are met.",
  },
];

export async function searchPolicy(question: string): Promise<PolicyChunk[]> {
  const q = question.toLowerCase();
  return policyIndex.filter(
    (chunk) =>
      chunk.title.toLowerCase().includes(q) ||
      chunk.text.toLowerCase().includes(q)
  );
}

2) Build a ConversableAgent with a retrieval tool

AutoGen’s ConversableAgent is the core conversation object. Give it a system message that forces grounded answers and citations.

import { ConversableAgent } from "@autogen/agentchat";
import { FunctionTool } from "@autogen/core";

const retrievePolicyTool = new FunctionTool({
  name: "search_policy",
  description: "Search approved healthcare policy documents by question.",
  parameters: querySchema,
  execute: async ({ question }) => {
    const results = await searchPolicy(question);
    return results.map((r) => ({
      id: r.id,
      title: r.title,
      version: r.version,
      text: r.text,
    }));
  },
});

export const policyAgent = new ConversableAgent({
  name: "policy_qna_agent",
  systemMessage:
    [
      "You answer healthcare policy questions using only retrieved sources.",
      "If evidence is insufficient, say you cannot confirm the policy.",
      "Always cite source id and version.",
      "Never request or expose PHI unless absolutely necessary for the workflow.",
    ].join(" "),
});

3) Add a reviewer agent for compliance-style verification

For healthcare workflows, one model generating an answer is not enough. Use a second agent to check whether the response stays within the retrieved evidence.

import { AssistantAgent } from "@autogen/agentchat";

export const reviewerAgent = new AssistantAgent({
  name: "policy_reviewer",
  systemMessage:
    [
      "Review answers for grounding in cited sources.",
      "Reject answers that introduce unsupported claims.",
      "Flag any missing citations or ambiguous policy interpretation.",
    ].join(" "),
});

4) Run the conversation and enforce grounded output

The actual flow is:

1. •User asks a question.
2. •Agent retrieves policy snippets.
3. •Agent drafts an answer with citations.
4. •Reviewer checks it before release.

import { GroupChat } from "@autogen/agentchat";

async function answerPolicyQuestion(question: string) {
  const retrieval = await retrievePolicyTool.execute({ question });

  const context = retrieval.length
    ? JSON.stringify(retrieval)
    : "No matching approved policies found.";

  const userMessage = `
Question: ${question}

Approved sources:
${context}

Return a concise answer with citations in this format:
- Answer
- Sources
`;

  const chat = new GroupChat({
    agents: [policyAgent, reviewerAgent],
    messages: [],
    maxRounds: 4,
  });

  const result = await chat.run(userMessage);
  return result.messages.at(-1)?.content ?? "";
}

That pattern keeps the model inside a narrow lane. In practice, you should also parse the final response and reject it if it lacks source IDs or if it mentions unsupported exceptions.

Production Considerations

• •Data residency
  • •Keep retrieval indexes and logs in-region if your healthcare contracts require it.
  • •If you use an external LLM endpoint, verify where prompts and embeddings are processed and stored.
• •Compliance logging
  • •Persist every question, retrieved source ID, final answer, model version, and user identity.
  • •Audit logs need to be immutable enough for compliance review and incident investigation.
• •PHI guardrails
  • •Redact identifiers before sending text to the model unless the workflow explicitly requires them.
  • •Add detectors for MRNs, member IDs, DOBs, addresses, and clinical notes.
• •Fallback behavior
  • •If retrieval returns no strong match or conflicting policies exist, force escalation to a human reviewer.
  • •In healthcare, “best guess” is usually worse than “cannot confirm.”

Common Pitfalls

• •

  Letting the model answer without citations

  This turns your agent into a hallucination machine with good formatting. Require source IDs from approved documents before any response goes out.

• •

  Mixing current policies with stale documents

  Healthcare policies change often across plan year updates and regulatory changes. Store version metadata on every chunk and filter retrieval by effective date.

• •

  Sending raw PHI into prompts by default

  That creates avoidable compliance risk. Redact first, minimize prompt content, and only reveal patient-specific details when the workflow truly needs them.

• •

  Skipping human review on ambiguous cases

  Prior auth exceptions, medical necessity edge cases, and coverage disputes need escalation paths. Build confidence thresholds so low-certainty answers go to staff instead of users.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.