LlamaIndex Tutorial (Python): building conditional routing for beginners

This tutorial shows you how to build a simple conditional router in LlamaIndex with Python: one query path for general questions, another for document-specific questions, and a fallback when the input is unclear. You need this when a single agent should decide which tool or index to use instead of sending every request through the same retrieval pipeline.

What You'll Need

• •Python 3.10+
• •llama-index
• •llama-index-llms-openai
• •llama-index-embeddings-openai
• •An OpenAI API key in OPENAI_API_KEY
• •A small local text file or two for testing
• •Basic familiarity with VectorStoreIndex, QueryEngine, and RouterQueryEngine

Step-by-Step

1. •Start by installing the packages and setting your API key. If you already have a virtual environment, keep using it; otherwise create one before installing anything.

pip install llama-index llama-index-llms-openai llama-index-embeddings-openai
export OPENAI_API_KEY="your-openai-api-key"

2. •Create two small knowledge sources: one for product support and one for company policy. In real systems, these would usually be separate document sets, databases, or internal knowledge bases.

from pathlib import Path

Path("support.txt").write_text(
    "Reset your password from the account settings page.\n"
    "Billing issues are handled by the finance team.\n"
    "For login failures, clear your browser cache first."
)

Path("policy.txt").write_text(
    "Employees may work remotely two days per week.\n"
    "Vacation requests must be approved by a manager.\n"
    "Expense claims require receipts under 30 days old."
)

3. •Build two separate query engines from those documents. Each engine will answer only from its own content, which is the core idea behind routing.

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.settings import Settings

Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

support_docs = SimpleDirectoryReader(input_files=["support.txt"]).load_data()
policy_docs = SimpleDirectoryReader(input_files=["policy.txt"]).load_data()

support_index = VectorStoreIndex.from_documents(support_docs)
policy_index = VectorStoreIndex.from_documents(policy_docs)

support_engine = support_index.as_query_engine()
policy_engine = policy_index.as_query_engine()

4. •Define routing choices and build a selector that maps user intent to the right engine. Here we use LlamaIndex’s selector-based routing so the model chooses between support and policy based on the query.

from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.tools import QueryEngineTool, ToolMetadata

support_tool = QueryEngineTool(
    query_engine=support_engine,
    metadata=ToolMetadata(
        name="support_docs",
        description="Use for password resets, billing issues, login problems, and product support."
    ),
)

policy_tool = QueryEngineTool(
    query_engine=policy_engine,
    metadata=ToolMetadata(
        name="company_policy",
        description="Use for remote work, vacation approval, and expense reimbursement policy."
    ),
)

router = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=[support_tool, policy_tool],
)

5. •Run a few test queries to confirm the router sends each question to the correct source. You should see support questions answered from support.txt and policy questions answered from policy.txt.

queries = [
    "How do I reset my password?",
    "How many remote work days do employees get?",
]

for q in queries:
    response = router.query(q)
    print(f"\nQ: {q}")
    print(f"A: {response}")

6. •Add a fallback pattern if you want safer production behavior. A common setup is to reject low-confidence routes or send them to a general FAQ engine instead of guessing.

from llama_index.core.tools import ToolMetadata

general_docs = SimpleDirectoryReader(input_files=["support.txt", "policy.txt"]).load_data()
general_index = VectorStoreIndex.from_documents(general_docs)
general_engine = general_index.as_query_engine()

general_tool = QueryEngineTool(
    query_engine=general_engine,
    metadata=ToolMetadata(
        name="general_fallback",
        description="Use when the question is broad, ambiguous, or does not clearly fit support or policy."
    ),
)

fallback_router = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=[support_tool, policy_tool, general_tool],
)

Testing It

Run at least one question that clearly belongs to each route and one that is ambiguous. For example, ask about password resets, vacation approval, and then something broad like “What should I know about working here?” so you can see whether the fallback gets used.

If you want stronger verification, print the selected tool metadata alongside each answer during debugging. In production, log the original query, selected route name, latency, and whether the answer came from a fallback path.

Also check that each source file only answers its own domain. If your support engine starts answering policy questions confidently, your metadata descriptions are too vague or your documents are too mixed together.

Next Steps

• •Replace LLMSingleSelector with a more advanced multi-route setup when you need multiple specialized agents.
• •Add confidence thresholds and fallback logging before exposing routing to users.
• •Connect each route to real data sources like SharePoint, Confluence, SQL tables, or ticketing systems.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.