Haystack Tutorial (Python): adding tool use for advanced developers

This tutorial shows how to add tool use to a Haystack pipeline so an LLM can call external functions, inspect results, and continue reasoning with real data. You need this when your agent must do more than answer from context: think ticket lookup, policy checks, database reads, calculator calls, or any workflow where the model should decide when to use a tool.

What You'll Need

• •Python 3.10+
• •haystack-ai
• •openai API key
• •Access to an OpenAI model that supports tool calling
• •Basic familiarity with Haystack pipelines and components
• •A terminal and a virtual environment

Install the package:

pip install haystack-ai

Set your API key:

export OPENAI_API_KEY="your-key-here"

Step-by-Step

1. •Start by defining a real tool as a normal Python function. Keep it deterministic and narrow in scope; tools should do one thing well and return plain text or structured data.

from typing import Literal

def lookup_policy_status(policy_id: str) -> str:
    fake_db = {
        "POL-1001": "Active",
        "POL-1002": "Lapsed",
        "POL-1003": "Pending renewal",
    }
    status = fake_db.get(policy_id, "Unknown policy ID")
    return f"Policy {policy_id} status: {status}"

print(lookup_policy_status("POL-1002"))

2. •Wrap the function in a Haystack Tool so the LLM can discover and call it. The description matters because it is what the model uses to decide whether this tool is relevant.

from haystack.tools import Tool

policy_tool = Tool(
    name="lookup_policy_status",
    description="Look up the current status of an insurance policy by policy ID.",
    parameters={
        "type": "object",
        "properties": {
            "policy_id": {
                "type": "string",
                "description": "The insurance policy ID, for example POL-1001.",
            }
        },
        "required": ["policy_id"],
    },
    function=lookup_policy_status,
)

print(policy_tool.name)

3. •Create a generator component that supports tool calling. In Haystack, OpenAIChatGenerator can emit tool calls when the prompt requires them and tools are attached.

from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage

generator = OpenAIChatGenerator(
    model="gpt-4o-mini",
    tools=[policy_tool],
)

messages = [
    ChatMessage.from_system("You are a support assistant. Use tools when needed."),
    ChatMessage.from_user("What is the status of policy POL-1003?"),
]

result = generator.run(messages=messages)
print(result)

4. •Add a small loop that executes tool calls and feeds results back to the model. This is the part most teams skip at first; without it, you only get the tool request, not the final answer.

from haystack.dataclasses import ChatMessage

messages = [
    ChatMessage.from_system("You are a support assistant. Use tools when needed."),
    ChatMessage.from_user("What is the status of policy POL-1003?"),
]

result = generator.run(messages=messages)
assistant_msg = result["replies"][0]

if assistant_msg.tool_calls:
    for call in assistant_msg.tool_calls:
        tool_result = policy_tool.function(**call.arguments)
        messages.append(assistant_msg)
        messages.append(ChatMessage.from_tool(tool_result, origin=call.id))

    final_result = generator.run(messages=messages)
    print(final_result["replies"][0].content)
else:
    print(assistant_msg.content)

5. •Put it into a pipeline when you want this behavior inside a larger system. This keeps your orchestration explicit and makes it easier to add retrieval, routing, or guardrails later.

from haystack import Pipeline

pipe = Pipeline()
pipe.add_component("llm", generator)

output = pipe.run({
    "llm": {
        "messages": [
            ChatMessage.from_system("You are a support assistant. Use tools when needed."),
            ChatMessage.from_user("Check policy POL-1001."),
        ]
    }
})

print(output["llm"]["replies"][0].content)

Testing It

Run the script and ask for a policy ID that exists in your fake database, then one that does not. You should see either a direct answer or an intermediate tool call followed by a final natural-language response.

If you want to confirm tool execution, log assistant_msg.tool_calls before invoking the function. For production work, test three cases: valid input, invalid input, and ambiguous input where the model should ask for clarification instead of calling the tool.

Also check that your tool returns stable output formats. If you plan to chain multiple tools later, keep return values structured and predictable so downstream prompts do not have to guess at parsing rules.

Next Steps

• •Add multiple tools and let the model choose between them based on descriptions.
• •Wrap tool outputs in JSON and validate them before sending them back to the LLM.
• •Combine tool use with retrieval so the agent can read documents before deciding whether to call an external system.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.