Haystack Tutorial (Python): adding tool use for intermediate developers

This tutorial shows how to add tool use to a Haystack pipeline in Python so an LLM can call external functions instead of guessing. You need this when your agent must fetch live data, run deterministic business logic, or query systems that should not be embedded in the prompt.

What You'll Need

• •Python 3.10+
• •haystack-ai installed
• •An OpenAI API key set as OPENAI_API_KEY
• •A working internet connection for the model call
• •Basic familiarity with Haystack pipelines and components

Install the package:

pip install haystack-ai

Step-by-Step

1. •Start by defining a real tool function. Keep it small, deterministic, and side-effect free for the first version so you can verify the agent wiring before you connect internal systems.

from typing import Annotated

def get_exchange_rate(
    base_currency: Annotated[str, "ISO currency code like USD"],
    target_currency: Annotated[str, "ISO currency code like EUR"],
) -> str:
    rates = {
        ("USD", "EUR"): 0.92,
        ("EUR", "USD"): 1.09,
        ("USD", "GBP"): 0.79,
        ("GBP", "USD"): 1.27,
    }
    rate = rates.get((base_currency.upper(), target_currency.upper()))
    if rate is None:
        return f"No rate available for {base_currency}->{target_currency}"
    return f"1 {base_currency.upper()} = {rate} {target_currency.upper()}"

2. •Wrap that function as a Haystack tool and create a chat generator that supports tool calling. In practice, this is where your model gets permission to decide when to ask for external help.

import os
from haystack import Pipeline
from haystack.tools import Tool
from haystack.components.generators.chat import OpenAIChatGenerator

exchange_rate_tool = Tool(
    name="get_exchange_rate",
    description="Get a sample FX rate between two currencies.",
    parameters={
        "type": "object",
        "properties": {
            "base_currency": {"type": "string"},
            "target_currency": {"type": "string"},
        },
        "required": ["base_currency", "target_currency"],
    },
    function=get_exchange_rate,
)

llm = OpenAIChatGenerator(
    model="gpt-4o-mini",
    api_key=os.environ["OPENAI_API_KEY"],
)

3. •Build a pipeline that connects the model and the tool. The important part is passing tools into the generator and then feeding it a user message that clearly needs external data.

from haystack.dataclasses import ChatMessage

pipe = Pipeline()
pipe.add_component("llm", llm)

user_message = ChatMessage.from_user(
    "What is the exchange rate from USD to EUR? Use the tool if needed."
)

result = pipe.run(
    {
        "llm": {
            "messages": [user_message],
            "tools": [exchange_rate_tool],
        }
    }
)

print(result)

4. •Inspect the output and confirm whether the model used the tool or answered directly. For production work, you want to verify both the final answer and any intermediate tool calls because that is where failures usually hide.

messages = result["llm"]["replies"]

for message in messages:
    print("ROLE:", message.role)
    print("CONTENT:", message.content)
    if getattr(message, "tool_calls", None):
        print("TOOL CALLS:", message.tool_calls)

5. •If you want a cleaner agent loop, let Haystack manage repeated tool execution by keeping the conversation state in messages. This pattern matters when you add more than one tool, because the model may need multiple rounds before it can answer correctly.

conversation = [
    ChatMessage.from_user("Convert 100 USD to EUR using the exchange rate tool.")
]

result = pipe.run(
    {
        "llm": {
            "messages": conversation,
            "tools": [exchange_rate_tool],
        }
    }
)

reply = result["llm"]["replies"][0]
print(reply.content)

Testing It

Run the script with OPENAI_API_KEY exported in your shell and check that it returns a response grounded in the tool output, not a hallucinated rate. If you see no tool call at all, your prompt may be too vague or your model may be ignoring tools because the setup is incomplete.

A good test is to ask for a currency pair that exists in your lookup table and one that does not. The first should produce a valid conversion-style response, while the second should return an explicit “No rate available” message from the tool.

If you are integrating this into an insurance or banking workflow, add logging around tool_calls and final assistant messages before shipping it. That gives you traceability when an agent makes a bad routing decision or uses stale business logic.

Next Steps

• •Add a second tool, such as lookup_policy_status or get_account_balance, and test multi-tool routing.
• •Replace the hardcoded lookup with an internal API client wrapped behind the same Tool interface.
• •Add guardrails: input validation, timeout handling, and structured logging for every tool invocation.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.