How to Integrate Haystack for banking with Elasticsearch for production AI

Connecting Haystack for banking with Elasticsearch gives you a practical RAG stack for regulated environments: structured retrieval over large document sets, fast filtering, and audit-friendly search. In banking, that usually means policy Q&A, customer support copilots, KYC/AML document lookup, and internal knowledge agents that need to answer from controlled sources instead of free-form generation.

Elasticsearch handles indexing, filtering, and retrieval at scale. Haystack for banking sits on top as the orchestration layer for pipelines, prompting, and answer generation. Put them together and you get an agent system that can search approved bank content quickly and ground responses in retrieved evidence.

Prerequisites

• •Python 3.10+
• •An Elasticsearch cluster:
  • •Local Docker instance or managed Elastic Cloud
  • •HTTP endpoint and credentials
• •Haystack installed:
  • •haystack-ai
  • •Banking-specific Haystack components if your distribution includes them
• •An embedding model access path:
  • •OpenAI, Azure OpenAI, Hugging Face, or local embeddings
• •A document corpus ready for ingestion:
  • •PDFs, policy docs, SOPs, product docs, FAQ exports
• •Environment variables set:
  • •ELASTICSEARCH_URL
  • •ELASTICSEARCH_API_KEY or username/password
  • •Model API keys if needed

Integration Steps

1. •Install dependencies and verify Elasticsearch connectivity.

pip install haystack-ai elasticsearch sentence-transformers

from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://localhost:9200",
    api_key="YOUR_ELASTICSEARCH_API_KEY",
)

print(es.info())

If es.info() returns cluster metadata, your connection is good. In production, use TLS and API keys rather than basic auth.

2. •Create an Elasticsearch index for banking documents.

Use a mapping that supports text search plus vector retrieval. For bank-grade retrieval, keep metadata fields explicit so you can filter by product line, jurisdiction, or document type.

from elasticsearch import Elasticsearch

INDEX_NAME = "banking_docs"

mapping = {
    "mappings": {
        "properties": {
            "content": {"type": "text"},
            "title": {"type": "text"},
            "doc_type": {"type": "keyword"},
            "jurisdiction": {"type": "keyword"},
            "embedding": {"type": "dense_vector", "dims": 384}
        }
    }
}

es = Elasticsearch("https://localhost:9200", api_key="YOUR_ELASTICSEARCH_API_KEY")

if not es.indices.exists(index=INDEX_NAME):
    es.indices.create(index=INDEX_NAME, **mapping)

This keeps retrieval flexible. You can do keyword search for exact policy terms and vector search for semantic matching.

3. •Write documents into Elasticsearch using Haystack documents.

Haystack uses Document objects as the standard data structure. You can prepare chunks from policies or procedures and push them into Elasticsearch with metadata attached.

from haystack import Document
from sentence_transformers import SentenceTransformer
from elasticsearch import Elasticsearch

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
es = Elasticsearch("https://localhost:9200", api_key="YOUR_ELASTICSEARCH_API_KEY")

docs = [
    Document(
        content="Mortgage applications require two forms of ID and proof of income.",
        meta={"title": "Mortgage Policy", "doc_type": "policy", "jurisdiction": "US"}
    ),
    Document(
        content="Suspicious activity reports must be escalated within one business day.",
        meta={"title": "AML SOP", "doc_type": "sop", "jurisdiction": "US"}
    ),
]

for doc in docs:
    emb = model.encode(doc.content).tolist()
    es.index(
        index="banking_docs",
        document={
            "content": doc.content,
            "title": doc.meta["title"],
            "doc_type": doc.meta["doc_type"],
            "jurisdiction": doc.meta["jurisdiction"],
            "embedding": emb,
        },
    )

In a real pipeline, chunk long PDFs before embedding them. Keep chunks small enough to retrieve precisely but large enough to preserve context.

4. •Build a Haystack retrieval pipeline backed by Elasticsearch.

Haystack’s pipeline API lets you wire retrieval into downstream generation. If you’re using the banking-oriented Haystack package variant in your environment, the same pattern applies: a retriever component queries Elasticsearch and returns grounded passages to the LLM.

from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators import OpenAIGenerator

# Example retriever component name may vary by Haystack version/package.
# Use the Elasticsearch-backed retriever available in your setup.
from haystack_integrations.components.retrievers.elasticsearch import ElasticsearchEmbeddingRetriever

retriever = ElasticsearchEmbeddingRetriever(
    document_store=None,
    query_embedding_model=model,
)

prompt_builder = PromptBuilder(
    template="""
You are a banking assistant.
Answer only from the provided documents.

Question: {{question}}

Documents:
{% for doc in documents %}
- {{ doc.content }} ({{ doc.meta.title }})
{% endfor %}

Answer:
"""
)

generator = OpenAIGenerator(model="gpt-4o-mini")

pipeline = Pipeline()
pipeline.add_component("retriever", retriever)
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("llm", generator)

pipeline.connect("retriever.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder.prompt", "llm.prompt")

The exact retriever class name depends on your Haystack integration package version. The pattern is stable: retrieve from Elasticsearch, format context with a prompt builder, then generate an answer.

5. •Run a query with metadata filtering for production controls.

Banking systems usually need jurisdictional filtering or product scoping. Use filters so an agent never answers from the wrong policy set.

query = {
    "question": "What documents are required for a mortgage application?"
}

result = pipeline.run({
    "retriever": {
        # Query text or vector input depending on your retriever setup
        "query": query["question"],
        # Example filter structure; adapt to your retriever implementation
        "filters": {"doc_type": ["policy"], "jurisdiction": ["US"]}
    },
    "prompt_builder": {
        **query
    }
})

print(result["llm"]["replies"][0])

For production AI, add guardrails around filters at the service layer. Don’t let user input directly control unrestricted retrieval across all indices.

Testing the Integration

Run a simple end-to-end check against one known policy document.

test_query = "What is required for suspicious activity escalation?"
response = pipeline.run({
    "retriever": {
        "query": test_query,
        "filters": {"doc_type": ["sop"], "jurisdiction": ["US"]}
    },
    "prompt_builder": {
        "question": test_query
    }
})

answer = response["llm"]["replies"][0]
print(answer)

Expected output should reference the AML SOP and mention escalation within one business day:

Suspicious activity reports must be escalated within one business day.

If you get an empty answer or irrelevant context:

• •Check that embeddings were generated with the same model dimension used in the index mapping
• •Confirm documents were indexed successfully in Elasticsearch
• •Verify your filters match actual metadata values
• •Inspect retrieved documents before generation

Real-World Use Cases

• •Internal banking policy assistant:
  • •Answer questions about lending rules, account opening requirements, AML procedures, or retention policies from approved source docs.
• •Compliance copilot:
  • •Retrieve jurisdiction-specific controls and generate evidence-backed summaries for auditors or operations teams.
• •Customer support agent:
  • •Search product guides and service terms fast enough to support live chat while keeping responses grounded in bank-approved content.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.