How to Fix 'vector search returning irrelevant results' in LlamaIndex (Python)

If your LlamaIndex vector search is returning irrelevant results, the retriever is usually working exactly as configured — just not configured the way you think. This typically shows up when chunking, embedding, indexing, or retrieval settings don’t match the data shape you’re querying.

The symptom is usually one of these:

• •VectorStoreIndex returns semantically “nearby” but wrong chunks
• •similarity_top_k=1 gives junk results
• •retrieved nodes contain the right topic but the wrong section
• •ResponseSynthesizer answers from irrelevant context even though the query looks correct

The Most Common Cause

The #1 cause is bad chunking combined with weak metadata boundaries.

In LlamaIndex, if you dump large documents into VectorStoreIndex without controlling chunk size and overlap, embeddings get noisy. The model then matches on broad semantic similarity instead of the exact passage you want.

Broken vs fixed pattern

# BROKEN
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

docs = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)

query_engine = index.as_query_engine()
response = query_engine.query("What is the refund policy?")
print(response)

# FIXED
from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import SimpleDirectoryReader

Settings.chunk_size = 512
Settings.chunk_overlap = 64

docs = SimpleDirectoryReader("./data").load_data()

# Better: preserve structure and chunk intentionally
parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = parser.get_nodes_from_documents(docs)

index = VectorStoreIndex(nodes)

query_engine = index.as_query_engine(similarity_top_k=5)
response = query_engine.query("What is the refund policy?")
print(response)

If your source docs are long PDFs or policy manuals, this matters even more. A single node that spans multiple sections will embed as a blended vector, which is exactly how you get “close but wrong” matches.

Other Possible Causes

1) Wrong embedding model for your domain

If you indexed with one embedding model and queried with another, retrieval quality drops fast. This also happens when you use a general-purpose model for highly technical or policy-heavy text.

from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

Make sure the same embedding config is used at index time and query time. If you changed models after building the index, rebuild it.

2) You are querying an old persisted index

A stale index can look like “irrelevant search,” but the real issue is that your vector store still contains old chunks.

# If you changed docs or chunking strategy, rebuild before querying.
index.storage_context.persist(persist_dir="./storage")

# Later:
from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

If the underlying documents changed and you didn’t re-ingest them, your results will be out of date.

3) Retriever settings are too narrow or too broad

A low similarity_top_k can miss the right node. A very high value can flood synthesis with junk context.

retriever = index.as_retriever(similarity_top_k=10)
nodes = retriever.retrieve("What is the refund policy?")
for n in nodes:
    print(n.score, n.node.text[:200])

If top results look weak, increase top_k, then inspect scores manually. Don’t guess — print them.

4) Metadata filtering is excluding the right nodes

Sometimes retrieval looks irrelevant because filters silently remove the best matches.

retriever = index.as_retriever(
    similarity_top_k=5,
    filters={"department": "claims"}  # example filter logic depends on your setup
)

If your metadata values are inconsistent — for example "Claims" vs "claims" — retrieval can degrade or fail entirely. Normalize metadata at ingestion time.

How to Debug It

1. •

   Inspect raw retrieved nodes Print node text and scores before blaming LLM synthesis.

   retriever = index.as_retriever(similarity_top_k=5)
   results = retriever.retrieve("What is the refund policy?")
   
   for r in results:
       print("SCORE:", r.score)
       print(r.node.get_content()[:400])
       print("-" * 80)
   

2. •

   Check whether chunk boundaries make sense If a chunk contains multiple topics, split it smaller. If chunks are too tiny, increase overlap so context isn’t lost.

3. •

   Verify embedding consistency Confirm the same Settings.embed_model was used when building and querying. If not, rebuild the index from scratch.

4. •

   Bypass synthesis Query retrieval only. If retrieval is bad, the problem is upstream of the LLM answer layer.

   retriever = index.as_retriever(similarity_top_k=5)
   nodes = retriever.retrieve("What is the refund policy?")
   

Prevention

• •Use explicit chunking rules for every new corpus. Default chunking is rarely good enough for policies, contracts, or support docs.
• •Persist and version your indexes alongside document versions and embedding config.
• •Add a retrieval smoke test that asserts known queries return expected source sections before shipping.

If you want reliable vector search in LlamaIndex, treat ingestion as part of application logic, not preprocessing noise. Most “irrelevant results” bugs are really data shaping bugs.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.