How to Fix 'vector search returning irrelevant results when scaling' in AutoGen (Python)

When vector search starts returning irrelevant results in AutoGen, it usually means your retrieval layer is no longer aligned with the data you indexed. The failure shows up after scaling: more documents, more chunks, multiple tenants, or a larger embedding corpus. In practice, the top-k results are still “valid” vectors, just not the right ones.

The common pattern is this: your prototype worked with a few files, then production added more sources and the retriever started surfacing semantically similar but operationally wrong chunks. In AutoGen, that often looks like RetrieveUserProxyAgent pulling poor context into the conversation, which then causes the assistant to answer confidently with the wrong grounding.

The Most Common Cause

The #1 cause is inconsistent chunking and embedding configuration between indexing and querying.

If you chunk documents one way during ingestion and query with a retriever configured differently, your vector space gets noisy fast. The same problem happens when you re-index with a new embedding model but keep old vectors around.

Broken vs fixed pattern

# BROKEN: mismatched ingestion/query setup
from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent
from autogen.retrieve_utils import TextSplitter

# Ingestion
splitter = TextSplitter(chunk_size=2000, chunk_overlap=0)
chunks = splitter.split_text(long_policy_doc)

# Later: query side uses different assumptions
retrieve_agent = RetrieveUserProxyAgent(
    name="retrieve",
    retrieve_config={
        "task": "qa",
        "vector_db": "chroma",
        "collection_name": "policy_docs",
        "chunk_token_size": 500,   # mismatch
        "embedding_model": "text-embedding-3-small",
    },
)

# This can produce irrelevant top-k hits as scale grows

# FIXED: keep ingestion and query aligned
from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent

COMMON_RETRIEVE_CONFIG = {
    "task": "qa",
    "vector_db": "chroma",
    "collection_name": "policy_docs_v2",
    "chunk_token_size": 1000,
    "embedding_model": "text-embedding-3-small",
}

retrieve_agent = RetrieveUserProxyAgent(
    name="retrieve",
    retrieve_config=COMMON_RETRIEVE_CONFIG,
)

# Rebuild the index with the same config used at query time.

If you changed embedding models, treat that as a schema migration. Mixing text-embedding-ada-002 vectors with newer embeddings in the same collection is a classic way to get garbage ranking.

Other Possible Causes

1) No metadata filtering in multi-tenant or multi-source indexes

If every tenant shares one collection without filters, nearest neighbors from other customers will win on semantic similarity alone.

# BAD: global search across all tenants
results = vector_db.search(query_text, top_k=5)

# GOOD: filter by tenant/source before similarity scoring
results = vector_db.search(
    query_text,
    top_k=5,
    where={"tenant_id": "acme", "source": "claims"}
)

In AutoGen setups using RetrieveUserProxyAgent, make sure your doc_paths, metadata tags, or backend filters isolate scope.

2) Chunk size too large or too small

Huge chunks dilute meaning. Tiny chunks lose context. Both get worse at scale because more near-duplicate fragments enter the index.

# BAD: giant chunks hurt precision
retrieve_config = {
    "chunk_token_size": 4000,
    "chunk_mode": "multi_lines"
}

# GOOD: balanced chunking for policy/KB text
retrieve_config = {
    "chunk_token_size": 800,
    "chunk_mode": "multi_lines"
}

For support docs and policies, start around 600–1000 tokens and tune from there.

3) Stale index after content updates

AutoGen will happily query an old collection if you don’t force a rebuild. That produces “irrelevant” results that are actually outdated.

# BAD: reusing old collection after document changes
retrieve_config = {
    "collection_name": "knowledge_base"
}

# GOOD: version your collections or rebuild explicitly
retrieve_config = {
    "collection_name": f"knowledge_base_v{index_version}"
}

If your backend supports it, delete and recreate on deploy. If not, use immutable collection names per release.

4) Similarity metric mismatch

Some stores default to cosine similarity; others use dot product or L2 distance. If your embeddings aren’t normalized and the backend metric changes under load or across environments, ranking shifts.

# Example config conceptually; exact keys depend on backend
vector_db_config = {
    "distance_metric": "cosine",   # keep consistent everywhere
}

Check your production store settings against local dev. A mismatch here can look like random retrieval drift.

How to Debug It

1. •

   Inspect raw top-k results before they hit the agent

   • •Query the vector DB directly.
   • •Print scores, chunk text, source metadata.
   • •If top-k is already wrong here, the issue is indexing or retrieval config, not AutoGen orchestration.
2. •

   Compare ingestion config to query config

   • •Chunk size
   • •Overlap
   • •Embedding model
   • •Collection name
   • •Distance metric
     If any of these differ between build-time and run-time, fix that first.
3. •

   Check whether stale vectors are still present

   • •Look for mixed document versions in one collection.
   • •Verify that deleted docs were actually removed.
   • •Rebuild into a fresh collection and rerun the same query.
4. •

   Test with metadata filters

   • •Narrow search to one tenant/source/category.
   • •If relevance improves immediately, your issue is scope leakage rather than embedding quality.

Prevention

• •Use immutable index versions:
  • •kb_v1, kb_v2, not one forever-growing collection.
• •Keep retrieval config in one shared module:
  • •Same chunking and embedding settings for ingestion and query.
• •Add retrieval smoke tests:
  • •For each known question, assert that top-3 contains an expected doc ID or source tag.

The real fix is not “tune AutoGen harder.” It’s making retrieval deterministic enough that scaling doesn’t change what “nearest” means. Once your index versioning, chunking, and metadata boundaries are stable, RetrieveUserProxyAgent stops surfacing random context and starts behaving like production software instead of a demo.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.