How to Fix 'embedding dimension mismatch during development' in LangGraph (Python)

When you see embedding dimension mismatch during development in a LangGraph app, it usually means the vector you generated from one embedding model does not match the vector size expected by your store, retriever, or existing index. This shows up most often after swapping models, changing providers, or reusing a persisted vector store from an earlier run.

In practice, the failure usually happens inside a retrieval node in your graph, not in the graph itself. LangGraph is just orchestrating the call chain; the real problem is that one component is producing 1536-dimension vectors while another expects 3072, 1024, or some other size.

The Most Common Cause

The #1 cause is changing embedding models without rebuilding the vector index.

A common pattern is this: you created a Chroma, FAISS, or PGVector store with one model during an earlier run, then later changed to a different embedding model in development. The persisted collection still contains old vectors, so the first similarity search throws an error like:

• •ValueError: Embedding dimension mismatch
• •InvalidDimensionException: Expected embedding dimension 1536, got 3072
• •chromadb.errors.InvalidDimensionException

Here’s the broken pattern versus the fixed pattern.

# broken.py
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

# Old collection was built with text-embedding-3-small (1536 dims)
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")  # 3072 dims now

vectorstore = Chroma(
    collection_name="docs",
    persist_directory="./chroma_db",
    embedding_function=embeddings,
)

docs = vectorstore.similarity_search("policy renewal terms", k=3)

# fixed.py
import shutil
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

# If you change embedding models, rebuild the index
shutil.rmtree("./chroma_db", ignore_errors=True)

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

vectorstore = Chroma.from_texts(
    texts=["policy renewal terms", "claims handling process"],
    embedding=embeddings,
    collection_name="docs",
    persist_directory="./chroma_db",
)

docs = vectorstore.similarity_search("policy renewal terms", k=3)

If you are using LangGraph, this usually sits inside a node:

from typing import TypedDict
from langgraph.graph import StateGraph, START, END

class State(TypedDict):
    query: str

def retrieve(state: State):
    return {"results": vectorstore.similarity_search(state["query"], k=3)}

graph = StateGraph(State)
graph.add_node("retrieve", retrieve)
graph.add_edge(START, "retrieve")
graph.add_edge("retrieve", END)
app = graph.compile()

The graph code is fine. The failure comes from stale vector data underneath it.

Other Possible Causes

1. Mixing embedding providers in different parts of the app

You might embed documents with OpenAI and queries with Cohere, Hugging Face, or Vertex AI. Even if both calls succeed, the dimensions may differ.

# broken
doc_embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
query_embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

# fixed
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# use the same embeddings object for indexing and querying

2. Reusing cached state across runs in LangGraph

If your graph stores retrieval artifacts in state or disk cache, you may be loading embeddings from an older execution path.

# broken state reuse
state["vectors"] = load_vectors_from_disk()  # built with old model

# fixed: version your cache by embedding model name + dim
cache_key = f"docs:{embeddings.model}"

3. Wrong dimension configured in a custom vector store or database column

With Postgres/pgvector or a custom ANN setup, the column dimension must match exactly.

-- broken if your model outputs 1536 but column expects 768
CREATE TABLE documents (
  id bigserial PRIMARY KEY,
  embedding vector(768)
);

-- fixed for text-embedding-3-small
CREATE TABLE documents (
  id bigserial PRIMARY KEY,
  embedding vector(1536)
);

4. Loading serialized embeddings from an older schema

If you saved raw vectors to JSON or pickle and later changed models, old records will still have the old shape.

# broken: blindly loads old vectors
vectors = json.load(open("embeddings.json"))

# fixed: validate shape before use
assert len(vectors[0]) == expected_dim

How to Debug It

1. •Print the embedding dimension at runtime
   • •Log one sample query vector and one document vector.
   • •If they differ, you found the issue immediately.

q_vec = embeddings.embed_query("test")
print(len(q_vec))

2. •

   Check what created your existing index

   • •Look at collection metadata, migration scripts, or seed jobs.
   • •Confirm which model originally populated Chroma, FAISS, or pgvector.
3. •

   Inspect LangGraph node boundaries

   • •Find where retrieval happens.
   • •The bug is often inside a node like retrieve_docs, search_knowledge_base, or rerank_context.
4. •

   Delete and rebuild local dev storage

   • •Remove persisted collections and rerun ingestion.
   • •If the error disappears after a clean rebuild, stale vectors were the cause.

Prevention

• •

  Version your embeddings explicitly

  • •Store model_name, provider, and expected dimension alongside every index.
  • •Treat embedding changes like schema migrations.
• •

  Never reuse a persisted store after changing models

  • •If you switch from text-embedding-3-small to text-embedding-3-large, rebuild everything downstream.
• •

  Add startup validation

  • •On app boot, embed one test string and verify its length matches your store configuration before compiling the LangGraph app.

If you want this class of bug to stop recurring in development, make embedding dimensions part of your application contract. In production systems, that contract should be enforced before any LangGraph node touches retrieval logic.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.