How to Fix 'token limit exceeded when scaling' in LlamaIndex (TypeScript)

What the error means

token limit exceeded when scaling usually means LlamaIndex tried to pack too much text into a single LLM call or embedding call after your data grew. In TypeScript projects, this often shows up when you scale from a few documents to hundreds and keep the same chunking, retriever, or synthesis settings.

You’ll typically hit it during indexing, query synthesis, or recursive retrieval when the framework tries to merge nodes and the prompt blows past the model context window. The fix is usually not “buy a bigger model”; it’s to control chunk sizes, retrieval depth, and response synthesis behavior.

The Most Common Cause

The #1 cause is oversized chunks combined with aggressive retrieval/synthesis. People ingest large documents as huge nodes, then ask VectorStoreIndex or QueryEngine to stuff too many of them into one prompt.

Here’s the broken pattern I see most often:

import { Document, VectorStoreIndex } from "llamaindex";

const docs = [
  new Document({ text: veryLargePolicyText }),
];

const index = await VectorStoreIndex.fromDocuments(docs);

const queryEngine = index.asQueryEngine({
  similarityTopK: 10,
});

const response = await queryEngine.query({
  query: "Summarize the exclusions in this policy.",
});

console.log(response.toString());

And here’s the fixed version side by side:

import {
  Document,
  VectorStoreIndex,
  SentenceSplitter,
} from "llamaindex";

const splitter = new SentenceSplitter({
  chunkSize: 800,
  chunkOverlap: 120,
});

const docs = [
  new Document({ text: veryLargePolicyText }),
];

const splitDocs = await Promise.all(
  docs.map(async (doc) => splitter.splitText(doc.text))
);

const flatDocs = splitDocs.flat().map((text) => new Document({ text }));

const index = await VectorStoreIndex.fromDocuments(flatDocs);

const queryEngine = index.asQueryEngine({
  similarityTopK: 3,
});

const response = await queryEngine.query({
  query: "Summarize the exclusions in this policy.",
});

console.log(response.toString());

If you’re using CompactAndRefine, Refine, or another synthesizer class, keep an eye on how many source nodes are being passed into each step. That’s where the token growth usually happens.

Other Possible Causes

1. Recursive retrievers pulling too many nodes

If you use recursive retrieval, one high-level hit can expand into a large set of child nodes.

// Risky: expansion can explode token usage
const queryEngine = index.asQueryEngine({
  similarityTopK: 8,
});

Fix it by tightening retrieval depth and top-k:

const queryEngine = index.asQueryEngine({
  similarityTopK: 2,
});

If you’re using classes like RecursiveRetriever or RouterRetriever, inspect how many nodes each branch returns.

2. Prompt templates are carrying too much context

A custom prompt that includes raw document text, metadata blobs, or prior chat history can push you over the limit even if retrieval is fine.

// Bad: dumping full metadata into every prompt
const context = JSON.stringify(metadata);

Trim it down:

const context = {
  source: metadata.source,
  page: metadata.page,
};

If you use custom prompt helpers like PromptTemplate, keep templates short and move bulky context into retrieved nodes instead of system prompts.

3. Embedding batch size is too large

This error can also show up during ingestion if your embedding pipeline batches too many chunks at once.

// Risky for large corpora
await embedNodes(nodes);

Reduce batch size in your ingestion pipeline:

for (let i = 0; i < nodes.length; i += 16) {
  const batch = nodes.slice(i, i + 16);
  await embedNodes(batch);
}

If you use an ingestion pipeline with IngestionPipeline, make sure batching is controlled before vectors are generated.

4. Chat memory is growing without trimming

If your app keeps appending every user turn into memory, the final prompt eventually exceeds context.

// Bad: unbounded history
messages.push(...newTurns);

Trim history aggressively:

const trimmedMessages = messages.slice(-8);

This matters when using chat engines built on top of ContextChatEngine or any custom conversation loop.

How to Debug It

1. •

   Log chunk sizes before indexing

   • •Print character counts and approximate token counts for your documents.
   • •If you see multi-thousand-word chunks, that’s your first problem.
2. •

   Lower retrieval settings temporarily

   • •Set similarityTopK to 1 or 2.
   • •If the error disappears, your issue is prompt assembly, not ingestion.
3. •

   Disable custom prompts and memory

   • •Remove custom system prompts, chat history, and extra metadata.
   • •Re-run with the default query engine path.
   • •If it works now, one of those layers was bloating context.
4. •

   Inspect which stage fails

   • •Ingest-time failure points to splitting/embedding/batching.
   • •Query-time failure points to retriever expansion or synthesis.
   • •If the stack mentions classes like ResponseSynthesizer, CompactAndRefine, or RetrieverQueryEngine, focus there first.

Prevention

• •Use a real splitter up front:
  • •Start with chunk sizes around 500–1000 tokens for policy/legal/claims content.
• •Keep retrieval narrow:
  • •Prefer topK=2–4 unless you have a strong reason to go wider.
• •Trim everything that accumulates:
  • •Chat history, metadata blobs, and prompt context should all be bounded.

The rule is simple: if your data grows faster than your token budget, LlamaIndex will eventually fail loudly. Control chunking first, then retrieval depth, then synthesis width.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.