How to Fix 'context length exceeded' in LlamaIndex (TypeScript)

context length exceeded in LlamaIndex means you sent more tokens to the model than its context window can handle. In TypeScript, this usually shows up when you stuff too much raw text into a Document, build a query over too many chunks, or let retrieval return too many nodes.

The failure is usually not in the model itself. It’s in how you prepared input for VectorStoreIndex, QueryEngine, or a chat/agent workflow that keeps appending history until the prompt blows past the limit.

The Most Common Cause

The #1 cause is passing oversized text directly into indexing or querying without chunking or trimming it first.

You’ll often see errors like:

• •Error: context length exceeded
• •BadRequestError: This model's maximum context length is 8192 tokens
• •400 Bad Request: maximum context length exceeded

Broken vs fixed pattern

// ❌ Broken: huge text goes in as one giant document
import { Document, VectorStoreIndex } from "llamaindex";

const doc = new Document({
  text: veryLargePdfText, // thousands of tokens
});

const index = await VectorStoreIndex.fromDocuments([doc]);
const queryEngine = index.asQueryEngine();

const response = await queryEngine.query({
  query: "Summarize the policy exclusions",
});
console.log(response.toString());

// ✅ Fixed: chunk intentionally and control retrieval
import {
  Document,
  VectorStoreIndex,
  SentenceSplitter,
} from "llamaindex";

const splitter = new SentenceSplitter({
  chunkSize: 512,
  chunkOverlap: 50,
});

const doc = new Document({ text: veryLargePdfText });
const nodes = splitter.getNodesFromDocuments([doc]);

const index = await VectorStoreIndex.fromDocuments(
  nodes.map((node) => new Document({ text: node.text }))
);

const queryEngine = index.asQueryEngine({
  similarityTopK: 3,
});

const response = await queryEngine.query({
  query: "Summarize the policy exclusions",
});
console.log(response.toString());

If you’re already using a splitter and still failing, the issue is usually one of the causes below.

Other Possible Causes

1) Retrieval is returning too many chunks

If similarityTopK is too high, your prompt can balloon even when each chunk is reasonable.

// Too many nodes retrieved
const queryEngine = index.asQueryEngine({
  similarityTopK: 10,
});

Use fewer nodes:

const queryEngine = index.asQueryEngine({
  similarityTopK: 3,
});

2) Your chat memory is growing unbounded

This happens in agent loops or chat apps where every prior message gets resent.

// Risky: no message trimming
chatHistory.push({ role: "user", content: userMessage });
chatHistory.push({ role: "assistant", content: assistantReply });

Trim history before each call:

const trimmedHistory = chatHistory.slice(-6);

If you’re using a memory abstraction, make sure it has a token cap, not just a message cap.

3) Your prompt template is too verbose

A long system prompt plus retrieved context plus user input can exceed limits fast.

const prompt = `
You are an expert insurance analyst.
Explain everything in detail.
Include all clauses, exceptions, edge cases, and citations.
${retrievedContext}
${userQuestion}
`;

Shorten the instructions and move reusable rules out of the runtime prompt:

const prompt = `
Answer using only the provided context.
If the answer is missing, say so.
Context:
${retrievedContext}

Question:
${userQuestion}
`;

4) You are using a model with a smaller context window than you think

A lot of people test on one model and deploy on another. The error may appear only after switching providers or model IDs.

// Example: model with smaller window than expected
llm.model = "gpt-3.5-turbo";

Check the actual context limit for your configured model and set your chunking/retrieval accordingly. If your app assumes an 8k window but production uses a 4k model, it will fail under load.

How to Debug It

1. •

   Log the raw prompt size

   • •Print the final rendered prompt before sending it to the LLM.
   • •Count characters if needed, but token count is what matters.
   • •If you can’t inspect it directly, log retrieved node text lengths and chat history length.
2. •

   Reduce similarityTopK to 1

   • •If the error disappears, retrieval volume was part of the problem.
   • •Then increase slowly until it breaks again.
3. •

   Disable memory/history temporarily

   • •Run the same request with no prior conversation state.
   • •If it works cleanly, your chat buffer is overflowing.
4. •

   Test with a smaller input sample

   • •Replace your full PDF or transcript with a short excerpt.
   • •If that passes, your ingestion/chunking strategy needs adjustment.

A good rule here: isolate one variable at a time. Don’t change splitter size, retriever settings, prompt templates, and model IDs in one pass.

Prevention

• •Set explicit chunk sizes and overlaps for every ingestion pipeline. Don’t rely on defaults for legal docs, claims files, or call transcripts.
• •Cap retrieval aggressively. Start with similarityTopK: 3 and only raise it if evaluation proves you need more.
• •Put token budgeting into code review. Any agent loop that appends history should have a hard stop or summarizer.

If you’re building on LlamaIndex TypeScript for enterprise workloads, treat context as a budgeted resource. Once you control chunking, retrieval depth, and memory growth, this error stops being random and becomes predictable to fix.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.