How to Fix 'JSON parsing error' in LlamaIndex (TypeScript)

A JSON parsing error in LlamaIndex TypeScript usually means the framework expected structured JSON from an LLM or tool call, but got plain text, malformed JSON, or extra tokens instead. You’ll see this most often when using structured outputs, function calling, query engines, or any parser that expects a strict schema.

In practice, this is rarely a “JSON” problem in your app code. It’s usually an output-format mismatch between the prompt, the model, and the parser LlamaIndex is using.

The Most Common Cause

The #1 cause is asking the model for JSON but not forcing a strict structured output path. In LlamaIndex TS, this shows up when StructuredOutputParser, PydanticProgram, or a response synthesizer expects valid JSON and the model returns something like:

• •markdown fences
• •explanatory text before/after JSON
• •trailing commas
• •single quotes instead of double quotes

A typical error looks like this:

• •Error: JSON parsing error: Unexpected token ...
• •Failed to parse response as JSON
• •OutputParserException: Invalid JSON

Broken vs fixed pattern

// BROKEN: asks for JSON, but does not enforce it strictly
import { OpenAI } from "@llamaindex/openai";
import { StructuredOutputParser } from "llamaindex";

const llm = new OpenAI({ model: "gpt-4o-mini" });

const parser = StructuredOutputParser.fromNamesAndDescriptions({
  riskLevel: "low, medium, or high",
  summary: "short summary",
});

const prompt = `
Analyze this claim and return JSON:
Claim: Customer reports water damage after pipe burst.
`;

const response = await llm.complete(prompt);

// This often fails if the model adds prose or markdown fences
const parsed = parser.parse(response.text);

// FIXED: use a schema-driven output path and keep the prompt strict
import { OpenAI } from "@llamaindex/openai";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

const llm = new OpenAI({ model: "gpt-4o-mini" });

const ClaimSchema = z.object({
  riskLevel: z.enum(["low", "medium", "high"]),
  summary: z.string(),
});

const schema = zodToJsonSchema(ClaimSchema);

const prompt = `
You are a claims classifier.
Return ONLY valid JSON matching this schema:
${JSON.stringify(schema)}

Claim: Customer reports water damage after pipe burst.
`;

const response = await llm.complete(prompt);
const data = ClaimSchema.parse(JSON.parse(response.text));

If you’re using a higher-level LlamaIndex helper that supports structured outputs directly, prefer that over manual string parsing. Manual parsing is where most of these failures start.

Other Possible Causes

1) The model is not good at strict formatting

Some models drift into explanation mode even when instructed not to. Smaller models are more likely to emit invalid JSON under load or with long prompts.

// Problematic with weaker models or too much context
const llm = new OpenAI({ model: "gpt-3.5-turbo" });

Fix:

// Better adherence to structured output
const llm = new OpenAI({ model: "gpt-4o-mini", temperature: 0 });

2) You are parsing streamed partial output

If you parse before the full completion is assembled, you’ll get broken braces or truncated arrays.

// Wrong: parsing before stream completion finishes
for await (const chunk of stream) {
  parser.parse(chunk.text); // chunk is not complete JSON
}

Fix:

let fullText = "";
for await (const chunk of stream) {
  fullText += chunk.text;
}
parser.parse(fullText);

3) Your tool/function schema does not match the actual return value

This happens when a tool says it returns one shape, but your implementation returns another. LlamaIndex then tries to deserialize into the declared type and fails.

// Mismatch: schema expects { answer: string }
const toolResult = {
  result: "approved",
};

JSON.parse(JSON.stringify(toolResult)); // downstream parser may fail on shape mismatch

Fix the return contract:

const toolResult = {
  answer: "approved",
};

4) Hidden markdown fences or extra text in prompt templates

A lot of prompts accidentally encourage fenced code blocks:

const prompt = `
Return your answer as:
\`\`\`json
{ "riskLevel": "high" }
\`\`\`
`;

That often causes parsers to choke if they expect raw JSON only.

Use this instead:

const prompt = `
Return ONLY raw JSON.
No markdown.
No explanation.
`;

How to Debug It

1. •

   Log the raw model output before parsing

   • •Don’t inspect only the parsed object.
   • •Print response.text and look for fences, prose, trailing commas, or truncation.
2. •

   Check whether the failure happens on first parse or after streaming

   • •If you stream tokens, verify you’re waiting for completion.
   • •If non-streaming works but streaming fails, your aggregation logic is wrong.
3. •

   Validate against your exact schema

   • •Compare the actual payload with your Zod/Pydantic-style contract.
   • •A missing field can trigger the same downstream “JSON parsing error” symptoms.
4. •

   Reduce the problem to a minimal prompt

   • •Remove retrieval context, tool calls, and long instructions.
   • •If the minimal version works, one of those layers is polluting the output.

Prevention

• •Use strict schemas for any structured output path.
• •Set temperature: 0 for extraction and classification tasks.
• •Avoid manual JSON.parse() on raw LLM text unless you fully control the format.
• •Keep prompts explicit:
  • •“Return ONLY valid JSON”
  • •“No markdown”
  • •“No explanation”

If you’re building production workflows in banking or insurance, treat LLM output like an untrusted external API. Parse defensively, validate aggressively, and never assume “it said JSON” means you actually got valid JSON.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.