How to Fix 'JSON parsing error during development' in LlamaIndex (TypeScript)

What this error means

JSON parsing error during development in LlamaIndex TypeScript usually means one of two things: the model returned text that is not valid JSON, or your code tried to parse a response that was never meant to be JSON in the first place. It shows up most often when using structured outputs, query engines with responseMode: "compact", or custom prompts that tell the model to “return JSON” without enforcing a schema.

In practice, you’ll see errors like:

• •SyntaxError: Unexpected token ... in JSON at position ...
• •Error: Failed to parse response as JSON
• •OutputParserException: Could not parse output into the expected schema

The Most Common Cause

The #1 cause is this: you asked the LLM for JSON, but you did not constrain the output tightly enough, so it returned markdown, commentary, or slightly invalid JSON.

This happens a lot with OpenAI + QueryEngine + a prompt like “respond in JSON only”. That is not enough. You need either a structured output API or a parser that can tolerate and validate the shape.

Broken vs fixed pattern

// BROKEN
import { OpenAI } from "llamaindex";

const llm = new OpenAI({
  model: "gpt-4o-mini",
});

const prompt = `
Return a JSON object with keys:
- customerName
- riskScore

Only return JSON.
`;

const response = await llm.complete({ prompt });

// This blows up if the model adds markdown, code fences, or commentary.
const data = JSON.parse(response.text);

// FIXED
import { z } from "zod";
import { OpenAI } from "llamaindex";

const RiskSchema = z.object({
  customerName: z.string(),
  riskScore: z.number(),
});

const llm = new OpenAI({
  model: "gpt-4o-mini",
});

const prompt = `
Extract customer name and risk score from the input.
Return only valid JSON matching this schema:
{"customerName": string, "riskScore": number}
`;

const response = await llm.complete({ prompt });

// Strip fences if your model sometimes adds them.
const raw = response.text.replace(/^```json\s*/i, "").replace(/```$/i, "");
const parsed = RiskSchema.parse(JSON.parse(raw));

If you are using LlamaIndex’s structured extraction APIs, prefer those over hand-rolled JSON.parse(). The point is to make invalid output fail early and predictably.

Other Possible Causes

1) You are parsing streaming chunks as full JSON

If you enabled streaming and try to parse each chunk independently, you’ll get partial JSON and hard failures.

// BAD
for await (const chunk of stream) {
  JSON.parse(chunk); // chunk is incomplete
}

Fix it by buffering until the stream ends.

let buffer = "";
for await (const chunk of stream) {
  buffer += chunk;
}
const data = JSON.parse(buffer);

2) Your prompt includes examples wrapped in markdown fences

Models often mirror formatting. If your few-shot examples show fenced code blocks, the model may return:

{
  "foo": "bar"
}

That breaks naive parsing if you don’t strip fences first.

const cleaned = text
  .replace(/^```json\s*/i, "")
  .replace(/```$/i, "")
  .trim();

3) The model is returning tool-call text instead of plain JSON

This happens when your agent is configured for function/tool calling but your code expects raw text. In LlamaIndex TS, check whether you’re using an agent runner or a plain completion call.

// Mismatch: expecting raw JSON from an agent tool response
const result = await agent.chat("Summarize claims");
JSON.parse(result.response); // often wrong shape

Use the actual structured payload returned by the agent or inspect the tool call result before parsing.

4) Your environment is mixing ESM/CJS imports and corrupting runtime behavior

This one is less obvious, but bad module setup can produce weird serialization issues around custom response handlers.

Check your config:

{
  "type": "module",
  "compilerOptions": {
    "module": "NodeNext",
    "moduleResolution": "NodeNext"
  }
}

And make sure imports match your runtime:

import { OpenAI } from "llamaindex";

Avoid mixing default and named imports incorrectly across transpilation boundaries.

How to Debug It

1. •

   Log the raw model output before parsing

   • •Print response.text exactly as received.
   • •Look for fences, leading prose, trailing commas, or partial content.
2. •

   Check whether you are dealing with full text or streaming chunks

   • •If using streaming APIs, confirm you buffer before parsing.
   • •A chunk that looks like { "a": is not valid JSON yet.
3. •

   Verify which LlamaIndex API path you’re using

   • •llm.complete() returns plain text.
   • •Agent/tool workflows may return structured objects.
   • •Query engines may wrap responses in metadata objects depending on configuration.
4. •

   Validate against a schema before trusting the payload

   • •Use zod, valibot, or another validator.
   • •If validation fails, dump the raw payload and compare it against expected shape.

A good debug loop looks like this:

console.log("RAW RESPONSE:", response.text);

try {
  const parsed = JSON.parse(response.text);
  console.log("PARSED:", parsed);
} catch (err) {
  console.error("JSON PARSE FAILED:", err);
}

If the raw output contains anything other than strict JSON, fix the generation step first. Do not keep stacking parsers on top of bad prompts.

Prevention

• •

  Use schema-driven outputs for anything important.

  • •Prefer Zod validation over ad hoc string parsing.
  • •Treat “return only JSON” as a hint, not a guarantee.
• •

  Keep prompts boring and explicit.

  • •Say exactly which keys are required.
  • •Avoid examples that include markdown unless you plan to strip it.
• •

  Separate generation from parsing.

  • •First get text.
  • •Then normalize it.
  • •Then validate it.

That pattern saves time when this error shows up again in production logs under load.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.