How to Fix 'JSON parsing error when scaling' in LangChain (TypeScript)

What this error actually means

JSON parsing error when scaling in LangChain TypeScript usually means one of your chain steps expected valid JSON, but the model returned text that could not be parsed. It shows up a lot when you use structured outputs, tool calling, or OutputFixingParser/StructuredOutputParser and then add more concurrency, retries, or batch processing.

In practice, the failure often appears as a SyntaxError: Unexpected token ... in JSON at position ..., wrapped by LangChain inside parser-related errors like OutputParserException or AIMessage content that no longer matches the schema.

The Most Common Cause

The #1 cause is asking the model for JSON, but not forcing a strict schema or parser contract. The code works in a single test run, then fails under load because the model starts returning extra prose, code fences, or partial JSON.

Here’s the broken pattern and the fixed pattern side by side:

// BROKEN
import { ChatOpenAI } from "@langchain/openai";
import { PromptTemplate } from "@langchain/core/prompts";

const llm = new ChatOpenAI({ model: "gpt-4o-mini", temperature: 0 });

const prompt = PromptTemplate.fromTemplate(`
Return JSON with fields:
- name
- riskScore

Customer: {customer}
`);

const chain = prompt.pipe(llm);

const result = await chain.invoke({
  customer: "ACME Corp",
});

// This often breaks when the model returns markdown or extra text
const data = JSON.parse(result.content as string);

// FIXED
import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";
import { StructuredOutputParser } from "@langchain/core/output_parsers";
import { PromptTemplate } from "@langchain/core/prompts";

const schema = z.object({
  name: z.string(),
  riskScore: z.number(),
});

const parser = StructuredOutputParser.fromZodSchema(schema);

const llm = new ChatOpenAI({ model: "gpt-4o-mini", temperature: 0 });

const prompt = PromptTemplate.fromTemplate(`
{format_instructions}

Customer: {customer}
`);

const chain = prompt.pipe(llm).pipe(parser);

const result = await chain.invoke({
  customer: "ACME Corp",
  format_instructions: parser.getFormatInstructions(),
});

// Already validated
console.log(result.name, result.riskScore);

If you are using ChatOpenAI, AzureChatOpenAI, or any other chat model wrapper, the important part is the parser contract. Don’t depend on JSON.parse() against raw model text unless you enjoy intermittent production failures.

Other Possible Causes

1) Code fences in the model output

The model returns:

{
  "name": "ACME Corp",
  "riskScore": 82
}

That looks valid to humans, but many parsers choke if you try to parse the whole string without stripping fences.

// Example bad output handling
const raw = "```json\n{\"name\":\"ACME Corp\",\"riskScore\":82}\n```";
JSON.parse(raw); // SyntaxError

Fix it by using a parser that handles formatting instructions, or strip fences before parsing if you absolutely must.

2) Streaming partial JSON into a parser

If you parse while tokens are still arriving, you will eventually hit incomplete JSON.

// BAD: parsing before stream is complete
let buffer = "";
for await (const chunk of stream) {
  buffer += chunk.content;
  JSON.parse(buffer); // fails on partial content
}

Only parse after the full completion is assembled. If you need incremental updates, use a streaming-friendly event format instead of raw JSON.

3) Tool calling mismatch

If your prompt expects plain JSON but your model is configured for tools/function calling, LangChain may return an AIMessage with tool calls instead of JSON text.

const llm = new ChatOpenAI({
  model: "gpt-4o-mini",
  temperature: 0,
}).bindTools([
  {
    name: "score_customer",
    description: "Score customer risk",
    schema: {
      type: "object",
      properties: {
        name: { type: "string" },
        riskScore: { type: "number" },
      },
      required: ["name", "riskScore"],
    },
  },
]);

In this setup, inspect message.tool_calls instead of parsing message.content.

4) Schema drift between prompt and parser

Your prompt says one shape, your parser expects another. This happens when someone changes a field name in one place and forgets the other.

// Prompt says risk_score, schema expects riskScore
const schema = z.object({
  name: z.string(),
  riskScore: z.number(),
});

Keep one source of truth. Generate format instructions from the same Zod schema you validate against.

How to Debug It

1. •

   Log the raw LLM output before parsing

   • •Print result.content, not just parsed objects.
   • •Look for markdown fences, trailing commentary, or truncated output.
2. •

   Check whether you are parsing streamed content

   • •If you use .stream() or async iterators, confirm parsing happens only after completion.
   • •Partial chunks will fail even if the final answer would have been valid.
3. •

   Verify your output contract

   • •If you use StructuredOutputParser, confirm format_instructions is injected into the prompt.
   • •If you use Zod schemas, ensure the runtime schema matches what the prompt requests.
4. •

   Inspect retries and batching

   • •Under scale, failures often come from one bad response in a batch.
   • •Wrap batch jobs with per-item logging so you can identify which input triggers malformed output.

A practical debug pattern:

try {
  const raw = await chain.invoke(input);
  console.log("RAW:", raw.content ?? raw);
} catch (err) {
  console.error("LangChain error:", err);
}

If you see OutputParserException, focus on formatting and schema alignment first. If you see SyntaxError, inspect whether something upstream already converted valid structured data into a string incorrectly.

Prevention

• •Use StructuredOutputParser.fromZodSchema() or another typed parser for every structured response.
• •Keep prompt schema and validation schema in one place; don’t duplicate field definitions across files.
• •Avoid parsing streamed chunks directly unless your protocol is designed for incremental decoding.
• •Set temperature: 0 for extraction workflows where consistency matters more than creativity.
• •Add tests with malformed outputs:
  • •plain prose
  • •fenced JSON
  • •missing fields
  • •extra fields

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.