How to Fix 'output parsing error when scaling' in LangChain (TypeScript)

When LangChain throws OutputParserException or a message like Error: Failed to parse output, it usually means the model returned text that does not match the schema your chain expected. The “when scaling” part shows up when a prompt or parser works in small tests, then breaks under higher traffic, longer prompts, different model settings, or more varied inputs.

In TypeScript, this often happens with StructuredOutputParser, JsonOutputParser, RunnableSequence, or agent/tool flows where the model is supposed to emit strict JSON but drifts into prose, markdown fences, or partial objects.

The Most Common Cause

The #1 cause is this: you told the model to return structured output, but your prompt does not constrain it tightly enough, or you parse too early in the chain.

A common broken pattern is using a parser without injecting its format instructions into the prompt.

// Broken
import { ChatOpenAI } from "@langchain/openai";
import { StructuredOutputParser } from "langchain/output_parsers";
import { PromptTemplate } from "@langchain/core/prompts";

const parser = StructuredOutputParser.fromNamesAndDescriptions({
  name: "customer_name",
  risk_level: "low | medium | high",
});

const prompt = PromptTemplate.fromTemplate(`
Extract customer risk from this note:

{note}
`);

const llm = new ChatOpenAI({ modelName: "gpt-4o-mini", temperature: 0 });

const chain = prompt.pipe(llm).pipe(parser); // parser gets raw prose

// Fixed
import { ChatOpenAI } from "@langchain/openai";
import { StructuredOutputParser } from "langchain/output_parsers";
import { PromptTemplate } from "@langchain/core/prompts";

const parser = StructuredOutputParser.fromNamesAndDescriptions({
  name: "customer_name",
  risk_level: "low | medium | high",
});

const prompt = PromptTemplate.fromTemplate(`
Extract customer risk from this note.

{format_instructions}

Note:
{note}
`);

const llm = new ChatOpenAI({ modelName: "gpt-4o-mini", temperature: 0 });

const chain = prompt.pipe(llm).pipe(parser);

const formattedPrompt = await prompt.format({
  note: "Customer John Smith has repeated chargebacks.",
  format_instructions: parser.getFormatInstructions(),
});

If you use StructuredOutputParser, JsonOutputParser, or Zod-based output parsing, make sure the model sees explicit formatting instructions before generation. If you skip that step, parsing becomes a best-effort guess.

Other Possible Causes

1. Temperature is too high

At scale, even small randomness causes malformed JSON.

const llm = new ChatOpenAI({
  modelName: "gpt-4o-mini",
  temperature: 0.7, // risky for strict parsing
});

Use:

const llm = new ChatOpenAI({
  modelName: "gpt-4o-mini",
  temperature: 0,
});

2. The model wraps JSON in markdown fences

This is a classic Unexpected token \ in JSON at position...` failure.

// Bad model output
```json
{ "risk_level": "high" }

If your parser expects raw JSON, strip fences first or tighten the prompt:

```ts
const prompt = PromptTemplate.fromTemplate(`
Return ONLY valid JSON.
No markdown.
No explanation.

{format_instructions}

{note}
`);

3. Your schema is stricter than your prompt

If your Zod schema requires fields the model cannot infer reliably, parsing will fail.

import { z } from "zod";

const schema = z.object({
  customerId: z.string(),
  riskLevel: z.enum(["low", "medium", "high"]),
  reason: z.string(),
});

If customerId is not present in the source text, don’t force the model to invent it. Make it optional or derive it separately.

const schema = z.object({
  customerId: z.string().optional(),
  riskLevel: z.enum(["low", "medium", "high"]),
  reason: z.string(),
});

4. Truncation under load

Long prompts plus low token limits can cut off output mid-object.

const llm = new ChatOpenAI({
  modelName: "gpt-4o-mini",
  maxTokens: 80,
});

If the response is being cut off, increase token budget and shorten the prompt. Partial JSON almost always leads to:

• •SyntaxError: Unexpected end of JSON input
• •OutputParserException
• •failed retries if you have retry wrappers around the chain

How to Debug It

1. •Log the raw model output before parsing
   • •Don’t inspect only the parsed result.
   • •Capture exactly what LangChain received from the LLM.

const raw = await llm.invoke(promptValue);
console.log(raw.content);

2. •

   Check whether format instructions are actually in the final prompt

   • •Print the rendered prompt.
   • •Confirm {format_instructions} was replaced with real parser instructions.
3. •

   Reduce variables

   • •Set temperature: 0.
   • •Remove memory, tools, retries, and extra middleware.
   • •Re-run with one known input that fails consistently.
4. •

   Validate against your schema manually

   • •Paste raw output into a JSON validator.
   • •If using Zod, test parsing directly:

const result = schema.safeParse(JSON.parse(rawContent));
console.log(result);

If that fails before LangChain gets involved, your issue is output shape, not chain wiring.

Prevention

• •Always include parser instructions in the prompt when using structured output parsers.
• •Keep generation deterministic for parsing tasks:
  • •temperature: 0
  • •minimal top-p variance if exposed by your wrapper
• •Design schemas around what the model can reliably infer.
• •Add logging for raw completions in staging so you can catch malformed outputs before production traffic hits them.

If you want one rule to keep in mind: don’t ask an LLM for strict structure unless you control both the prompt and the output path. In LangChain TypeScript, most “output parsing error when scaling” incidents come from loose prompting plus brittle schemas.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.