How to Fix 'JSON parsing error' in LlamaIndex (Python)

When you see JSON parsing error in LlamaIndex, it usually means one of the components expected structured JSON and got something else back. In practice, this shows up when using structured outputs, output parsers, tool calling, or reading malformed JSON from a file or API response.

The key point: the failure is usually not “LlamaIndex is broken.” It means the model output, prompt, or input payload did not match the JSON shape LlamaIndex was trying to parse.

The Most Common Cause

The #1 cause is asking an LLM to return JSON but not constraining the output enough. LlamaIndex then tries to parse free-form text as JSON and throws errors like:

• •ValueError: Failed to parse JSON
• •JSONDecodeError: Expecting value
• •OutputParserException depending on the component in use

Here’s the broken pattern versus the fixed pattern.

# BROKEN
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI

llm = OpenAI(model="gpt-4o-mini")
Settings.llm = llm

response = llm.complete("""
Return the customer record as JSON:
name: Alice
age: 34
""")

# This often fails if the model returns:
# ```json
# {"name": "Alice", "age": 34}
# ```
data = response.json()  # ValueError / JSONDecodeError

# FIXED
from pydantic import BaseModel
from llama_index.core.program import LLMTextCompletionProgram
from llama_index.llms.openai import OpenAI

class Customer(BaseModel):
    name: str
    age: int

llm = OpenAI(model="gpt-4o-mini")

program = LLMTextCompletionProgram.from_defaults(
    output_cls=Customer,
    llm=llm,
    prompt_template_str="""
Extract the customer record.
Return only valid JSON matching this schema:
{name: string, age: integer}

Text:
{input}
"""
)

result = program(input="Alice is 34 years old.")
print(result)

If you’re using QueryEngine with structured responses, this same issue appears when the prompt allows extra prose. The fix is still the same: make the output contract explicit.

Other Possible Causes

1) Malformed JSON in your source data

If you’re loading a file or tool response that already contains bad JSON, parsing fails before the LLM even gets involved.

# BROKEN
import json

raw = '{"customer_id": 12, "name": "Alice",}'  # trailing comma
data = json.loads(raw)  # JSONDecodeError: Expecting property name enclosed in double quotes

# FIXED
raw = '{"customer_id": 12, "name": "Alice"}'
data = json.loads(raw)

This often happens when a webhook payload or local fixture was hand-edited.

2) Markdown fences around JSON

Models frequently return fenced blocks like json ... . LlamaIndex parsers expect raw JSON unless you strip wrappers first.

# BROKEN OUTPUT EXAMPLE
"""
```json
{"policy_number": "P123", "status": "active"}

"""

```python
# FIXED APPROACH
import json

text = '{"policy_number": "P123", "status": "active"}'
data = json.loads(text.strip())

If you can’t control the model fully, post-process with a fence stripper before parsing.

3) Wrong response format for function/tool calling

If your agent expects tool-call style structured data but the model returns plain text, parsing breaks inside classes like ReActAgent, FunctionCallingAgentWorker, or related response handlers.

# BROKEN CONFIG IDEA
# Agent expects tool calls but model isn't configured for them.
from llama_index.core.agent import ReActAgent

agent = ReActAgent.from_tools(tools=[], llm=llm)
response = agent.chat("Create a claim object as JSON")

Fix it by aligning the agent type with the model capability and expected output mode.

# FIXED IDEA
# Use a structured extraction program instead of free-form chat.
from llama_index.core.program import LLMTextCompletionProgram

4) Parser mismatch in custom pipelines

If you plug a custom parser into a pipeline and its expected schema doesn’t match what upstream produces, you’ll get parse errors even though each component works alone.

# BROKEN: parser expects keys that aren't present
expected = {"claim_id", "amount", "currency"}
payload = {"claimId": "C-1", "amount": 1000}

Make sure field names match exactly. claimId and claim_id are not interchangeable unless you normalize them.

How to Debug It

1. •

   Print the raw model output before parsing

   • •Don’t inspect only the final exception.
   • •Log the exact string returned by the LLM.
   • •Look for markdown fences, commentary, truncated text, or invalid quotes.
2. •

   Check whether the failure is from generation or ingestion

   • •If you call json.loads(...), it’s likely your own parsing step.
   • •If it happens inside LLMTextCompletionProgram, PydanticOutputParser, or an agent worker, it’s likely structured-output mismatch.
3. •

   Reduce to a minimal prompt

   • •Remove context windows, retrieval chunks, and long system prompts.
   • •Test with one short input and one strict schema.
   • •If it works there but fails in production, your prompt is too loose or too long.
4. •

   Validate against a schema early

   • •Use Pydantic models instead of ad hoc dicts.
   • •Catch bad fields immediately rather than passing malformed objects downstream.

from pydantic import BaseModel, ValidationError

class Claim(BaseModel):
    claim_id: str
    amount: float

try:
    claim = Claim.model_validate({"claimId": "C-1", "amount": 1000})
except ValidationError as e:
    print(e)

Prevention

• •Use Pydantic models or explicit schemas for every structured LlamaIndex output.
• •Never rely on “please return JSON” alone; enforce structure in code.
• •Log raw LLM responses during development so you can see exactly what failed to parse.
• •Normalize field names and strip markdown fences before deserialization.
• •Keep prompts short and deterministic when extracting structured data.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.