LlamaIndex Tutorial (TypeScript): adding observability for intermediate developers

This tutorial shows you how to add observability to a TypeScript LlamaIndex app so you can trace retrieval, prompts, and model calls instead of guessing where latency or bad answers come from. You need this when your agent works in development but becomes hard to debug once it starts calling real tools, real models, and real data.

What You'll Need

• •Node.js 18+ installed
• •A TypeScript project with npm or pnpm
• •LlamaIndex TS packages:
  • •llamaindex
  • •@llamaindex/openai
• •An OpenAI API key
• •A Phoenix account or local Phoenix instance for tracing
• •Environment variables set in a .env file:
  • •OPENAI_API_KEY
  • •PHOENIX_API_KEY if using hosted Phoenix
  • •PHOENIX_COLLECTOR_ENDPOINT if using a local collector

Step-by-Step

1. •Install the dependencies and make sure your project can talk to both OpenAI and Phoenix. For observability, you want the app to emit traces from the first request, not after you’ve already shipped a broken chain.

npm install llamaindex @llamaindex/openai @arizeai/phoenix-client dotenv
npm install -D typescript tsx @types/node

2. •Add your environment variables. Keep them out of source control and load them at process start so both the LLM client and the tracing client can read them.

OPENAI_API_KEY=sk-your-openai-key
PHOENIX_API_KEY=px-your-phoenix-key
PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006/v1/traces

3. •Create a small index that uses a real embedding model and query engine. This gives you something concrete to trace: document ingestion, retrieval, and response generation.

import "dotenv/config";
import { Document, VectorStoreIndex } from "llamaindex";
import { OpenAIEmbedding, OpenAI } from "@llamaindex/openai";

async function main() {
  const docs = [
    new Document({ text: "Claims under $5,000 are auto-approved." }),
    new Document({ text: "Fraud review is required for duplicate bank accounts." }),
  ];

  const index = await VectorStoreIndex.fromDocuments(docs, {
    embedModel: new OpenAIEmbedding({ model: "text-embedding-3-small" }),
  });

  const queryEngine = index.asQueryEngine({
    llm: new OpenAI({ model: "gpt-4o-mini" }),
  });

  const result = await queryEngine.query({
    query: "What claims get auto-approved?",
  });

  console.log(result.toString());
}

main();

4. •Wire in tracing with Phoenix before you run any LlamaIndex code. The important part is initializing observability first so every downstream call gets captured consistently.

import "dotenv/config";
import { ArizePhoenixExporter } from "@arizeai/phoenix-client";

export function initObservability() {
  const exporter = new ArizePhoenixExporter({
    apiKey: process.env.PHOENIX_API_KEY,
    endpoint: process.env.PHOENIX_COLLECTOR_ENDPOINT,
  });

  return exporter;
}

5. •Connect the exporter to your app flow and wrap the query path with a span-friendly structure. In practice, this means keeping your observable boundary around indexing and querying so you can inspect timings and failures by request.

import "dotenv/config";
import { Document, VectorStoreIndex } from "llamaindex";
import { OpenAIEmbedding, OpenAI } from "@llamaindex/openai";
import { initObservability } from "./observability";

async function main() {
  const tracer = initObservability();

  const docs = [
    new Document({ text: "Claims under $5,000 are auto-approved." }),
    new Document({ text: "Fraud review is required for duplicate bank accounts." }),
  ];

  await tracer.export({
    name: "build-index",
    attributes: { app: "claims-assistant" },
  });

  const index = await VectorStoreIndex.fromDocuments(docs, {
    embedModel: new OpenAIEmbedding({ model: "text-embedding-3-small" }),
  });

  const queryEngine = index.asQueryEngine({
    llm: new OpenAI({ model: "gpt-4o-mini" }),
  });

  const result = await queryEngine.query({
    query: "What claims get auto-approved?",
  });

  console.log(result.toString());
}

main();

6. •Run the app and inspect the trace in Phoenix. If everything is wired correctly, you should see at least one top-level run for indexing plus the query path with model calls underneath it.

npx tsx src/index.ts

Testing It

Start Phoenix locally or point PHOENIX_COLLECTOR_ENDPOINT at your hosted collector before running the script. Then execute the TypeScript file and confirm that traces appear with the expected operation names like build-index and the LLM request metadata.

If you only see console output but no traces, check these first:

• •The API key is present in the runtime environment
• •The collector endpoint matches your Phoenix setup
• •Your process loads .env before creating the exporter

A good smoke test is to change the query text and rerun it. You should see a fresh trace for each execution, along with different retrieval results if your documents are small enough to make ranking obvious.

Next Steps

• •Add per-request metadata like tenant ID or claim type to traces so production debugging is usable.
• •Instrument tool calls separately from retrieval so you can tell whether failures come from search, reasoning, or external APIs.
• •Add evaluation runs on top of traces so you can compare answer quality across prompt changes.

Keep learning

• •The complete AI Agents Roadmap — my full 8-step breakdown
• •Free: The AI Agent Starter Kit — PDF checklist + starter code
• •Work with me — I build AI for banks and insurance companies

By Cyprian Aarons, AI Consultant at Topiax.