LLM Calls Are the New Blind Spot

Many teams we talk to have Python services calling LLM providers in production. Chat features, RAG pipelines, summarization, AI agents. It's everywhere.

The calls go out and responses come back, but that's about all you know. You have no idea how long the model took, how many tokens it burned, or whether it even used the model you thought you configured.

If you look at your traces today, you'll probably see an outgoing HTTP request to something like api.openai.com. You get a status code and a duration. Nothing else. The actual GenAI context (which model, how many tokens, what finish reason came back) is just missing from your traces.

This is a problem we kept running into ourselves, and hearing about from users. So we decided to fix it.

What You're Missing Without GenAI-Specific Spans

Where'd those 3 seconds go? LLM calls regularly take anywhere from 1 to 5+ seconds. That time gets buried inside a generic HTTP client span. When a request is slow, you can't tell whether the bottleneck was your application code, the network, or the model sitting there thinking. You need a span that represents the LLM call itself, separate from the HTTP transport.

You can't track token spend. Every LLM call consumes tokens, and tokens cost money. Without attributes like gen_ai.usage.input_tokens and gen_ai.usage.output_tokens on your spans, there's no way to attribute cost back to a specific endpoint, feature, or user. You end up looking at your monthly OpenAI bill and shrugging.

You don't know which model actually responded. You set gpt-5-mini in code, but did the API actually use it? Without gen_ai.request.model and gen_ai.response.model on the span, you'd never know.

What We Added

We've added OpenTelemetry GenAI instrumentations to the Odigos Python auto-instrumentation distro. If your Python service is already running with Odigos and it calls one of the supported providers, you'll start seeing GenAI spans in your traces. You don't have to change your code or install anything new.

Here are the 7 instrumentations we bundled:

Provider	Package	What It Covers
OpenAI	opentelemetry-instrumentation-openai-v2	Chat completions, embeddings
OpenAI Agents	opentelemetry-instrumentation-openai-agents-v2	Agent, tool, and generation spans from the Agents SDK
Anthropic	opentelemetry-instrumentation-anthropic	Messages API
LangChain	opentelemetry-instrumentation-langchain	Chains, agents, LLM calls through LangChain
Google GenAI	opentelemetry-instrumentation-google-genai	Google's GenAI client
Vertex AI	opentelemetry-instrumentation-vertexai	Google Cloud Vertex AI
Weaviate	opentelemetry-instrumentation-weaviate	Vector DB queries (common in RAG setups)

These all come from the upstream OpenTelemetry community and follow the OTel GenAI semantic conventions. We bundle them into the Odigos distro and auto-load them alongside the rest of our instrumentations.

What the Traces Look Like

Once these are active, the spans look like this.

OpenAI Chat Completion

When your code calls client.chat.completions.create(model="gpt-5-mini", ...), you'll see a span like this:

• Span name: chat gpt-5-mini
• gen_ai.system: openai
• gen_ai.request.model: gpt-5-mini
• gen_ai.response.model: gpt-5-mini
• gen_ai.usage.input_tokens: 12
• gen_ai.usage.output_tokens: 10
• gen_ai.response.id: chatcmpl-abc123...

The model, token counts, and response ID are all right there. If you're tracking costs, you can calculate per-request spend from those attributes.

Anthropic Messages

A call to client.messages.create(model="claude-3-5-sonnet-20241022", ...) produces:

• Span name: anthropic.chat
• gen_ai.system: Anthropic
• gen_ai.request.model: claude-3-5-sonnet-20241022
• gen_ai.response.model: claude-3-5-sonnet-20241022
• gen_ai.usage.input_tokens: 15
• gen_ai.usage.output_tokens: 8

It looks the same as the OpenAI one. That's the whole point of semantic conventions.

LangChain

If you're using LangChain's ChatOpenAI wrapper, the instrumentation sees through the abstraction:

• Span name: ChatOpenAI.chat
• gen_ai.system: openai
• gen_ai.request.model: gpt-5-mini
• gen_ai.response.model: gpt-5-mini

Even though your code just calls llm.invoke("Say hello"), the span still shows the underlying model and provider. Saves you from digging through LangChain's abstractions when something breaks.

How They Fit Into Existing Traces

These spans nest inside your existing HTTP and framework traces like you'd expect. Flask endpoint calls OpenAI? You'll see the Flask server span as the parent and the chat gpt-5-mini span as a child. One trace, everything's connected.

Getting Started

If you're already running Odigos with Python services, there's nothing to do. The GenAI instrumentations are included in the latest version of the distro. Deploy your services and the spans will start appearing for any supported provider calls.

If you're new to Odigos, check out the getting started guide to set up auto-instrumentation for your cluster.

The OTel GenAI semantic conventions are still in development status, so expect some attributes to change over time. We'll keep tracking upstream and update the distro as things settle.