Tracing an LLM Application With OpenTelemetry

There is a published specification for what a model call looks like as a trace. It has names for the span, the attributes, the metrics and the events. Emit those names and every OTLP-speaking backend understands your traces without a mapping — which is the entire reason to prefer it over a vendor SDK.

Why the convention matters more than the SDK

OpenTelemetry is two things that get confused. One is a set of SDKs that batch and export telemetry. The other is the semantic conventions: an agreed vocabulary so that a span produced by a Python service and a span produced by a Go service describe the same thing with the same keys. For HTTP that vocabulary settled years ago (http.request.method, server.address). For generative AI it is the gen_ai.* namespace, and it is the part worth learning, because it outlives whichever library you use to emit it.

The practical payoff is that dashboards, alerts and queries you write against gen_ai.usage.output_tokens keep working when you swap instrumentation libraries, swap providers or move backends. A vendor SDK’s own field names do not survive any of those.

The span: name, kind, attributes

A call to a remote model is a CLIENT span. The convention names it {gen_ai.operation.name} {gen_ai.request.model} — so a chat completion against gpt-4o produces a span literally called chat gpt-4o, and a tool execution is execute_tool {gen_ai.tool.name}. The attributes that matter most:

Working instrumentation

Auto-instrumentation exists for the common SDKs and is the right starting point, but it is worth seeing the manual version once, because it is short and because it shows exactly where each attribute comes from.

import { trace, SpanKind, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("my-app/llm", "1.0.0");

export async function chat(model, messages, opts = {}) {
  return tracer.startActiveSpan(
    "chat " + model,
    {
      kind: SpanKind.CLIENT,
      attributes: {
        "gen_ai.operation.name": "chat",
        "gen_ai.provider.name": "openai",
        "gen_ai.request.model": model,
        "gen_ai.request.max_tokens": opts.maxTokens,
        "gen_ai.request.temperature": opts.temperature,
        "server.address": "api.openai.com",
      },
    },
    async (span) => {
      try {
        const res = await client.chat.completions.create({
          model, messages, ...opts,
        });

        span.setAttributes({
          "gen_ai.response.id": res.id,
          "gen_ai.response.model": res.model,
          "gen_ai.response.finish_reasons": res.choices.map((c) => c.finish_reason),
          "gen_ai.usage.input_tokens": res.usage.prompt_tokens,
          "gen_ai.usage.output_tokens": res.usage.completion_tokens,
        });
        return res;
      } catch (err) {
        span.setAttribute("error.type", err.constructor.name);
        span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
        throw err;
      } finally {
        span.end();
      }
    },
  );
}

Two details that are easy to get wrong. startActiveSpan rather than startSpan, so that anything the call fans out to — your vector search, your tool HTTP calls — becomes a child rather than an orphan. And span.end() in a finally, because a span that is never ended is not a missing span; it is a leak that keeps its context alive.

The two client metrics

Spans answer “what happened to this request”. Metrics answer “what is happening now”, are not sampled, and cost almost nothing to keep for a year. The convention defines two on the client side:

• gen_ai.client.operation.duration — a histogram in seconds, with the recommended bucket boundaries growing by powers of two from 0.01 to about 82, because model latency spans four orders of magnitude and the default HTTP buckets are useless for it.
• gen_ai.client.token.usage — a histogram of tokens, split by the gen_ai.token.type attribute with values input and output. A histogram rather than a counter, so you can ask about the distribution of prompt sizes rather than only their total.

Server-side conventions add gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token. If you are the one serving models, those two are the pair that separate queueing from generation; if you are a client, TTFT is something you have to time yourself at the first streamed chunk.

Message content is opt-in, by design

Prompts and completions are not span attributes in the current convention. They are carried as events, and capture is off unless you switch it on — in the language SDKs via an environment variable along the lines of OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. This is the correct default and worth understanding before you override it: turning it on ships every user message to your observability backend, under that backend’s retention and access control, which is a disclosure decision rather than a debugging one.

The shape of the content events has moved more than any other part of the spec — from per-role events, toward a single inference-details event carrying input and output messages. If you depend on it, read the version of the convention you are actually emitting rather than a blog post about it.

It is still experimental — pin it

As of mid-2026 the GenAI conventions remain marked experimental, which in OpenTelemetry’s process means attribute names can change between releases. They have: gen_ai.usage.prompt_tokens became gen_ai.usage.input_tokens, and gen_ai.system has been giving way to gen_ai.provider.name. Nothing warns you — your dashboard simply goes flat.

A related trap is mixed emitters. If one service uses auto-instrumentation from one library, another uses a vendor SDK, and a gateway emits its own spans, you can easily have three generations of the convention in one trace — one span reporting gen_ai.system, another gen_ai.provider.name, and a query that sums over only one of them silently under-reporting. Normalising at the collector with an attributes processor is usually less work than upgrading every emitter at once, and it gives you one place to fix the next rename.

So: pin the semantic-conventions package version explicitly rather than floating it, record which version you emit as a resource attribute, and when you upgrade, treat it as a dashboard migration. Where an SDK offers a stability opt-in switch (OpenTelemetry uses OTEL_SEMCONV_STABILITY_OPT_IN for exactly this kind of transition), setting it deliberately is better than inheriting whichever default your image happened to ship.