Tracing Your Own Calls With Langfuse

An LLM trace answers a question ordinary application logs cannot: for this one user request, what was every prompt, in order, with what output, at what cost, and which step produced the answer that was wrong. Getting that is a day of work. Getting it in a form that outlives your choice of tool is the same day, done in a particular order.

Traces, spans and generations

That last row is the one teams skip and later wish they had not. A trace store with no scores can answer “what happened”; a trace store with scores can answer “show me the fifty worst answers this week”, which is where improvement actually starts.

Three ways to instrument

1. Decorators and context managers. Mark a function as a span; nested calls nest automatically. The least invasive option for code you own, and the right default for application code.
2. Framework integrations. A callback handler passed to a framework captures its internal steps without you writing anything. High coverage, and the coupling runs the wrong way — you get the framework’s idea of what a step is, which changes when it does.
3. OpenTelemetry. Emit standard spans and export them wherever your collector points. More setup, and the only option that makes the destination a configuration value rather than a code dependency.

The recommendation is a mixture: OpenTelemetry as the transport, your own decorators for the spans that matter to your domain, and framework integrations only where the framework is doing something you cannot otherwise see. That combination is why the naming conventions in the next section matter more than any particular SDK call.

What to record, and what not to

The fields below are the ones you will actually query. Record them deliberately, with the same names everywhere, or you will have a large quantity of data you cannot aggregate.

• The rendered prompt, not the template. The exact string sent, after variable substitution and after retrieval was inserted. Debugging without this is guesswork, and it is the single most valuable field in the trace.
• Model identifier and version. Including whatever the provider returns as the resolved model, which can differ from what you asked for when an alias is involved.
• Token counts, split. Input, output, and cached input separately. A single total makes cost analysis impossible, because the three are priced differently.
• Latency, split. Time to first token and total duration are different numbers with different causes.
• Your own identifiers. Request id, user or tenant id, feature name, prompt version, deployment version. These are what let you ask whether the regression started with a deploy.
• Not raw personal data, by default. Prompts contain whatever your users typed. A trace store is a copy of that, usually with a longer retention period and a wider access list than your primary database. Redact before export, or set retention and access deliberately — the specific hazard is personal data in LLM logs.

Keeping it portable

Tracing is the framework job that becomes stickiest fastest, because within a quarter your dashboards, alerts and evaluation reports all read one schema. Three cheap decisions keep the exit open.

Emit OpenTelemetry spans. Langfuse and its competitors accept OTLP, which means the destination is a collector endpoint in configuration rather than an SDK in your dependency tree. Changing backend then means changing an environment variable, which is the entire argument.

Name your own attributes and keep them stable. Whatever conventions exist for AI spans, your business attributes — tenant, feature, prompt version — are yours. Define them once in a constants module. Portability is mostly about attribute names surviving a move.

Wrap the tracing calls in one thin module. One function to start a trace, one to record a generation, one to attach a score. Ten lines of indirection, and it means the vendor SDK appears in one file rather than in eighty. The same discipline described in adopting at the seam applies here with the highest return of anywhere it is applied.

Using traces to find the broken step

Instrumentation that nobody queries is a cost with no benefit. Three workflows account for most of the value, and all three need the scores from the first section.

1. From complaint to trace in one lookup. The user reports a bad answer; you have the request id; the trace shows every prompt and every retrieved chunk. This alone justifies the work, and it requires only that your identifier is on the trace.
2. From score to pattern. Filter traces by a negative score and read twenty of them. Patterns emerge quickly — a document type that always retrieves badly, a prompt that fails on non-English input — and each becomes a test case rather than an anecdote.
3. From trace to evaluation set. Real failing inputs are the most valuable evaluation data available, because they came from actual users rather than from an imagination. Export them into the harness described in building an eval set and the loop closes.

Sampling, retention and cost

Traces containing full prompts and outputs are large — often larger than the application’s own database writes — and volume grows with traffic, not with team size. Decide three things before turning it on everywhere.

Sampling rate. Head sampling — a fixed percentage — is simple and drops exactly the rare failures you needed. Prefer keeping everything that errored, everything that scored badly and a small percentage of the rest. The trade-offs are in trace sampling.

Retention. Detailed traces for a short window, aggregates for a long one. Nobody reads a full prompt from four months ago; everybody wants the cost trend for the year.

Export path. Whatever the backend, you should be able to get your traces out in bulk. Check that before you have a year of them, not after — this is the concrete form of the lock-in the rest of the page is about.