Observability Vendors vs Rolling Your Own
4 min read · updated August 3, 2026
Every comparison of LLM observability tools is a feature matrix, and feature matrices go stale in a quarter. The parts that do not change are what each approach makes hard to leave and what happens to your data — so this page compares on those, and dates everything else.
The question is not which tool
It is which of these three you are actually deciding:
- Where does capture happen — in your code via an SDK, in your infrastructure via an OpenTelemetry collector, or in the request path via a proxy or gateway?
- What data model do the records use — OpenTelemetry spans, or a vendor’s own trace/run/generation objects?
- Which of the adjacent jobs do you want in the same product — evals, datasets, prompt management, annotation queues, playgrounds?
Those three decide your lock-in almost entirely, and they are orthogonal to whether a given product has a nicer timeline view.
Four axes that decide it
| What to evaluate, in order of how expensive it is to change later | Description |
|---|---|
| Data model | If records are OpenTelemetry spans with gen_ai.* attributes, any OTLP backend can receive them and migration is a config change. If they are a proprietary run/generation object, migration means rewriting instrumentation and abandoning history. This is the largest single lock-in axis and the cheapest one to get right at the start. |
| Capture path | An SDK in your code means a dependency and per-language coverage. A collector means an operational component but no application coupling. A proxy or gateway means zero code change and a new hop in the request path — capture happens even for services you did not instrument, at the cost of routing your traffic through it. |
| Where the data lives | Self-hosted keeps prompts and completions inside your perimeter, which for regulated workloads is the whole decision, and costs you an operational burden. Hosted is less work and puts user content in a third party's system under their retention and access control — a data-protection question with a data-protection process, not an engineering preference. |
| Export path out | Ask this before signing, not after: can you get a full historical export in a documented format, on demand, without support involvement? A product that can only be exited by losing your history has a switching cost equal to your entire trace history. |
The families, and what each locks in
As of mid-2026 the landscape sorts into four families. Products move between them, and several of the names below appear in more than one, so verify against current documentation before deciding — this is a map, not a catalogue.
1 · The OpenTelemetry-native path
Emit gen_ai.* spans, run a collector, send OTLP to a general APM or trace backend you already operate. Instrumentation libraries in this family — OpenLLMetry from Traceloop, and OpenInference from Arize — are open specifications plus SDKs that produce OTel-compatible spans, and several dedicated LLM observability products additionally accept OTLP on an ingest endpoint.
Locks in: almost nothing on the data model, which is the point. Costs you: the LLM-specific affordances — side-by-side output comparison, prompt playgrounds, annotation queues — which a general trace backend does not have, because a span viewer is not a prompt-debugging tool.
2 · Dedicated LLM observability products
Langfuse, Arize Phoenix, LangSmith, Braintrust, Weights & Biases Weave and others. Built around traces of model calls with prompt and completion rendering, plus evals, datasets and prompt management in the same product. Some are open source and self-hostable (Langfuse and Phoenix are the common examples, though licence terms vary by component and by version — read the licence, not the README badge); others are hosted only, or self-hostable on enterprise terms.
Locks in: varies enormously with the data model. One that ingests and exports OTLP is a much shallower commitment than one whose SDK is the only way in. The deeper lock-in is usually not the traces at all — it is the evals, datasets and annotations you build up inside it, which have no standard format anywhere. Costs you: a second observability system to operate or pay for, and a second place engineers have to look during an incident.
3 · Proxy and gateway capture
A component in the request path that sees every call and records it, with no application instrumentation at all. Helicone is a well-known example built explicitly as a proxy; multi-provider gateways generally capture the same data as a side effect of routing.
Locks in: your request path, which is the most consequential dependency on this page — it is now in the availability and latency budget of every model call, so its failure modes are yours. Gets you: complete coverage without touching any code, including services nobody instrumented, plus the fields only something in the path knows — which provider served it, how many attempts, what it cost.
4 · Rolling your own
A table, a dashboard tool you already run, and a few queries. Every SQL snippet in this cluster is this option.
Locks in: nothing. Costs you: the things that look small and are not — a trace viewer, content storage with retention, redaction, sampling, an eval runner, an annotation UI, and the maintenance of all of it. It is the right answer far more often than vendors suggest and far less often than engineers estimate; the honest version is that the request log and the cost queries are genuinely a weekend, and the trace UI and eval platform are not.
Pricing models, not prices
Published prices change too fast to be worth quoting, and the shape of the meter is what determines whether a bill surprises you. Four shapes are common; identify which one you are being offered and model it against your own volume before comparing headline numbers.
| How these products meter | Description |
|---|---|
| Per trace or per event ingested | Predictable and scales directly with traffic. The trap is agent workloads: one user action can be forty spans, so per-event pricing multiplies against a number you do not control. |
| Per ingested volume (bytes) | Scales with prompt and completion size rather than request count. Long-context and RAG workloads are dramatically more expensive under this meter than request counts suggest. |
| Per seat | Decoupled from traffic, which is comfortable, but tends to discourage exactly the broad access that makes observability useful. Check whether read-only access needs a seat. |
| Self-hosted / open source | No licence meter, and a real cost in storage, operations and the engineer who owns it. Check the licence boundary carefully: several products in this space are open-core, with SSO, RBAC or retention controls in the commercial tier. |
Whichever shape, run the arithmetic against your own numbers before committing, and model it at three times current volume. And note the interaction with sampling: under a per-event or per-byte meter, the sampling policy is a pricing decision, and biased tail sampling can change the bill by an order of magnitude without losing anything you look at.
A decision procedure
- Emit OpenTelemetry
gen_ai.*spans regardless. This is the one choice that is nearly free to make and expensive to reverse. It keeps every other door open. - Own the request log yourself, always. One narrow row per request, in your own database, unsampled. It is small, it costs nothing, and it means cost attribution and per-tenant billing never depend on a vendor’s retention window.
- Buy the LLM-specific product for the LLM-specific jobs. Prompt debugging, output comparison, eval runs, annotation. That is where these products are clearly better than a generic trace backend and where building your own is a project rather than a weekend.
- Prefer self-hosting if user content is sensitive. Not because hosted vendors are careless, but because the decision to send prompts to a third party is a data-protection decision that should be made explicitly and documented, not made by whoever added the SDK.
- Test the export before you depend on it. Export your history once in the first month. If it is hard in month one, it is impossible in year two.