Observability for Agents: Tracing a 40-Step Run
5 min read · updated August 3, 2026
A forty-step run that went wrong is not one incident, it is forty decisions of which one or two were bad. Logging the request and the response gives you the first and the last. The debugging happens entirely in between.
Why request logging is not enough
The questions people actually arrive with are all cross-step: at which step did it stop making sense, what did it see just before that, why did it call that tool twice, where did the eleven dollars go, and is this the same failure as last Thursday. None of them are answerable from a pair of log lines, and all of them are answerable from a trace with the right attributes.
Use OpenTelemetry rather than inventing a schema. Its GenAI semantic conventions already name the fields — gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.tool.name — which means your traces are readable by tooling you have not chosen yet, and your agent spans sit in the same trace as the HTTP request that started them. That last property is worth more than any agent-specific dashboard.
The span tree
agent.run task, run id, halt reason, TOTAL cost ├─ agent.step (0) step index │ ├─ gen_ai.chat model, tokens, finish reason, latency │ └─ agent.tool list_dir args hash, bytes out, error class ├─ agent.step (1) │ ├─ gen_ai.chat │ ├─ agent.tool read_file <- parallel siblings share a parent │ └─ agent.tool read_file ├─ agent.compaction tokens before/after, what was dropped ├─ agent.step (2..38) └─ agent.step (39) └─ gen_ai.chat finish_reason = tool_calls, halted here
Four properties of this layout matter more than the naming. The step span is a real span rather than an event, so step duration and step cost are aggregatable — “p95 step latency by tool” is a query, not a project. Parallel tool calls are siblings under one step, which is how the trace shows concurrency you may not have intended. Compaction is a span because it is a paid, lossy, load-bearing event and it is the first thing to check when an agent forgets something. And every span carries the run id, so a sub-agent’s spans join the parent trace instead of forming an orphan you have to correlate by timestamp.
Attributes that earn their place
- The exact messages array, by reference. Put a content hash on the span and the payload in object storage keyed by that hash. Reproducing a failure requires the input verbatim, and “a summary of the prompt” has never once been enough. The hash doubles as a deduplicator: identical prefixes across runs share a blob.
- Tokens per step, both directions. Plotted against step index this is the single most diagnostic chart in agent operations. A smooth ramp is normal; a step change means a tool returned something enormous, and you can see exactly which one.
- Cumulative cost on the root span. Not per-call cost summed later at query time — the run is the unit anyone budgets in, and you want to sort runs by cost without a join.
- Halt reason as a first-class attribute. The enumeration from stopping conditions. Its distribution over a week is the health metric for an agent, and it is one string.
- Arguments hashed, and stored by reference. Raw arguments frequently contain user data; the hash is what you need for the repeat-detection query anyway. Same treatment for tool output, plus a
bytesattribute so you can find the tool that blew up the context without reading any payloads. - Retry and provider metadata. Attempt number, the provider’s request id, whether the response was served from a prompt cache. When a run behaves differently from an identical run an hour earlier, these three explain it more often than anything in your own code.
And two things to keep out. Never put prompt content in a span name or in a metric label — that is unbounded cardinality and it will take your metrics backend down. Never log raw arguments for tools that take credentials or personal data; the hash-plus-blob pattern exists so that access to payloads can be governed separately from access to traces.
Question to attribute
| At 2am you ask | Description |
|---|---|
| Where did it go wrong? | Step index on every span, plus the first step whose gen_ai.chat has an unexpected finish reason or whose tool span carries an error class. Scan the step list, not the log. |
| What did it see? | Content hash on the step span, resolved to the stored messages blob. This is the only attribute that makes a run reproducible. |
| Why did it repeat itself? | Args hash on tool spans. Group by hash within a run: any count above one is a repeat, above two is a loop. |
| Where did the money go? | Tokens per step and cumulative cost on the root. The answer is nearly always one tool returning too much, visible as a discontinuity. |
| Is this the same as Thursday? | Halt reason plus the tool name of the last failing span. Two attributes turn 'agent misbehaved' into a groupable incident class. |
| Did compaction eat it? | The compaction span's before/after token counts and its list of dropped message ids. Without this span the symptom looks like the model forgetting for no reason. |
Sampling and retention
Full traces on every run get expensive faster than the inference does, because a forty-step run with stored payloads is megabytes. Head sampling is the wrong tool here — you cannot know at step zero whether a run will be interesting. Use tail sampling with an explicit keep list:
- Keep every run whose halt reason is not
finish, and every run containing a tool error. - Keep every run in the top few percent by cost or step count. These are where the money is and they are rare by definition.
- Keep a small uniform sample of successful runs — perhaps one in a hundred — so you have a baseline to compare against. Without it, every trace you have ever looked at is a failure, and you lose all sense of what normal looks like.
- Retain span metadata far longer than payloads. Attributes are small and support trend queries for months; message blobs can expire in days under whatever data policy applies to their contents.