Building an LLM Cost Dashboard

Cost dashboards usually fail in one of two directions: a single total that nobody can act on, or forty panels that nobody reads. Five charts, each answering a question somebody actually asks out loud, is about the right size — and each of them is a query you can run today.

Three audiences ask genuinely different questions of the same data, and a dashboard that ignores the split ends up serving none of them. Finance asks what this month will be and why it differs from last month. Engineering asks what a particular change did. Product asks whether a feature can be afforded at ten times the current user count. The five charts below cover all three, in roughly that order — which is also why the top of the dashboard is a trend line and not a breakdown: the first question anyone has is whether the number is moving, and only then which part of it moved.

Everything runs against the llm_request table from the logging page and the daily rollup from per-customer tracking. One rule for all of them: where environment = 'prod', always, because eval and staging spend contaminates every trend it touches.

1 · Spend and run rate

Daily spend, with a month-to-date total and a straight-line projection to month end. The projection is the panel finance looks at; the daily series is what makes a step change obvious.

with daily as (
  select started_at::date as day, sum(cost_usd) as spend
  from llm_request
  where environment = 'prod'
    and started_at >= date_trunc('month', now()) - interval '2 months'
  group by 1
),
mtd as (
  select sum(spend) as spend_mtd,
         count(*)   as days_elapsed
  from daily where day >= date_trunc('month', now())::date
)
select d.day,
       d.spend,
       avg(d.spend) over (order by d.day rows between 6 preceding and current row)
         as spend_7d_avg,
       (select round(spend_mtd, 2) from mtd) as mtd,
       (select round(spend_mtd / nullif(days_elapsed, 0)
              * extract(day from date_trunc('month', now())
                        + interval '1 month - 1 day'), 2) from mtd) as projected_month
from daily d
order by d.day;

The seven-day average is there because model spend has a strong weekday shape for most B2B products, and a raw daily line makes every Monday look like an incident. Keep both series on the chart; the gap between them is informative.

2 · The breakdown, with an honest “other”

Stacked spend by whichever dimension you are investigating — feature, model, or tenant. The important detail is the explicit top-N with an other bucket, because a stacked chart with sixty series is a colour test rather than a chart, and because an other bucket that is growing is itself a finding.

with ranked as (
  select feature, sum(cost_usd) as spend,
         row_number() over (order by sum(cost_usd) desc) as rn
  from llm_request
  where environment = 'prod' and started_at >= now() - interval '30 days'
  group by 1
)
select started_at::date                                   as day,
       coalesce(r.feature, 'other')                       as bucket,
       round(sum(l.cost_usd), 2)                          as spend
from llm_request l
left join ranked r on r.feature = l.feature and r.rn <= 8
where l.environment = 'prod' and l.started_at >= now() - interval '30 days'
group by 1, 2
order by 1, 3 desc;

Build it so the dimension is a parameter rather than three separate panels. The same query with served_model instead of feature answers a different question — whether a migration actually moved traffic — and with tenant_id it answers whether one customer is the entire story. Three views of one query beats three queries that drift apart.

A note on the stacking order: sort series by current spend descending rather than alphabetically, so the band that matters is at the bottom of the stack where a change in its height is readable. In a stacked chart, only the bottom series can be read accurately; everything above it inherits the wobble of everything below.

3 · Unit economics

Cost per unit of value delivered, not cost per request. This is the only chart on the list that can go down while spend goes up, which is exactly the situation you want to be able to demonstrate.

-- 'outcome' is your own table: one row per completed task, ticket
-- resolved, document generated, suggestion accepted. If you do not have
-- one, building it is worth more than any chart on this page.
select date_trunc('week', o.completed_at)                     as wk,
       o.outcome_type,
       count(*)                                               as outcomes,
       round(sum(l.cost_usd), 2)                              as spend,
       round(sum(l.cost_usd) / nullif(count(*), 0), 4)        as cost_per_outcome,
       round(avg(l.calls_per_outcome), 2)                     as avg_calls
from outcome o
join lateral (
  select sum(cost_usd) as cost_usd, count(*) as calls_per_outcome
  from llm_request
  where trace_id = o.trace_id and environment = 'prod'
) l on true
where o.completed_at >= now() - interval '12 weeks'
group by 1, 2
order by 1, 2;

The avg_calls column is the one that explains movement. Cost per outcome rises for two reasons — the model got dearer, or your agent started taking more turns to finish — and only the second is something you control.

If you do not have an outcome table, this chart is the reason to build one. It does not need to be sophisticated: a row per completed task with a type, a timestamp and the trace id that produced it. The trace id is the load-bearing column, because it is what lets you attribute every model call in a multi-step flow to the one outcome it was working toward — which a per-request join cannot do, since a single outcome may involve a retrieval call, four agent turns and a summarisation.

4 · Waste

Spend that produced nothing a user received. Four categories, and nearly every product has more of it than expected.

select date_trunc('week', started_at) as wk,
       round(sum(cost_usd) filter (where error_type is not null), 2) as failed,
       round(sum(cost_usd) filter (where attempt > 1), 2)            as retried,
       round(sum(cost_usd) filter (where finish_reason = 'length'), 2) as truncated,
       round(sum(cost_usd) filter (where cancelled), 2)              as abandoned,
       round(sum(cost_usd), 2)                                       as total,
       round(100 * sum(cost_usd) filter (
               where error_type is not null or attempt > 1
                  or finish_reason = 'length' or cancelled
             ) / nullif(sum(cost_usd), 0), 1)                        as waste_pct
from llm_request
where environment = 'prod' and started_at >= now() - interval '12 weeks'
group by 1 order by 1;

abandoned — the user closed the tab while a long answer was still streaming — needs a cancellation flag written on the request row and is the category most often missing entirely. For any feature that streams long outputs it can be a meaningful share of spend, and unlike the other three it is fixed by propagating cancellation rather than by changing anything about the model.

5 · Concentration

The distribution of cost per request, not its average. Spend is usually concentrated in a small tail of very large requests, and an average request cost hides that completely.

with r as (
  select cost_usd,
         ntile(100) over (order by cost_usd) as pct
  from llm_request
  where environment = 'prod' and started_at >= now() - interval '7 days'
)
select round(avg(cost_usd) filter (where pct <= 50), 6)  as p50_cost,
       round(avg(cost_usd) filter (where pct = 95), 6)   as p95_cost,
       round(avg(cost_usd) filter (where pct = 99), 6)   as p99_cost,
       round(100 * sum(cost_usd) filter (where pct = 100)
             / nullif(sum(cost_usd), 0), 1)              as top1pct_share,
       round(100 * sum(cost_usd) filter (where pct > 90)
             / nullif(sum(cost_usd), 0), 1)              as top10pct_share
from r;

If the top 1% of requests carries a large share of spend, your optimisation target is a context-length or fan-out problem in a handful of code paths, not a model choice. If spend is evenly spread, it is a model-price or volume conversation. The two lead to entirely different work, and this is the chart that tells you which one you are in.

What these five cannot tell you

• Whether the spend was worth it. No cost chart contains value. Chart 3 is the closest, and only if your outcome table is honest about what counts as an outcome.
• Whether your numbers are right. Every figure here depends on your pricing table. Reconcile the monthly total against the provider invoice and put that comparison on the dashboard too — a persistent gap means a stale rate, a missing code path, or a key being used outside your client.
• Where a spike came from, on its own. Charts 1 and 2 show that something changed; the split by release from cost attribution is what names it.
• Anything about a dimension you did not capture. Which is the real reason to read the attribution page before the dashboard page.