Cost Allocation Across Teams and Environments

Cost allocation fails for one reason: a large share of the bill has no obvious owner. Shared inference capacity, idle GPU hours, a vector database used by four teams. Tagging solves the easy part and leaves that part untouched, so the report gets disputed, and a disputed report changes nobody’s behaviour. This page is mostly about the hard part.

Two cost streams, two mechanisms

An AI system generates costs through two channels that are billed differently and must be attributed differently.

Most teams do one and not the other, which yields a report that is either “infrastructure by team, model spend as one lump” or the reverse. Both leave the biggest number unexplained. Do both, and join them on a common dimension — team, service, environment — so one query answers “what did this product line cost?”

Tags that survive contact with reality

A tagging scheme fails through omission, not through design. The fix is to make untagged resources impossible to create rather than to send reminders.

# The required set. Small, mandatory, and enforced.
cost-center   = "cc-4471"          # who pays; matches finance's chart of accounts
team          = "search-platform"  # who to ask about it
service       = "retrieval-api"    # what it is part of
environment   = "prod"             # prod | staging | dev
managed-by    = "terraform"        # so unmanaged resources are visible
data-class    = "customer"         # customer | internal | public — drives policy too

Three enforcement points, in increasing order of effectiveness. A variable validation block in the Terraform module rejects a missing or malformed tag at plan time. A policy check in CI fails the pipeline on a plan that would create an untagged resource. And an organisation-level policy that refuses creation catches everything created outside your pipeline, which is where the untagged resources actually come from. Infrastructure as code for AI stacks has the module structure that carries the tags.

Two practical notes. Tag keys and values are usually case-sensitive, so Team and team become different dimensions and split your report in half; pin the casing in the module. And some resources cannot be tagged, or their charges appear on a line that carries no tags at all — data transfer between zones is the usual example. Those land in the unallocated bucket by construction, which is the last section.

Attribution on the request path

Model spend is attributed by what the request carries. Decide the dimensions once, put them in a header or a metadata field, and propagate them through every hop.

# On the way in, from your own services:
X-Cost-Team:        search-platform
X-Cost-Service:     retrieval-api
X-Cost-Environment: prod
X-Cost-Feature:     answer-synthesis     # finer than service; optional
X-Cost-Tenant:      cust_8f21           # for per-customer unit economics

# The log line that makes the report possible. One per model call.
{
  "ts": "2026-08-04T09:14:22Z",
  "request_id": "req_01J...",
  "team": "search-platform",
  "service": "retrieval-api",
  "environment": "prod",
  "feature": "answer-synthesis",
  "tenant": "cust_8f21",
  "model": "vendor/model-name",
  "input_tokens": 4180,
  "cached_input_tokens": 3900,
  "output_tokens": 260,
  "unit_price_input_per_mtok": 0.15,
  "unit_price_cached_per_mtok": 0.0375,
  "unit_price_output_per_mtok": 0.60,
  "cost_usd_micros": 198,
  "latency_ms": 2410,
  "status": "ok"
}

Four things about that record are deliberate. It stores the unit prices used at the time, so a later price change does not silently rewrite last quarter’s report. It counts cached input tokens separately, because they are usually priced differently and a report that ignores them overstates cost for exactly the workloads you optimised. It stores cost in integer micros, never a float, for the usual reason — money in integers. And it records failures too: retries and failed calls are billed and must be attributed, or your cost per successful request is wrong in the direction that flatters you.

Propagation is where this breaks. A background job, a retry from a queue consumer, or an agent step three levels deep loses the headers unless somebody carried them. Put the cost dimensions in the same context object that carries your trace identifier so that they travel together — cost attribution and per-customer LLM cost cover the application-level design.

The hard part: shared and idle capacity

Here is the situation that ruins reports. A GPU pool costs a fixed amount per month. Three teams use it. It is 55% utilised. Who pays for the other 45%?

There are two defensible answers and one indefensible one.

Setup, all figures illustrative:
  pool cost this month .................. 30,000 (currency units)
  GPU-hours available ................... 2,160  (3 nodes × 720 h)
  GPU-hours consumed .................... 1,188  (55% utilised)
  by team:  search 594 (50%),  support 356 (30%),  labs 238 (20%)

METHOD A — proportional to usage.
  Every consumed hour carries a share of the whole bill, idle included.
    effective rate = 30,000 / 1,188 = 25.25 per consumed GPU-hour
    search  594 × 25.25 = 15,000
    support 356 × 25.25 =  8,990
    labs    238 × 25.25 =  6,010
  Property: nothing is unallocated. Cost: a team's bill changes when
  OTHER teams change their usage, which teams find intolerable and which
  makes month-on-month comparison meaningless.

METHOD B — usage at a fixed rate, remainder to the platform.
  Publish a rate in advance from planned utilisation, e.g. 70%:
    published rate = 30,000 / (2,160 × 0.70) = 19.84 per GPU-hour
    search  594 × 19.84 = 11,785
    support 356 × 19.84 =  7,063
    labs    238 × 19.84 =  4,722
    charged to teams .... 23,570
    remainder ...........  6,430  → platform team's budget line
  Property: a team's bill depends only on its own usage. The 6,430 is a
  visible, owned number that the platform team is accountable for reducing.
  Cost: someone must own that line and defend it.

Method B is almost always the right choice. The unallocated remainder is
not a failure of the report — it is the report doing its job, by putting a
price on idle capacity and giving it an owner who can act on it.

THE INDEFENSIBLE ONE — splitting the remainder equally between teams.
It charges a team that used 5% of the pool the same idle cost as one that
used 60%, so it rewards heavy users and penalises light ones. It is chosen
because it is easy to explain, and it is disputed for the rest of its life.

The same choice arises for anything shared: a vector database, an embedding service, a monitoring stack. Use Method B wherever a reasonable usage measure exists, and where it does not — a shared logging pipeline, say — allocate by a stated proxy such as ingested bytes and say in the report that it is a proxy.

Reserved and committed capacity needs one more decision, made in advance: does a team that used reserved capacity get the discounted rate, or the on-demand rate with the saving accruing to the platform? Both are defensible. Charging the blended effective rate is the usual answer, because it keeps the incentive to plan capacity with the team that can do it.

The report people accept

Acceptance comes from three properties: a team can reproduce its own number, the number changes only when that team’s behaviour changes, and every line is actionable. One page per team, monthly.

1. The total, with the change against last month in both absolute and percentage terms.
2. Split by stream: infrastructure versus model spend. These have different levers and mixing them hides which one moved.
3. Unit cost, not just total. Cost per thousand successful requests, or per active customer. A total that rose 40% while unit cost fell 10% is a growth story; the same total with unit cost up 25% is a regression. Only the unit number distinguishes them.
4. Top five drivers of the change, each with a quantity: a model change, a prompt that grew, a retry loop, a new feature, a price change. This is the section people read.
5. Waste, named. Failed and retried calls, requests whose client had already disconnected, idle reserved capacity, oversized instances. Every one is money spent on nothing and every one has an owner.
6. The unallocated line, with its absolute value and its share, and who owns reducing it.

Send it monthly and unchanged in format. A report whose shape changes every quarter cannot be compared with itself, and comparison is the only thing that makes it useful. Building an LLM cost dashboard covers the live view that sits behind it, and AI unit economics covers turning unit cost into a pricing decision.

The unallocated bucket

There will always be one. Costs that arrive without tags, resources created before the policy, cross-zone transfer, support charges, a managed service that bills at the account level. Report it as a line rather than distributing it silently, and hold it to a target — under five per cent of the total is a reasonable one, and above ten per cent the whole report loses its authority.

Work it down in the order that pays: find the largest untagged line first, identify the resource, tag it or delete it. In practice a meaningful share of a first unallocated bucket turns out to be resources nobody owns at all, which is the most satisfying kind of cost reduction available.