The Quarterly Capacity Review

A capacity review is a ninety-minute meeting, once a quarter, that exists to catch the constraints you will hit in the next two quarters while there is still time to do something about them. It works when it is eight numbers with a decision attached to each, and fails when it is a dashboard tour.

What the meeting is for

Three constraints in an AI system have lead times measured in weeks or months: accelerator capacity, provider quota increases, and budget approval. All three are invisible in daily operations and all three arrive as a surprise if nobody looks two quarters ahead. That is the meeting’s job — not to review last quarter, but to identify which ceiling gets hit first and when.

It is therefore a forward-looking meeting with a backward-looking input. The rule that keeps it useful: every number must have a decision attached, and a number with no decision comes off the list.

The eight numbers

1. Peak concurrency against provisioned concurrency

The p99 of simultaneous in-flight generations over the quarter, against what the pool can serve. Not the mean — the peak is what fails. Decision: above 80% of provisioned, start the procurement or quota conversation now, because both have lead times longer than the gap between reviews.

2. Queue wait, p95, per tier

From the queueing metrics in priority queueing. Report the trend across the quarter, not a single figure. Decision: rising while request rate is flat means per-request cost is growing — longer prompts, longer outputs, a slower model. That is a product change to investigate, not a capacity purchase.

3. Tokens per second per replica

The productivity of a unit of hardware. Decision: a decline of more than ten per cent quarter on quarter has three candidate causes — degraded devices, throttling, or a shift in the input distribution — and GPU monitoring has the counters that distinguish them. Any of the three is worth a ticket.

4. Cost per thousand successful requests

Total spend divided by successful requests, in thousands, trended over four quarters. The unit number, not the total. Decision: rising means efficiency work — caching, routing to a cheaper model, prompt shortening. Falling while total spend rises is healthy growth and needs no action beyond a note. Cost allocation produces this number.

5. Headroom to the nearest hard limit

Not one number but a short list: provider rate limits, provider token limits, GPU quota, context window against your p99 prompt length, database connections, index size against the memory of the machine holding it. Report current usage as a percentage of each. Decision: anything above 70% gets a runway calculation in the next section; anything above 90% gets an action this week.

6. Error budget consumed

Against the availability and latency objectives, per service. Decision: budget exhausted means a freeze on risky changes and a reliability sprint. Budget untouched all quarter means the objective is too loose to be informative, or you are over-provisioned and could take more risk. SLOs for AI services covers setting them.

7. Capacity mix: reserved, on-demand, spot

The share of GPU-hours from each, plus the realised spot interruption rate and the effective discount from the arithmetic in spot and preemptible GPUs. Decision: a stable floor of demand served by on-demand capacity is money left on the table — move it to reserved. A spot share whose realised discount has fallen because interruptions rose is a signal to shift work back.

8. Waste

Four lines, each an absolute cost: failed and retried calls, requests whose client had disconnected before the answer was produced, idle reserved capacity, and unallocated spend. Decision: waste above five per cent of total spend becomes an engineering item with a named owner. This is consistently the fastest saving available and consistently the least discussed.

Runway: turning growth into a date

A percentage of a limit is not actionable. The number people act on is a date, and getting one is a two-line calculation.

For a limit L, current usage U, and growth rate g per period:

  periods_remaining = ln(L / U) / ln(1 + g)

Worked, all assumptions labelled:

  provider rate limit ................ L = 10,000 requests/minute
  current peak ....................... U =  6,200 requests/minute
  observed growth .................... g = 0.15 per month (15%)

  ln(10,000 / 6,200) = ln(1.613) = 0.478
  ln(1.15)                        = 0.140
  periods = 0.478 / 0.140 = 3.4 months

So the limit is reached in about three and a half months — before the next
quarterly review. That is the finding, and it converts "we are at 62% of
our rate limit" into "we must have the increase approved within ten weeks".

Doubling time, as a sanity check on any growth rate:
  t_double = ln(2) / ln(1 + g) = 0.693 / 0.140 = 5.0 months

Two cautions on this arithmetic. It assumes exponential growth, which
overstates the urgency for a maturing product and understates it for one
that is accelerating — compute it on the last two quarters as well as the
last four and report both. And it uses peak, not mean: a limit is breached
at the peak, and the gap between peak and mean is often a factor of three.

Run that for every limit above 70%, and sort the results by date. The top of that sorted list is the agenda item that matters; everything else can wait a quarter.

The agenda

1. Last quarter’s actions (10 min). Done, not done, and why. Without this the meeting stops producing anything, because nothing produced has consequences.
2. The eight numbers (25 min). Circulated 48 hours in advance and not presented in the meeting — read in advance, discussed in the room. Time here is for the ones that moved.
3. Runway (15 min). Every limit above 70%, with its date, sorted. This is the core of the meeting.
4. Demand forecast (15 min). From the product side, not from the metrics: launches, customer onboardings, a new feature that changes cost per request. The metrics extrapolate the past; only product knows about the step change coming in week six.
5. Decisions (20 min). Each with an owner and a date.
6. Risks accepted (5 min). Explicitly, and recorded. “We are choosing not to raise the quota this quarter and accept the risk of throttling in December” is a legitimate decision, and writing it down is what separates a decision from an oversight.

Attendance: whoever operates the system, whoever owns the budget, whoever knows the product roadmap. Missing the third is the usual failure, and it turns a forecast into an extrapolation.

What comes out of it

One page, circulated, containing exactly four things: the eight numbers with their trend arrows; the sorted runway list with dates; the decisions with owners and dates; and the risks explicitly accepted. Nothing else, because a longer document is not read and an unread document produces no decisions.

The test of whether the meeting is working is simple and worth applying after a year: were you ever surprised by a capacity constraint? A surprise means either a number is missing from the eight, or the runway calculation was not run on the limit that bit. Both are fixable, and fixing them is what makes the next four reviews worth attending.