GPU Cloud Pricing: Building the Cost Model Instead of Reading the Sticker

Nearly every comparison of GPU cloud pricing is a table of advertised hourly rates, which is the one number that is both easiest to find and least likely to determine what you pay. The rate is a term in a formula, and it is usually not the dominant one.

No prices on this page, and why

Rates move, offers are regional, discounts are negotiated, and any figure printed here would be quietly wrong long before the page is revised. Worse, a stale table invites the exact error this page exists to prevent — picking the lowest number in a column. So the deliverable is the model. You supply this week’s quotes; the formula tells you which of them is actually cheaper for the way you will use it.

The effective-cost formula

effective_cost_per_compute_hour =

    ( rate * H_billed
    + storage_per_month * (H_billed / 730)
    + egress_per_GB * GB_out
    + fixed charges: IP, licences, support, orchestration )
    ---------------------------------------------------------
                        H_compute

H_billed  = hours the instance is billed for
H_compute = hours it is actually serving requests
U = H_compute / H_billed   is the utilisation

Written that way, one thing is immediately visible. Every numerator term is divided by H_compute, so a low utilisation multiplies all of them. That is the structure of the whole problem.

The formula also makes clear why the unit matters. Providers quote per hour, per second, per device and per node, and comparing those directly is a category error until they are converted to the same basis. Convert everything to cost per compute hour for one device of the configuration you will actually run, and only then compare. If a provider bills a whole multi-device node and you need part of one, the conversion has to include the devices you are paying for and not using — they belong in the numerator, because you are buying them.

Utilisation is the term that dominates

Ignore the extras for a moment and take the simplest case: effective cost is rate / U. Now compare two providers where provider B charges 40% more per hour but its instances start in seconds while provider A’s take minutes, or where B has capacity in the region you need and A leaves you holding a reserved instance you cannot fill.

The general result: a rate difference of tens of percent is routinely swamped by a utilisation difference of a factor of two or more. Comparing advertised rates without dividing by your own utilisation answers a question nobody asked.

The features that raise utilisation are therefore worth paying a premium for, and they are rarely the ones highlighted on a pricing page: per-second rather than per-hour billing, fast instance start, the ability to scale to zero between bursts, and available capacity in the region you need so that you are not holding an instance as a hedge. Each of those raises U directly, and raising U from 0.3 to 0.6 is worth more than any discount you are likely to negotiate on the rate itself.

The terms that are not on the price page

• Storage that outlives the instance. Model weights are tens to hundreds of gigabytes. Persistent volumes and object storage are billed whether or not anything is running, and for a spiky workload they can be a meaningful fraction of the total.
• Egress. Data leaving the provider is usually billed per gigabyte and is often asymmetric with ingress. For text inference the token traffic is small; for image, audio and video work, or for shipping logs and artefacts out, it is not.
• Cold-start time you pay for. Loading a 140 GB checkpoint from storage into device memory takes real minutes and those minutes are billed as compute while serving nothing. The amortised penalty is that time divided by the length of the session, which is why short sessions on large models are so expensive.
• Interruption on cheaper tiers. Preemptible and spot capacity is priced below on-demand precisely because it can be taken away. The correct accounting is not the discount but the discount minus the cost of the work lost and re-run, which is why spot suits checkpointed batch work and suits a latency-sensitive endpoint poorly.
• Minimum commitments and node granularity. If the smallest unit is an eight-device node and you need two devices, your effective rate is four times the per-device rate — a factor no price table shows.

Comparing two quotes honestly

• Estimate H_compute from your traffic before you look at any rate. Requests per day times average seconds of compute per request, divided by 3,600.
• Choose the billing model that matches the shape. Steady traffic wants a committed rate; spiky traffic wants per-second billing and fast starts even at a higher headline rate.
• Convert everything to cost per million tokens. That is the unit in which a rented instance and a metered API are comparable, and the conversion is the effective hourly cost divided by the tokens per hour your throughput arithmetic implies.
• Re-run it after a month of real traffic. The initial utilisation estimate is the least reliable input and the one the answer is most sensitive to.

One thing to check before any of that, because it invalidates the comparison rather than skewing it: confirm that the two quotes are for the same thing. Device generation, memory capacity per device, the number of devices per instance, the interconnect between them, the host CPU and RAM, the local storage, and the network all vary between offerings that carry similar names. An instance with the same accelerator but a slower device-to-device link is a different product for any deployment that splits a model, and an instance with too little host RAM to stage a checkpoint is a different product for everyone. Normalise the configuration first; the rate arithmetic only means something afterwards.