What Training a GNN on a Real-World Graph Actually Costs
11 min read · updated August 11, 2026
There is no public price list for training a graph neural network, and any figure quoted without its assumptions is worthless. So here is the derivation instead, from inputs you can replace with your own. The conclusion that survives changing any of them is that the GPU is not what you are paying for.
Every input, named
The graph: 100,000,000 nodes and 1,600,000,000 edges with 128-dimensional float32 features. Those are deliberately close to the Open Graph Benchmark’s ogbn-papers100M, which holds 111,059,956 nodes and 1,615,685,872 edges at 128 dimensions, so the scale is a real one rather than a round number chosen for convenience.
The model and schedule: three-layer GraphSAGE, hidden width 256, neighbour sampling with fan-out [15, 10, 5], batch of 1,000 seed nodes, 1,000,000 labelled training nodes (1% of the graph), and 30 epochs.
The hardware and price: one 80 GB data-centre GPU attached over PCIe, assumed to sustain 40 TFLOP/s on these layer shapes and 20 GB/s of effective host-to-device transfer, on an instance billed at an assumed $2.00 per GPU-hour.
The arithmetic is almost free
Work out how many nodes need an output at each layer. Only nodes within two hops of a seed need a layer-1 output, only nodes within one hop need a layer-2 output, and only the seeds need a layer-3 output. A dense transform from d_in to d_out is 2 × d_in × d_out FLOPs per node.
nodes per seed: hop0 = 1, hop1 = 15, hop2 = 150, hop3 = 750
layer 1 (128 -> 256), needed for hops 0..2 = 166 per seed
166,000 nodes x 2 x 128 x 256 = 10.88 GFLOP
layer 2 (256 -> 256), needed for hops 0..1 = 16 per seed
16,000 nodes x 2 x 256 x 256 = 2.10 GFLOP
layer 3 (256 -> 256), needed for seeds = 1 per seed
1,000 nodes x 2 x 256 x 256 = 0.13 GFLOP
-----------
forward per step = 13.11 GFLOP
backward ~ 2x forward = 26.22 GFLOP
total per step = 39.33 GFLOP
steps per epoch = 1,000,000 / 1,000 = 1,000
FLOPs per epoch = 1,000 x 39.33 GF = 39.3 TFLOP
at 40 TFLOP/s sustained = ~1.0 second per epochOne second of arithmetic per epoch, thirty seconds for the run. If compute were the only cost, training this model would be free. It is worth stating why the number is so small: the layers are narrow (256 units) and the per-node transform is a tiny matrix multiply, so a GNN’s FLOP count is orders of magnitude below a transformer’s at comparable data volume. The aggregation itself adds a sparse scatter-add over the sampled edges, which is bandwidth work rather than arithmetic and is counted below.
The data movement is not
nodes touched per seed = 1 + 15 + 150 + 750 = 916 slots per batch = 1,000 x 916 = 916,000 feature bytes per step = 916,000 x 128 x 4 = 469 MB bytes per epoch = 1,000 steps x 469 MB = 469 GB at 20 GB/s effective host-to-device = 23.4 s per epoch
23.4 seconds against 1.0 second. Data movement exceeds arithmetic by a factor of about 23, from the same inputs, and that ratio is the durable finding here — it stays above 10× under any plausible substitution of the assumptions. De-duplication within a batch reduces the transfer, sometimes by half or more on a graph with hubs, and it does not change the ordering.
The 23.4 s is a lower bound that assumes the sampler keeps the link saturated. In practice the CPU-side work — expanding 916,000 random neighbourhood lookups into the adjacency, deduplicating, relabelling to local indices — is the actual bottleneck unless several worker processes are prefetching. Assume 60 seconds per epoch end to end, a little over 2.5× the transfer bound, which is a stated assumption rather than a measurement.
The bill for one run, and for the sweep
30 epochs x 60 s = 1,800 s = 0.50 GPU-hours 0.50 h x $2.00/h = $1.00 per training run hyperparameter sweep, 40 configurations: 40 x 0.50 h = 20 GPU-hours = $40.00 weekly retraining for a year: 52 x 0.50 h = 26 GPU-hours = $52.00
A dollar. That is the number people find surprising and it is the point of doing the arithmetic: at this graph size a single GNN training run is cheap enough that nobody should be optimising it. The sweep is forty times the run, and it is still forty dollars.
What changes the answer is depth and fan-out, because the working set is multiplicative. Hold everything else and vary only the fan-out:
fan-out slots/seed bytes/epoch epochs at 60s -> run cost [15, 10] 166 85 GB ~11 s/epoch $0.18 [15, 10, 5] 916 469 GB 60 s/epoch $1.00 [15, 10, 10] 1,666 853 GB ~109 s/epoch $1.82 [25, 10, 10] 2,776 1,421 GB ~182 s/epoch $3.03
Costs here scale with the transfer, holding the per-epoch time proportional to bytes moved. A fourth layer at fan-out 5 would multiply the top row’s slot count by roughly five again — and given that a four-layer model is likely already oversmoothing, you would be paying five times as much for a worse model. That is the most common way to waste money on this workload.
The costs that are not GPU-hours
- Host memory, which you rent by the instance. The feature matrix is 100,000,000 × 128 × 4 = 51.2 GB and the int64 CSR adjacency is 1,600,000,000 × 8 = 12.8 GB plus 0.8 GB of offsets, so roughly 65 GB must be resident. You are renting a machine with 128 GB and many CPU cores for the sampler, not a bare GPU, and the instance price reflects the whole box.
- Preprocessing, which is one-off and often larger than the training. Building the graph, materialising features and partitioning are CPU-hours that happen before any epoch runs. On a graph this size that is commonly hours, and it recurs every time the schema changes.
- Storage. 65 GB of graph plus checkpoints plus the raw tables it was derived from, held for as long as the model is in service, and duplicated per version if you want reproducibility.
- Full-graph inference, which is the recurring cost. Scoring all 100,000,000 nodes means one pass per layer over every edge. At 256-dimensional float32 embeddings, one layer’s aggregation reads 1,600,000,000 × 256 × 4 = 1.64 TB of embedding values. Three layers is roughly 4.9 TB of reads — more traffic than ten training epochs. If you score the graph nightly, inference, not training, is your bill.
- Engineering time, which dominates everything above. A training run that costs a dollar sits under a data pipeline, a feature store and an evaluation harness that cost person-weeks. Any cost model that reports only GPU-hours is reporting the smallest term.