How Much Electricity Training One Model Uses: Disclosures and Gaps

Training energy is calculable from accelerator-hours, and for the handful of models whose creators disclosed both energy and emissions, the calculation reproduces their figures. For the frontier models everyone actually asks about, nothing has been disclosed at all.

The formula

  energy_kWh  =  gpu_hours  *  power_per_gpu_W  *  PUE  /  1000

  gpu_hours     accelerator-hours for the run, from the model card
  power_per_gpu average draw in watts; TDP is an upper bound and real
                average draw on a sustained training job is typically
                70-90% of it
  PUE           power usage effectiveness of the facility; modern
                purpose-built data centres report roughly 1.1, older
                mixed estate is worse

Then, for emissions:

  emissions_kg  =  energy_kWh  *  grid_intensity_gCO2e_per_kWh  / 1000

Two inputs are usually published and two are usually not. GPU-hours appear on many open-weights model cards, and accelerator TDP is a datasheet figure. PUE and grid intensity are properties of the facility and the location, which is why the same run in two places has different emissions and identical energy.

Checking it against a disclosure

Meta’s Llama 2 paper reports both accelerator-hours and estimated emissions for its training runs — in the region of 3.3 million A100-80GB hours in total across the model sizes, and around 539 tonnes of CO₂ equivalent. That gives us both ends of the chain, so the derivation can be checked by back-solving the middle.

Forward, to energy. NVIDIA's specification puts the A100 SXM at 400 W:

  3.3e6 hours  *  400 W  =  1.32e9 Wh  =  1,320 MWh   (IT energy)
  with PUE 1.1:                          1,452 MWh   (facility energy)

Back-solve the grid intensity implied by the disclosed emissions:

  539 tonnes  =  539,000 kg  =  5.39e8 g
  5.39e8 g  /  1.452e6 kWh  =  371 gCO2e per kWh

371 g/kWh is an entirely ordinary grid carbon intensity for a mixed
grid. The derivation and the disclosure are consistent, which is the
check we wanted: if the back-solved intensity had come out at 40 or at
2,000 g/kWh, one of the inputs would be wrong.

The other route: from FLOPs

Where GPU-hours are not published but parameters and tokens are, you can go through compute instead, using the estimate from the training-compute page. This route needs one more assumption, and it is worth doing because it also reproduces a published figure.

  energy_J  =  total_FLOPs  /  (peak_FLOPs_per_W  *  MFU)

Worked on GPT-3, for which the paper reports about 3.14e23 FLOPs of
training compute, trained on V100-class hardware.

NVIDIA's specification for the V100 gives roughly 125 TFLOP/s of tensor
throughput at a 300 W board power:

  peak efficiency  =  125e12 / 300  =  4.17e11 FLOP per joule

At an assumed 25% model FLOPs utilisation:

  effective        =  4.17e11 * 0.25  =  1.04e11 FLOP per joule
  energy           =  3.14e23 / 1.04e11  =  3.02e12 J
                   =  3.02e12 / 3.6e6     =  839,000 kWh
                   =  839 MWh (IT), or 923 MWh at PUE 1.1

Patterson and colleagues (Google, 2021), in "Carbon Emissions and Large
Neural Network Training", estimate GPT-3's training at roughly 1,287 MWh
and about 552 tonnes CO2e.

Back-solving our MFU assumption against their figure:

  0.25 * (923 / 1287)  =  0.179

An implied 18% model FLOPs utilisation, which is plausible for a run of
that era and scale. The two independent routes agree to within about
40%, which for an estimate spanning this many assumptions is close
agreement rather than loose.

What has actually been disclosed

The list of models whose training energy or emissions has been published by their creators or by peer-reviewed analysis is short. These are the ones worth citing by name.

• Strubell, Ganesh and McCallum (2019), Energy and Policy Considerations for Deep Learning in NLP. The paper that started the conversation. Read it before quoting anything from it, because what it measured was largely a neural architecture search — hundreds of training runs — rather than a single model, and the difference is the subject of the most widely repeated misquotation in this field.
• Patterson et al. (2021), Carbon Emissions and Large Neural Network Training. A Google-authored analysis giving energy and emissions estimates for several named models including GPT-3, with the method fully shown. The single most useful reference in this area.
• Luccioni, Viguier and Ligozat (2022), Estimating the Carbon Footprint of BLOOM. A full life-cycle accounting for one 176-billion-parameter model, including the manufacturing of the hardware and the energy of the idle infrastructure, which almost no other study attempts. Its dynamic training energy figure is in the region of 433 MWh.
• The Llama model cards. Meta has disclosed cumulative GPU-hours and estimated tonnes of CO₂ equivalent for each release, including a market-based figure of zero where emissions were matched by renewable purchasing, alongside the location-based figure. Publishing both is the correct practice and is still unusual.
• Operator sustainability reports. Google, Microsoft and Amazon publish annual environmental reports with total electricity consumption and fleet PUE. These are audited and are about facilities rather than models, which limits what you can conclude from them and is also why they are trustworthy.

What nobody has published

This is the more important half of the page, and it is short enough to state precisely.

• No frontier closed model has a published training energy figure. Not GPT-4 or its successors, not the Claude models, not Gemini. No energy, no emissions, and in most cases no parameter count or token count either, which removes even the derivation route.
• Nobody publishes the failed runs. Every disclosed figure is for the final run. Experiments, ablations and abandoned attempts consume real electricity and appear nowhere.
• Embodied energy is nearly always excluded. Manufacturing the accelerators and building the facility are real and are outside almost every published boundary. The BLOOM study is the notable exception.
• Post-training is invisible. Reinforcement learning from human feedback, preference tuning and the large-scale generation of synthetic training data all consume compute and are not in any headline training figure.
• The trend is towards less disclosure, not more. The early large models were published with full technical reports; the commercial frontier models are described in system cards that omit architecture, data and energy alike.

One thing may change this. The EU AI Act’s technical documentation requirements for general-purpose AI models include, as drafted, the known or estimated energy consumption of the model. If enforced as written, that would be the first regulatory requirement anywhere to disclose the figure, and it would apply to models placed on the EU market regardless of where they were trained. Read the annex text rather than a summary before relying on it — see the EU AI Act page.

How these figures get misused

• Quoting a research-programme total as one model. The most common error, and it originates with the 2019 paper above. A figure covering hundreds of runs is not a figure for one model.
• Confusing energy with emissions. The same run has one energy figure and many possible emissions figures depending on the grid. Reporting one as the other is the most frequent unit error in this area.
• Ignoring amortisation. Training happens once and is spread across every inference the model ever serves. For a widely used model the per-query share of training is small — and the divisor is unknown outside the provider, so anyone who states the amortised figure is guessing at the denominator. See the inference carbon derivation.
• Comparing across years without adjusting. Hardware energy efficiency per FLOP has improved by large factors between generations, so an energy figure from a 2019 run says nothing about the same computation done today.
• Using market-based figures to claim zero. Renewable energy certificates make a market-based emissions figure lower, and sometimes zero, while the physical electricity drawn from the grid is unchanged. Both numbers are legitimate under the GHG Protocol and reporting only the flattering one is not.