Cooling, Power Density and the Physical Limits of a Rack

A datacentre does no mechanical work and stores no energy, so every watt delivered to a rack leaves it as heat. That conservation statement is the whole of thermal design, and one equation applied to it explains why dense AI hardware stopped being air-cooled.

Every watt in is a watt of heat out

Computation dissipates essentially all the electrical energy it consumes; the negligible remainder leaves as light down a fibre or as radio. So a rack drawing Q kilowatts is a Q-kilowatt heater, and the cooling system’s job is to carry exactly that away, continuously, without letting any component exceed its junction temperature limit.

The equation that governs it is the same one for any fluid:

Q = m_dot * cp * dT

Q     = heat removed, watts
m_dot = mass flow rate, kg/s
cp    = specific heat capacity, J/(kg*K)
dT    = temperature rise of the coolant, K

Rearranged:  m_dot = Q / (cp * dT)

Three properties of the coolant decide everything: its specific heat, its density (which converts mass flow into the volume you must actually move), and how large a temperature rise you can tolerate.

How much air a dense rack needs

Air: cp ≈ 1,005 J/(kg·K), density ≈ 1.2 kg/m³ at ordinary conditions. Take a 15 K rise across the rack, which is a realistic ceiling before the exhaust is too hot to handle.

10 kW rack:
  m_dot = 10,000 / (1005 * 15) = 0.663 kg/s
  volume = 0.663 / 1.2 = 0.553 m^3/s  (~1,170 CFM)

50 kW rack:
  m_dot = 3.32 kg/s
  volume = 2.76 m^3/s  (~5,860 CFM)

100 kW rack:
  volume = 5.53 m^3/s  (~11,700 CFM)

Now consider what moving that much air through a rack-sized opening means. Volume flow divided by the free frontal area of a rack — call it roughly 1 m² once you subtract structure — gives face velocities of well over 5 m/s at the high end. Fan power rises steeply with flow rate, the noise becomes an occupational problem, and the pressure drop across densely packed heatsinks rises faster still. Air does not stop working at a specific number of kilowatts; it becomes progressively absurd, and somewhere in the tens of kilowatts per rack the absurdity wins.

Two of the terms in that calculation are less adjustable than they look. Raising dT reduces the required flow proportionally, but the inlet temperature is bounded below by what the facility can supply without expensive refrigeration, and the outlet is bounded above by what the components tolerate — so the usable range is perhaps 10 to 20 kelvin, not a free parameter. And air’s density falls with altitude, so the same mass flow means a larger volume in a facility built well above sea level, a real derating that catches people comparing sites on rack power alone.

The same equation with water in it

Water: cp ≈ 4,180 J/(kg·K), density 1,000 kg/m³. Same 15 K rise, same equation.

10 kW with water, dT = 15 K:
  m_dot  = 10,000 / (4180 * 15) = 0.160 kg/s
  volume = 0.160 / 1000 = 0.00016 m^3/s = 0.16 litres/second

Compare with air at the same duty: 0.553 m^3/s.

Volumetric ratio = 0.553 / 0.00016  ~=  3,500x

Roughly four times the specific heat and about eight hundred times the density, multiplying to something on the order of three thousand times less volume to move for the same heat. That single ratio is the reason direct-to-chip cold plates and immersion cooling exist, and it is not a preference or a trend — it is what the numbers say.

Liquid buys two further things worth naming. Higher permissible coolant temperatures mean the heat can often be rejected to the outside air without mechanical refrigeration for much of the year, which removes the largest overhead term. And a cooler junction temperature reduces leakage current and avoids thermal throttling, so the same silicon delivers more sustained throughput — which loops directly back into the serving arithmetic, since a throttled device in a tensor-parallel group sets the pace for every device it synchronises with.

PUE, and where the overhead goes

Power usage effectiveness is total facility power divided by the power delivered to the computing equipment. A PUE of 1.5 means half again as much energy is spent on cooling, power conversion and distribution as on the computation itself.

• Chillers and compressors are the largest term when they are required, and the whole point of raising coolant temperatures is to run without them for more of the year.
• Fans move air and consume power doing it, and that power becomes heat inside the room, which must then also be removed.
• Power conversion losses at every step from the grid to the board become heat at that step.
• PUE flatters dense facilities. It is a ratio, so a higher IT load in the denominator improves it even if absolute overhead is unchanged. It is a useful efficiency measure and a poor comparison between facilities of different densities.

Why the constraint is the rack, not the chip

Accelerator power per device has risen with each generation, and the interconnect arithmetic pushes strongly toward putting many of them close together — tensor parallelism wants short, fast links, so it wants devices in the same chassis. Those two pressures multiply into rack power densities far above what conventional halls were designed for.

The result is that the binding constraint on deploying accelerators is frequently not silicon supply at all. It is a facility with enough power delivered to the row, enough cooling capacity at the rack, and a grid connection that exists. Each of those has a lead time measured in quarters or years and none of them is fixed by ordering more chips.

Two practical consequences follow for anyone costing hardware. Quoted rack power limits, not floor space, determine how much you can install in a given hall. And the efficiency figure that matters for capacity planning is tokens per watt rather than tokens per second — when power is the constraint, a more efficient device does not merely save money, it is the only way to add throughput at all.

There is a corollary that reaches back into the software. Every technique that reduces bytes moved per token — quantisation, a smaller KV cache, sparse activation, speculative decoding — reduces energy per token as well, because moving data is where the energy goes. In a power-constrained facility those are not merely cost optimisations; they are capacity, delivered without a procurement cycle, a construction project or a grid connection. That is an unusual position for a software change to be in, and it is the strongest practical argument for doing the inference arithmetic before ordering anything.