Forecasting Log Volume for Capacity Planning
9 min read · updated August 11, 2026
Log volume grows with traffic, with headcount, and with every new service and every added debug line — so it compounds, and compounding is the part people get wrong. The arithmetic below is worth more than any published growth statistic, because you can run it on your own numbers.
The four inputs
A capacity forecast needs exactly four numbers, and getting them is usually harder than the arithmetic.
- Current volume, measured at the ingest point. Assumed here: 2.0 TB per day of raw, uncompressed log bytes. Measure this after the shipper and before any sampling, because that is where the pipeline capacity constraint lives.
- Growth rate, per period, over enough periods to be a trend. Assumed here: 6% per month. Derive it from at least twelve months of history, and derive it as a ratio rather than a difference — log volume is multiplicative, so a straight line fitted to it will underpredict.
- Retention. Assumed here: 30 days hot.
- Compression ratio achieved in storage. Assumed here: 10:1, which is realistic for repetitive text in a columnar or block-compressed format and should be measured on your own data rather than assumed.
Deriving the growth rate is where forecasts go wrong before any arithmetic happens. Take monthly totals rather than daily ones, so weekday effects cancel; take the logarithm of each total and fit a straight line to it, because a line fitted to the logarithm is an exponential fitted to the volume, and the slope is directly the growth rate you want. Fitting a line to the raw totals instead gives a constant number of terabytes per month, which will underpredict every future period and does so by more the further out you look. Exclude months containing a major incident or a one-off backfill from the fit and note them separately: a retry storm that tripled volume for two days is a real cost but it is not a trend, and leaving it in tilts the slope for years. If twelve months of history is not available, fit what you have and state the uncertainty as a range — a rate estimated from four points can easily be off by a factor of two, and a forecast presented as a single number invites a decision it cannot support.
What compounding does
monthly growth factor g = 1.06 12 months 1.06^12 = 2.012 -> 2.0 TB/day becomes 4.02 TB/day 24 months 1.06^24 = 4.049 -> 8.10 TB/day 36 months 1.06^36 = 8.147 -> 16.29 TB/day doubling time = ln(2) / ln(1.06) = 0.6931 / 0.05827 = 11.9 months steady-state stored bytes = daily volume x retention / compression today 2.00 TB/day x 30 / 10 = 6.00 TB +12 mo 4.02 TB/day x 30 / 10 = 12.07 TB +24 mo 8.10 TB/day x 30 / 10 = 24.29 TB
Six percent a month sounds modest and doubles the estimate in under a year. That gap between how a monthly rate feels and what it does over a planning horizon is the entire reason to write the exponent down. The same 6% stated annually would be a rounding error; stated monthly it is a factor of eight over three years.
Note also that stored volume is steady-state under a constant retention: you are always holding the last 30 days, so storage tracks the current daily rate rather than accumulating. That makes retention a direct multiplier — moving from 30 to 90 days triples storage immediately and permanently, and is usually a larger single lever than anything you can do to the volume itself.
The date you hit the ceiling
Ingest is a rate, not a volume, and rate is what saturates. Convert.
2.0 TB/day = 2,000,000 MB / 86,400 s = 23.15 MB/s average at an assumed 800 bytes per event: 23.15e6 / 800 = 28,935 events/s peak-to-average ratio, assumed 2.5x -> 57.9 MB/s at peak assume the pipeline is provisioned for 40 MB/s average months until the average reaches 40 MB/s = ln(40 / 23.15) / ln(1.06) = ln(1.7279) / 0.05827 = 0.5470 / 0.05827 = 9.4 months months until PEAK reaches 40 MB/s: already exceeded (57.9 > 40)
The last line is the point of doing this at all. Planning against the average says there are nine months of headroom; planning against the peak says the pipeline is already over its rating during the busy hour and is surviving only on buffering. Whether that is fine depends entirely on how deep the buffer is and how long the peak lasts — multiply the excess rate by the peak duration to get the backlog, and compare it to the broker’s retention. A 17.9 MB/s excess for two hours is 129 GB of backlog that must be absorbed and then drained, and it drains only at whatever spare capacity exists off-peak.
Cost over the planning horizon
Take an assumed storage price of $0.023 per GB-month and 1 TB = 1,000 GB. This is a placeholder for object-storage list pricing, not a quoted rate; substitute your own.
month 0 6.00 TB = 6,000 GB x $0.023 = $138.00 / month month 12 12.07 TB = 12,070 GB x $0.023 = $277.61 / month total over 12 months (geometric series, first term 138, ratio 1.06) = 138 x (1.06^12 - 1) / 0.06 = 138 x (1.0122 / 0.06) = 138 x 16.87 = $2,328 the same 12 months at a flat 6 TB = 138 x 12 = $1,656 growth premium = 2,328 / 1,656 = 1.41x
Budgeting the current monthly cost multiplied by twelve underestimates by 41% at 6% monthly growth. The geometric sum — first term times (gn − 1) ÷ (g − 1) — is the piece to keep, and it applies to any compounding line item, not just storage. Note that this covers storage only; on most platforms query and ingest charges dominate it, which is worked out separately in what it costs to store and query event logs.
Why the forecast will be wrong
Log volume is not smooth, and the discontinuities are larger than the trend. A single new service at debug level, a retry storm during an incident, or one team adding structured request logging can add more in a week than the trend adds in a year. Forecast the trend, but size for the step: the useful question is not “what will volume be in twelve months” but “what happens on the day it doubles without warning”.
Three defences, in order of value. Enforce a per-service quota at ingest with an explicit shed-and-report behaviour, so one team cannot consume the pipeline and the overage is attributed rather than mystery growth. Make severity-based sampling a switch you can throw during an incident rather than a change you deploy — dropping debug and info lines from a chosen service is the fastest available relief and is worthless if it takes a release. And run deduplication before the index rather than after, because the storm that breaks the forecast is almost always repetition rather than variety, and collapsing it bounds the damage at the point where the cost is incurred.