What It Costs to Store and Query High-Volume Event Logs

The storage bill for event logs is usually the number people optimise and rarely the number that hurts. Below is the whole derivation from one stated event rate, with query cost separated out, because on scan-priced platforms one careless dashboard can outspend a year of storage.

The assumptions

• 50,000 events per second, sustained average across the day.
• 900 bytes per event as serialised JSON before compression.
• 8:1 compression once converted to a columnar format with dictionary encoding. Repetitive log fields compress well; measure your own.
• 90-day retention in the queryable store.
• $0.023 per GB-month object storage, and $5.00 per TB scanned for query, both as stand-in list prices. 1 TB is taken as 1,000 GB throughout.

Storage

raw ingest rate
  50,000 ev/s x 900 B      = 45,000,000 B/s = 45 MB/s
  45 MB/s x 86,400 s       = 3,888,000 MB/day = 3,888 GB/day = 3.888 TB/day

after 8:1 compression
  3,888 GB/day / 8         = 486 GB/day

steady-state stored at 90-day retention
  486 GB/day x 90          = 43,740 GB = 43.74 TB

storage cost
  43,740 GB x $0.023       = $1,006.02 / month
  annualised               = $12,072

A thousand dollars a month to keep ninety days of a 50,000 events-per-second stream. That number surprises people in the downward direction, and it is why “we should store less” is so often the wrong first instinct — at object-storage prices, retention is cheap. The costs that are not cheap are the ones that scale with reading rather than with keeping.

Two additions the simple figure omits. Object storage bills per-request as well as per-byte, so writing 486 GB/day as 50 million tiny objects costs more in PUT requests than in storage; batch into files of at least tens of megabytes. And a managed log product typically prices ingest per GB at rates one to three orders of magnitude above raw object storage, in exchange for indexing — if that is your platform, the ingest line, not the storage line, is where 3,888 GB/day is charged, and the arithmetic changes character completely.

Two more multipliers apply to the 43.74 TB before it becomes a real number. If any part of the pipeline keeps a replicated copy — a search index with a replica for availability, or a broker retaining the same events for seven days at replication factor three — that copy is charged too, and the broker retention alone at 3.888 TB/day raw is 27.2 TB before replication and 81.7 TB after. And a search index is not merely the compressed data: inverted indexes typically expand rather than shrink relative to the compressed columnar form, so a hot tier holding the most recent seven days can cost more per day retained than the whole ninety-day cold tier. Price the tiers separately and state how many days sit in each, because a single blended per-GB figure across a hot and cold tier hides the only decision that matters here.

Query, which is usually the larger number

a dashboard query scanning one full day
  486 GB = 0.486 TB
  0.486 TB x $5.00         = $2.43 per query

40 such queries per day (dashboards, alerts, ad-hoc)
  40 x $2.43               = $97.20 / day
                           = $2,916 / month

ratio to storage           = 2,916 / 1,006 = 2.9x

one query scanning the full 90-day retention
  43.74 TB x $5.00         = $218.70 for a single query

an auto-refreshing dashboard, 6 panels, 30-second refresh,
each scanning one day
  6 x 2,880 refreshes/day x $2.43 = $41,990 / day

That last line is not a hypothetical shape of failure; it is the arithmetic of a dashboard left open on a wall screen with a thirty-second refresh, and it is the single most common way a scan-priced log platform produces a bill nobody can explain. The defences are caching results with a TTL at least as long as the refresh interval, pre-aggregating anything a dashboard reads so the panel queries a summary table rather than raw events, and setting a per-query and per-principal scan limit so the platform refuses rather than charges.

What pruning actually saves

Scan pricing charges for bytes read, and two mechanisms reduce bytes read by large factors. Both require the data to be laid out for them, which is a decision made at write time and cannot be retrofitted to existing partitions.

baseline: one day, all services, all columns
  0.486 TB  ->  $2.43

partition pruning: partitioned by (date, service),
query touches 1 service of an assumed 20
  0.486 x (1/20) = 0.0243 TB   ->  $0.1215      (20x)

column projection: columnar format, query reads
3 columns of an assumed 40, assumed to be an
average 7.5% of bytes
  0.0243 x 0.075 = 0.00182 TB  ->  $0.0091      (13.3x)

combined reduction                                267x
cost of the same query                 $2.43 -> $0.0091

the 40 daily queries
  $2,916 / month  ->  $10.93 / month

The column-projection factor is the weakest assumption in that block — columns differ enormously in width, and a query that touches the free text message field reads most of the bytes no matter how few columns it names. Treat 7.5% as illustrative and measure the bytes-scanned figure your platform reports for your own real queries, which is the only honest input.

Partition granularity has its own trade-off. Partitioning by hour instead of day gives 24 times finer pruning for time-ranged queries and creates 24 times more files, and small files are the standard way a well-partitioned table becomes slow: per-file metadata and open costs start to dominate below roughly a hundred megabytes per file. Partition by the dimension queries actually filter on — usually date and service — and compact small files on a schedule.

The levers, ranked

From the arithmetic above, in descending order of effect on the total.

1. Stop scanning raw data for repeated questions. Pre-aggregate into per-minute summary tables and point every dashboard at them. This is a 100× to 1,000× reduction on the dominant line item and it is a schema decision, not a negotiation with a vendor.
2. Partition and project. The 267× combined factor derived above costs nothing except getting the layout right at write time.
3. Reduce volume before it lands. Deduplication of repeated lines and severity-based sampling attack storage, ingest and query simultaneously, since every downstream cost is proportional to bytes stored.
4. Tier by age. Move data past the point where it is queried interactively — often 7 to 14 days — to a colder class. Confirm the retrieval cost and latency of the colder class first: a tier that is cheap to store and expensive to read is a good deal only if the read genuinely never happens, and a compliance retrieval will eventually happen.
5. Shorten retention. Listed last deliberately. It is the lever people reach for first and, at the storage prices assumed here, it saves about $11 per TB-month while destroying the ability to answer questions about last quarter. Reach for it after the other four, and drive the decision from the cost derived above rather than from the intuition that ninety days sounds like a lot.

Finally, forecast this forward rather than treating it as a fixed monthly figure. All five lines scale with volume, volume compounds, and the arithmetic for that is in forecasting log volume for capacity planning.