The Cost of RAG: A Full Per-Query Breakdown

Every rate below is a placeholder. The point of this page is the structure of the arithmetic and the conclusions that survive whatever numbers you substitute — because the rates change quarterly and the shape of the model does not.

Seven line items

A RAG query touches at most seven billable things. Name them, because most cost estimates omit two or three and then wonder where the bill came from.

Notice that items 1 and 2 are the ones people worry about and items 5 to 7 are where the money goes. That inversion is the single most useful thing on this page.

The one-off: building the index

ASSUMPTIONS — replace all of these

  corpus            2,000,000 tokens
  chunk size        500 tokens, 10% overlap  ->  ~4,400 chunks
  embedding rate    E   = $0.02 per 1M tokens   (assumed)
  dimensions        1,536, float32

INDEX BUILD
  tokens embedded   4,400 x 550  =  2.42M
  cost              2.42 x $0.02 =  $0.048        one-off

STORAGE
  raw vectors       4,400 x 1,536 x 4 bytes = 27 MB
  with HNSW graph   roughly 1.5-2x that      = ~50 MB

Five cents and fifty megabytes. This is the number that surprises people who have been putting off building a RAG system because “embedding the whole corpus” sounded expensive. Scale it up tenfold and it is fifty cents; a hundredfold and it is five dollars and 2.7 GB, at which point storage starts to be a real line and the embedding still is not.

The one-off becomes a recurring cost only through churn. If 5% of the corpus changes daily and you re-embed only what changed — which content-addressed chunk ids give you — that is a fortieth of a cent a day. If you re-embed everything nightly because your pipeline has no delta detection, it is $17 a year at this corpus size and $1,700 at a hundred times it, for no benefit whatsoever.

The per-query model

ASSUMPTIONS — replace all of these

  retrieve 25 candidates, rerank to 5
  context           5 x 550          = 2,750 tokens
  scaffolding       system + question =  250 tokens
  input total       3,000 tokens
  output            300 tokens
  rerank rate       R    = $2.00 per 1,000 searches   (assumed)
  generation in     Gin  = $0.30 per 1M tokens        (assumed, small model)
  generation out    Gout = $1.20 per 1M tokens        (assumed, small model)

PER QUERY
  query embedding   40 x $0.02/1M          = $0.0000008    ~0%
  retrieval compute (self-hosted, in-mem)  = ~$0           ~0%
  reranking         $2.00 / 1000           = $0.0020      61%
  generation input  3,000 x $0.30/1M       = $0.0009      28%
  generation output   300 x $1.20/1M       = $0.00036     11%
                                             ---------
                                             $0.00326

Three structural observations, all of which hold across a wide range of substituted rates.

• Query embedding is free. Forty tokens through the cheapest model class in existence. Anyone optimising this is optimising the wrong thing, and any architecture that avoids embedding the query at the cost of a generation call has made itself worse.
• The reranker is a real line item, and at these assumed rates the largest one. Per-search pricing does not shrink when your context does, so a reranker in front of a cheap small model can cost more than the model. Worth it for quality, but it belongs in the estimate.
• Generation input beats generation output. Output has the higher rate and input has ten times the volume. This is why context length discipline — fewer chunks, tighter chunks, no duplicated overlap — is the main cost lever in most pipelines.

The same model at three volumes

At $0.00326 per query (from the assumptions above):

  1,000 queries/mo      $3.26     index build dominates; ignore all of it
  100,000 queries/mo    $326      worth one afternoon of tuning
  5,000,000 queries/mo  $16,300   worth an engineer

Where the effort goes at each scale:
  small   nothing. Do not cache, do not tune. Ship.
  medium  cut context tokens; check whether the reranker earns its 61%.
  large   cache the head of the query distribution; consider a
          self-hosted reranker; route easy questions to a smaller model.

The scale column is the actionable part. At a thousand queries a month the entire annual cost is under forty dollars and every hour spent optimising it is a loss. Optimisation effort should be triggered by the absolute number, not by the per-query one — a per-query cost that sounds high is irrelevant if you have no queries.

Which line actually matters

Hold everything else fixed and vary one input at a time. This is the part to redo with your own rates, because it is where the decision lives.

The third row is the conclusion. Chunking strategy, reranker choice and cache design move the cost by tens of percent; the generation model moves it by multiples. Any cost work that does not start by asking whether the largest model is needed for every request is starting in the wrong place — and the natural answer is usually a cascade, where a small model answers and a larger one is called only when the small one declines or the question is classified as hard.

What this model leaves out

• Failed and retried requests. Timeouts, rate limits and validation failures are billed or partially billed and are invisible in a per-query estimate. Multiply by your actual retry rate.
• The evaluation loop. Every offline evaluation run is real spend, and if a judge model scores three hundred questions on every commit, that is a line item with its own growth curve.
• Managed vector store minimums. Many are priced per pod, index or hour rather than per query, so the first query costs the monthly minimum and the millionth is free. This inverts the small-scale conclusion above: at a thousand queries a month, a managed store can cost more than every model call combined.
• Egress and orchestration. Small per request, and not zero at five million.
• People. An engineer maintaining the pipeline costs more than any of the above until you are well past the third row of the scale table. Cost models that ignore this recommend self-hosting far too early.

The way to stop this being a spreadsheet exercise is to compute the cost of each request at the time you serve it. Every provider returns a usage block with input, output and cached-token counts; multiply those by the rates you have on file, add the fixed per-search fees for any rerank calls, and store the total on the trace alongside the tenant, the model and the number of chunks. That is perhaps thirty lines of code, and it replaces every estimate on this page with a measurement of your own system.

Once it exists, two views earn their keep. Cost per request as a distribution rather than a mean, because a p99 that is twenty times the median means some query shape is pathological and worth finding. And cost grouped by tenant or feature, because spend is almost never uniform — one customer, one endpoint or one badly-shaped retry loop is usually responsible for a share of the bill wildly out of proportion to its traffic, and no amount of per-token optimisation finds it.

A last note on how to read any cost model, including this one. The rates are the least durable thing in it — they move every quarter, and almost always downward per unit of capability — while the structure is stable. Input volume dominates output volume because retrieval puts thousands of tokens in and asks for hundreds back. Per-search fees do not shrink when your context does. Embedding is negligible at every scale that is not a full reindex. And the model tier is a multiplier on the whole thing, which is why it is the only decision on this page that changes the answer by more than a factor of two.