RAG Cost Calculator

Almost every RAG cost discussion is about the vector database, and almost every RAG bill is about the generation call. The breakdown above usually makes that plain within a few seconds of arriving: storing a few hundred thousand vectors costs less per month than a single engineer's lunch, while sending several thousand retrieved tokens into a model on every one of a hundred thousand queries costs real money. If your fully loaded cost per query is dominated by generation — and it generally will be — then tuning your index is optimising the wrong term.

The lever that matters is top-k multiplied by tokens per chunk. That product is your prompt, and it scales linearly with the bill. Halving the number of retrieved chunks halves the retrieval half of the input cost, and if a reranker lets you go from ten mediocre chunks to three good ones you have cut cost and improved the answer at the same time. The second lever is chunk size, which is a retrieval-quality decision long before it is a cost decision, but it has a cost consequence people forget to price.

What is missing here is the part that ages worst: re-embedding. The one-time index cost looks trivial until you change embedding model, change chunking strategy, or add a field, at which point you pay it again in full. Two re-indexes a year is a modest assumption for a system under active development. If yours is closer to monthly, raise that field and watch the amortised figure stop being a rounding error.