Do You Even Need a Vector Database?

The honest answer to “at how many rows do I need a vector database” is that there is no such number, because it depends on your dimension, your latency budget and your memory bandwidth. All three are things you know, so you can compute it in about a minute.

The brute-force ceiling, derived

An exhaustive search reads every vector once and computes a dot product. Dot products of this size are trivially vectorised, so the operation is bound by how fast you can pull the data through memory, not by arithmetic. The whole model is one line:

scan_time = (N * 4 * d) / bandwidth

at d = 1536 and a conservative 20 GB/s of sequential bandwidth:

    N =  10,000:      61 MB  ->   3 ms
    N = 100,000:     614 MB  ->  31 ms
    N = 500,000:   3,072 MB  -> 154 ms
    N = 1,000,000: 6,144 MB  -> 307 ms

Now invert it against your latency budget. If you can spend 50 ms on retrieval inside a request that also calls a language model, the ceiling at 1536 dimensions is a bit over 160,000 vectors. Change one input and the ceiling moves proportionally: at 384 dimensions it is around 650,000; storing float16 instead of float32 doubles it again; a machine with genuinely fast memory doubles it once more. Those are all multiplicative on the same formula, which is why a single remembered row count is useless advice.

Two adjustments make the estimate honest. Sequential bandwidth is the optimistic case and you get it only if the vectors are contiguous in one array — a scan over Python objects or ORM rows is an order of magnitude worse and you should assume that unless you have written it otherwise. And concurrency divides your budget: ten simultaneous queries share the same memory bus, so a 31 ms scan is 310 ms of bus time when ten of them arrive together.

An index in fifteen lines

Below the ceiling, this is a complete and correct vector search. It has recall of exactly 1.0, no build step, no tuning, no extra process, and no way to be subtly wrong:

import numpy as np

# (N, d) float32, L2-normalised once at load time
V = np.load("vectors.npy")
V /= np.linalg.norm(V, axis=1, keepdims=True)

def search(q, k=10):
    q = q / np.linalg.norm(q)
    scores = V @ q                 # one BLAS call, all N at once
    idx = np.argpartition(-scores, k)[:k]      # O(N), not a full sort
    return idx[np.argsort(-scores[idx])]

The argpartition matters at scale: a full argsort of a million scores costs more than the dot products did. Beyond that there is nothing to get wrong, which is worth something. Every approximate index you will later adopt is a machine for trading away the exactness of these five lines, and you should know what you are trading before you trade it.

Postgres, and its real limits

The next rung is pgvector, and the argument for it is not performance — it is that the vector and the row it belongs to are in the same transaction. No dual-write, no reconciliation job, no window where the index describes a document that has been deleted. Its concrete limits are worth knowing before you plan around it:

The 2,000-dimension index limit catches people, because the obvious modern default is 3,072. Creating the column succeeds; creating the index fails with an error stating the column cannot have more than 2,000 dimensions for an HNSW index. The fixes are all fine — truncate a Matryoshka model to 1,536 or below, or store as halfvec — but discovering the constraint after loading 40 million rows is a bad afternoon.

Two settings do most of the tuning work. hnsw.ef_search defaults to 40 and is the recall dial at query time; raise it per session for queries that need better recall. For IVFFlat, pgvector’s own guidance is to use roughly rows/1000 lists up to a million rows and about sqrt(rows) beyond that, then set ivfflat.probes to about the square root of the list count.

Five signals you have outgrown it

• The vectors no longer fit in RAM alongside everything else. This is the real threshold and it is a memory calculation, not a row count. Once the working set exceeds RAM, performance does not degrade gracefully — it falls off a cliff as random reads start hitting disk.
• Filtered queries return too few rows. The symptom of post-filtering, and the clearest sign you need an engine whose filtering strategy you have deliberately chosen.
• Rebuild time exceeds your update window. If the index takes six hours to build and your corpus changes daily, you have an operational problem no amount of tuning fixes.
• You need a second index type. Wanting binary vectors for a first pass and full-precision for rescoring, or a sparse index alongside a dense one, is where purpose-built engines start earning their keep.
• You are running multi-tenant search at scale. Thousands of tenants each needing isolated, selective search is the workload that most reliably breaks a single shared graph.

Note what is not on the list: “we crossed a million rows”. A million 768-dim vectors is 3 GB. That is a laptop.

The ladder

Climb it one rung at a time and stop as soon as the numbers say you can. numpy in the application process; then pgvector with an exact scan and no index; then pgvector with HNSW; then a dedicated engine; then a sharded one. Each rung adds a system to operate, and the cost of that system is paid continuously while the benefit only materialises after you have actually crossed the threshold that justifies it.