Building a Vertex AI Vector Search Index

Vertex AI Vector Search is fast and it is not serverless. The index is cheap to hold and the thing that answers queries is a fleet of machines that runs until you undeploy it. Everything else follows from that.

The data shape

Input is JSON Lines in Cloud Storage, one record per line, each with an id and an embedding array. The array length must equal the dimensionality you declare when creating the index and must be identical across every record; a single row of the wrong width fails the whole build, which is why validating dimensionality before upload is worth the five lines it takes.

{"id": "doc-1041", "embedding": [0.0123, -0.0412, ...], "restricts": [{"namespace": "tenant", "allow": ["acme"]}]}
{"id": "doc-1042", "embedding": [0.0087, 0.0330, ...], "restricts": [{"namespace": "tenant", "allow": ["acme"]}]}

The restricts field is the part worth designing for on day one. It attaches namespaced tokens to a datapoint and lets a query filter on them before the neighbour search, which is how you get per-tenant isolation without one index per tenant. Adding it later means rebuilding. A crowding_tag is the other optional field, used to cap how many results may share a value so a single verbose document does not occupy every slot in a top-ten.

Creating the index

The index points at a Cloud Storage directory, not a file — the contentsDeltaUri — and reads every JSONL file underneath it. Two algorithm choices exist: a tree-AH approximate index, which is what you want for anything large, and brute force, which is exact and exists mainly to give you ground truth for measuring the approximate index’s recall.

from google.cloud import aiplatform

aiplatform.init(project="PROJECT", location="us-central1")

index = aiplatform.MatchingEngineIndex.create_tree_ah_index(
    display_name="docs-index",
    contents_delta_uri="gs://PROJECT-embeddings/docs/",
    dimensions=768,
    approximate_neighbors_count=150,
    distance_measure_type="DOT_PRODUCT_DISTANCE",
)

Google’s Vector Search quickstart notes that index creation takes “under 10 minutes if the dataset is small, otherwise about 60 minutes or more”, so this is a step to start and walk away from rather than one to iterate on. approximate_neighbors_count tunes the recall/latency trade — it is not the number of results you get back, which is a query-time parameter.

Index and index endpoint are different things

An index is data. An IndexEndpoint is serving infrastructure, and deploying an index onto one is a third operation with its own identifier. Google’s quickstart notes the first deployment can take “around 30 minutes to automatically build and initiate the backend”.

endpoint = aiplatform.MatchingEngineIndexEndpoint.create(
    display_name="docs-endpoint",
    public_endpoint_enabled=True,
)

endpoint.deploy_index(index=index, deployed_index_id="docs_v1")

That deployed_index_id is required on every query, and it is a string you choose rather than one the platform hands back. Choose one that encodes the version, because the clean way to reindex is to deploy a second index onto the same endpoint under a new deployed index ID, switch your queries over, and undeploy the old one.

The billing consequence is the headline. A deployed index occupies machines continuously. There is no scale-to-zero and no per-query price; an index endpoint that answered nothing all month still billed for the month. Set public_endpoint_enabled=False and use Private Service Connect if the traffic must stay off the public internet, but that changes the network path, not the cost model.

Querying for neighbours

neighbors = endpoint.find_neighbors(
    deployed_index_id="docs_v1",
    queries=[query_embedding],
    num_neighbors=10,
    filter=[Namespace("tenant", ["acme"], [])],
)

You supply the embedding, not the text: nothing in Vector Search embeds for you, so the query vector has to come from the same model and the same version that produced the index. Mixing embedding model versions between build and query is the classic silent failure here — the call succeeds, the distances are meaningless, and relevance quietly collapses. Pin the embedding model identifier next to the index name in configuration so the two cannot drift apart.

The response gives you IDs and distances. It does not give you your documents; Vector Search stores vectors and identifiers, not payloads. Plan on a second lookup against whatever holds the text, and note that this second hop, not the neighbour search, is usually where retrieval latency actually goes.

Keeping it current

Two update modes exist and they are chosen at index creation time, not later. Batch update rebuilds from a new contentsDeltaUri on a schedule you drive. Stream update accepts individual upserts and removals against a live index, which is what you want if documents change during the day and a rebuild cycle is too slow. Streaming costs more to hold and constrains some index parameters, so the decision is worth making deliberately rather than defaulting into.

The word delta in contentsDeltaUri is doing more work than it looks like. A batch update applies the contents of that directory as a change set against what the index already holds, which means deletions are not implied by absence: a document you simply stop including is still in the index and will still be returned. Removal is an explicit operation, and the usual way this is discovered is a deleted customer record turning up in a search result months later. Keep a tombstone path in your pipeline for anything that must actually leave, and treat “the source of truth no longer contains it” as insufficient.

A rebuild is the honest answer to a schema change rather than an update. Changing dimensionality, changing the distance measure, or adding a restricts namespace that did not exist all require a new index — none of them can be applied to a live one. Because deployment is the slow step and a single index endpoint can host several deployed indexes, the standard procedure is to build the new index alongside the old, deploy it under a second deployed_index_id, run both in parallel long enough to compare results on real queries, cut over, and only then undeploy. That is also the only safe way to change embedding models, because the query vectors and the index have to change at the same instant.

One measurement is worth building the harness for while you are here. A brute-force index over the same data gives exact neighbours, so comparing its results against the tree-AH index’s on a sample of real queries tells you the recall you are actually getting rather than the recall you assumed when you set approximate_neighbors_count. That is a number you can only get from your own data, and it is the one that decides whether a retrieval quality complaint is about the index or about the embeddings.

When not to use it

If your vectors already live in BigQuery and your query volume is bursty, the BigQuery VECTOR_SEARCH path is priced per query rather than per node-hour and involves no always-on infrastructure. It is slower per query and it is dramatically cheaper at low volume. The honest rule of thumb is that Vector Search earns its endpoint when queries are continuous and latency is user-facing; below that, a managed index is a monthly bill for capacity you are not using. The BigQuery ML remote model path is the same argument applied to inference rather than retrieval.