Grounding Vertex AI Responses With Your Own Data

Grounding is a tool declaration in the request and a metadata block in the response. The request half takes ten minutes. The response half is where the work is, because turning it into citations a reader can click requires knowing exactly what the offsets mean.

Three grounding sources, one response shape

Google documents grounding against Google Search, against a Vertex AI Search data store holding your own documents, and against Google Maps, with additional connectors for third-party search backends. They are different tools in the request and they converge on the same groundingMetadata object in the response, which is the useful thing: code that renders citations for one source renders them for all of them, with one field’s worth of difference.

The distinction that matters for a private corpus is that grounding with your own data does not mean the model reads a bucket. It means a Vertex AI Search data store has already ingested and indexed those documents, and the tool issues retrieval queries against that index at generation time. Building the data store is a prerequisite, not part of the request.

The request

Grounding is declared in the tools array of an ordinary generateContent call. For a data store, the retrieval tool names the data store by its full resource path, which is regional in a misleading way — data stores commonly live under locations/global even when the model call is regional.

POST https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT/locations/us-central1/publishers/google/models/MODEL:generateContent

{
  "contents": [
    { "role": "user", "parts": [{ "text": "What is our refund window for annual plans?" }] }
  ],
  "tools": [
    {
      "retrieval": {
        "vertexAiSearch": {
          "datastore": "projects/PROJECT/locations/global/collections/default_collection/dataStores/DATASTORE_ID"
        }
      }
    }
  ]
}

Swapping that tool for a Google Search tool changes the source and nothing else about how you read the answer. What it does change is your obligations: Search grounding returns a searchEntryPoint containing rendered HTML that Google requires you to display alongside the response, so a UI that discards it is not compliant with the terms it was served under.

What comes back

The answer text arrives in the usual candidates[0].content.parts. The citation apparatus sits beside it in candidates[0].groundingMetadata, and Google’s API reference for GroundingMetadata defines these fields:

• groundingChunks — the supporting references retrieved from the source. Each carries either a web object or a retrievedContext object, both with uri and title. This is your bibliography.
• groundingSupports — the links between the generated text and those chunks. This is the part that makes inline citations possible.
• segment, inside each support — the span of the answer this support covers, as startIndex, endIndex and a copy of the text.
• groundingChunkIndices — indices into groundingChunks. Google’s reference gives the example that a value of [1, 3] means chunks 1 and 3 support that claim.
• confidenceScores — a list parallel to the chunk indices, scored 0.0 to 1.0. Google’s documentation states that for Gemini 2.5 and later this list is empty and should be ignored.
• retrievalQueries — the queries the retrieval tool actually ran, populated for retrieval sources. Log these. They are the fastest way to understand why a grounded answer missed a document you know exists.
• webSearchQueries and searchEntryPoint — the Search equivalents, populated when the source is Google Search.

Turning supports into citations

1. Read the answer text once and keep it as a byte buffer, not a string.
2. For each entry in groundingSupports, take its segment.startIndex and segment.endIndex and slice that range out of the buffer.
3. Map its groundingChunkIndices onto groundingChunks to get the URIs and titles, and render a marker after the slice.
4. Walk the supports in reverse index order when inserting markers, so each insertion does not shift the offsets of the ones you have not processed yet.

Step one is not fussiness. Segment offsets are byte offsets into the UTF-8 encoding of the answer, and JavaScript string indices are UTF-16 code units. For pure ASCII the two agree and the bug never appears; the first time a response contains an em dash, an accented name or an emoji, every citation after it lands a character or two off. Decode from bytes, or work in a language where string indexing is already byte-based.

How much of the retrieval you control

Grounding hands the retrieval step to the platform. The model decides whether to search, formulates the query itself — that is what retrievalQueries is showing you — and decides how many chunks to pull. You do not choose the embedding model, you do not set a similarity threshold, and you do not get to rerank before generation. In exchange you write one tool declaration instead of a retrieval pipeline.

That trade is excellent until the first quality complaint, at which point the levers available to you are: the contents of the data store, the instructions in the prompt, and the choice of model. If a document is in the store and never retrieved, the fix is on the ingestion side — chunking, titles, metadata — because there is no retrieval parameter to turn. Reading retrievalQueries against the document you expected is the fastest diagnosis available, and it usually shows the model searching for a phrase your documents do not use.

Where you need control — a specific embedding model, per-tenant filtering enforced before the search rather than after, a reranker, a hybrid of keyword and vector scoring — the answer is to run retrieval yourself and pass the results in as context, which is what a Vector Search index is for. The cost of that choice is that you also inherit the citation problem: nothing populates groundingMetadata for context you supplied yourself, so the spans and the bibliography above become something you construct. Teams frequently want managed grounding’s citations with self-managed retrieval’s control, and those are the two halves you are choosing between.

A middle path exists and is underused: ground against a data store for the general case and fall back to your own retrieval only for the query classes where you have measured the managed path failing. Grounding metadata tells you which those are, because a grounded answer with no chunks is a visible event rather than a silent one.

Four things that go wrong

• Empty groundingMetadata on a perfectly good answer. The model decides whether to invoke the retrieval tool. If it answers from parametric knowledge, there is no metadata and no citation, and nothing has failed. If you require grounding, you have to check for the absence and handle it, not assume it.
• Building a confidence filter on confidenceScores. Reasonable-looking code that drops citations below 0.6 silently drops all of them on 2.5-and-later models, because the list is empty.
• Overlapping supports. Segments are not guaranteed disjoint. A naive renderer that assumes one citation per span produces nested markup or drops the second of a pair.
• Forgetting grounding is billed separately. Google’s Vertex AI pricing page lists grounding with your own data at $2.50 per 1,000 prompts at the time of writing, charged on top of the tokens. A high-volume grounded endpoint can spend more on retrieval than on generation.