Prompts for Understanding a Search Query

Query understanding is worth one small call and no more. This prompt returns intent, entities, filters, a rewrite and expansions in a single response of about 120 tokens, and the section on latency explains why you should race it rather than wait for it.

The prompt

A user typed a query into the search box of {{product_description}}.
Interpret it. Return JSON only, with exactly these keys.

{
  "intent": "navigational" | "informational" | "transactional" | "support"
            | "unclear",
  "entities": [{"text": "<verbatim substring of the query>",
                "type": "product" | "brand" | "sku" | "person" | "place"
                        | "date_range" | "price_range" | "other",
                "normalised": "<canonical form, or null>"}],
  "filters": {"price_min": null, "price_max": null, "date_from": null,
              "date_to": null, "in_stock": null, "category": null},
  "rewrite": "<the query as a well-formed search, keeping every entity>",
  "expansions": ["<up to 3 alternative phrasings that retrieve documents this
                   query would miss>"],
  "spelling": {"corrected": "<or null>", "confidence": "high" | "low"},
  "unclear_because": "<or null>"
}

Rules:
- Every "entities[].text" must be a verbatim substring of the query. If you
  cannot point at it in the query, it is not an entity.
- Never set a filter the query does not state. "cheap" is not a price_max.
  "recent" is not a date_from. Leave them null.
- "rewrite" may reorder, expand an abbreviation and add an implied noun. It
  may not add a constraint, a brand, or a category the query does not name.
- "expansions" are for recall: a synonym, the singular or plural, the
  expanded acronym, the term the vendor uses for the same thing. Not
  paraphrases of the whole query, not adjacent products, not narrower
  variants.
- Correct spelling only when the corrected form appears in <vocabulary>.
  Otherwise leave "corrected" null. A confident wrong correction is worse
  than none.
- Set intent "unclear" for a query with no entity and no verb, and say why.
  Do not guess an intent to avoid the label.

<vocabulary>
{{brand_and_category_terms}}
</vocabulary>

Query: {{query}}

Why these fields and not others

Notice what is not here: no relevance judgement, no ranking hint, no answer. Every one of those makes the call bigger, slower and harder to cache, and each is better done by the retrieval stack. Query understanding is a parsing job.

What the call costs

The whole point is that this is cheap, so count it rather than assuming it.

Fixed prefix   instruction + schema + rules   ~380 tokens
               <vocabulary> (300 terms)       ~450 tokens
                                              ----------
                                              ~830 tokens, identical every call

Variable       the query itself               ~4-10 tokens
Output         the JSON above                 ~110-140 tokens

Per call:      ~840 input, ~125 output

With prompt caching on the fixed prefix, roughly 830 of the 840 input tokens
are charged at the cached rate. Fill in your own model's two input prices and
its output price:

   cost = 830 x cached_in + 10 x in + 125 x out

At 1,000,000 queries a month, the output tokens dominate: 125,000,000 output
tokens is the line to look at, not the input.

Two consequences follow from that arithmetic. Keep the output schema tight — every optional field you add is paid a million times, and the expansions array is the one that grows. And put vocabulary in the fixed prefix rather than interleaving it with the query, so the cache prefix covers it; the mechanics of that boundary are in cached tokens.

This is also a task where a small model is usually enough, because nothing here requires world knowledge beyond your vocabulary list. Check it on your own traffic before assuming so — the place it breaks first is normalised.

Latency is the real constraint

The cost is trivial. The latency is not: this call sits in front of search, so its time to first token is added to every search a user performs, and search is a place where a hundred milliseconds is noticeable.

Do not block on it. Race it.

1. Fire the lexical search on the raw query and the understanding call at the same instant.
2. Give the understanding call a hard timeout — 200 to 300 ms is a realistic budget for a short output on a small model, but measure your own p95 rather than trusting that range.
3. If it returns in time, use the rewrite and the filters, and issue the expansion queries.
4. If it does not, serve the raw-query results. They are worse and they are not broken, which is the correct trade at the ninety-fifth percentile.
5. Cache the result keyed on the normalised query string. Head queries repeat constantly, and a cache hit removes the call entirely.

The cache is the largest win available here and it is often skipped. In most search traffic a small number of distinct queries account for a large share of volume, so a query-string cache with a long expiry converts most of your calls into lookups. Measure the distribution before sizing it; the shape varies enormously between products.

Expansions that help retrieval

The instruction is deliberately narrow — synonym, plural, expanded acronym, vendor’s term — because unconstrained expansion makes retrieval worse. Left open, models produce related-but-different queries: ask for expansions of “running shoes” and you get “marathon training plans”, which retrieves documents that are genuinely about something else and dilutes the result set.

Run expansions as additional queries and merge, rather than replacing the original. The original is the only query you know the user meant. For the retrieval-side treatment of the same problem — turning one bad question into three good queries and retrieving with a hypothetical answer — the pages in the RAG cluster go further into what to do with the results.

Four queries, worked

A hardware retailer, with vocabulary containing its brands and categories. Fields left at their defaults are omitted here for space; the real output always carries the full filters object.

-- "levis 501 34x32" --------------------------------------------------------
{"intent": "transactional",
 "entities": [{"text": "levis", "type": "brand", "normalised": "Levi's"},
              {"text": "501", "type": "product", "normalised": "501 Original"},
              {"text": "34x32", "type": "other", "normalised": "W34 L32"}],
 "filters": {...all null...},
 "rewrite": "Levi's 501 Original jeans W34 L32",
 "expansions": ["Levis 501 straight fit 34 32"],
 "spelling": {"corrected": null, "confidence": "high"},
 "unclear_because": null}

Note what is NOT here: no category filter for "jeans", even though the
rewrite mentions them. The rewrite may add an implied noun; the filter may
not, because a wrong category filter returns nothing and the user cannot see
why.

-- "cheap running shoes" ----------------------------------------------------
{"intent": "transactional",
 "entities": [{"text": "running shoes", "type": "product",
               "normalised": "running shoes"}],
 "filters": {"price_max": null, ...},
 "rewrite": "running shoes low price",
 "expansions": ["trainers running", "jogging shoes"],
 "unclear_because": null}

"cheap" produced no price_max. It is the rule that most often gets deleted by
somebody trying to be helpful, and it is the one that causes empty result
pages: there is no number in the query, so any threshold is invented.

-- "return policy" ----------------------------------------------------------
{"intent": "support",
 "entities": [],
 "rewrite": "returns policy",
 "expansions": ["how to return an item", "refund policy"],
 "unclear_because": null}

Intent "support" rather than "informational", so your handler can search help
content rather than the catalogue. This is the single highest-value field in
the whole schema for most retail search: it routes the query to a different
index.

-- "hoka" -------------------------------------------------------------------
{"intent": "navigational",
 "entities": [{"text": "hoka", "type": "brand", "normalised": "HOKA"}],
 "rewrite": "HOKA",
 "expansions": [],
 "spelling": {"corrected": null, "confidence": "high"},
 "unclear_because": null}

One word with one entity: navigational, and the right response is the brand
page rather than a result list. Contrast with a one-word query that matches
nothing in <vocabulary>, which returns intent "unclear" and a suggestion UI.

When it stops working

• Filters start appearing on queries that do not state them. The most damaging failure here, because a phantom filter removes results silently and the user sees an empty search. Assert it: any non-null filter must correspond to a token in the query.
• unclear collapses to zero. One-word queries with no entity exist in every search log. If none are labelled unclear, the model has started guessing intents.
• Spelling corrections outside the vocabulary. Check mechanically — the corrected form must be in the vocabulary list. This is the field that most often starts hallucinating after a model change.
• The timeout fires more often. Look at output length before blaming the provider. A schema that has grown by two fields is a common cause, and a longer output at the same rate is a slower call.