Fixing 429 RESOURCE_EXHAUSTED on Vertex AI

429 RESOURCE_EXHAUSTED is six different problems wearing one status code. The quota identifier inside the message tells you which, and for two of the six a quota increase request is not the answer at all.

Read the whole message

The Python client raises it as google.api_core.exceptions.ResourceExhausted: 429, and the interesting part is the text after the code, which normally names the limit that was hit:

google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for
aiplatform.googleapis.com/online_prediction_requests_per_base_model
with base model: gemini-2.0-flash. Please submit a quota increase request.

Three things are encoded there: the metric, the dimension the quota is counted along (here a base model, often also a region), and whether Google thinks an increase is available. If your logs are swallowing the message body and keeping only the status code, fix that first — without the metric name every remaining step is guesswork.

Note the dimension carefully, because it is not the dimension people expect. Quotas here are counted per base model, not per tuned model and not per endpoint: three services in one project calling the same base model share one allowance, and the one that gets throttled is whichever happened to be sending when the limit was reached. That is why a 429 frequently appears in a service whose own traffic has not changed at all. Before touching that service, check what else in the project talks to the same model — a batch script somebody ran by hand is a common answer, and it is not visible from the affected service’s own dashboards.

Named quotas: per minute and concurrent

The two named quotas that produce most 429s count entirely different things, and the distinction decides the fix.

• Requests per minute per base model per region. A rate. Sixty requests spread evenly across a minute and sixty fired in the first second both consume the same amount, but the second pattern hits the limit and the first does not, because the counter is not perfectly smooth. The fix is pacing and backoff.
• Concurrent requests per base model. A depth, seen in the wild as Quota exceeded for online_prediction_concurrent_requests_per_base_model. This one counts requests in flight, so it is hit by long generations rather than by many of them. A workload that streams two-thousand-token answers holds each slot for many seconds; a hundred such requests overlap even at a modest request rate. Backoff barely helps. Bounding your own client concurrency does.
• Tokens per minute. Some models are limited on token throughput rather than request count, which is why a project can see a 429 while its request-per-minute dashboard looks half empty. A handful of very long prompts is enough.

When there is no number to raise

Several Gemini models are served from a shared pool rather than a per-project allowance. Google’s troubleshooting page for error 429 describes the pay-as-you-go model as using a shared pool of resources, and states that if resources are not available when you make a request, Vertex AI returns a 429.

This is the case where the usual instinct fails. There is no quota counter sitting at 100% for you to raise; the request was refused because the pool was busy, and the same request a second later may succeed. Two responses are real: retry properly, or stop sharing. Google documents Provisioned Throughput as the mechanism for a consistent level of service — you commit to reserved capacity and stop competing for the shared pool. It is a purchase, not a form.

The diagnostic tell is the quota page itself. If you filter the Quotas page to aiplatform.googleapis.com and cannot find a numeric limit matching the metric in your error, you are in this case and a quota increase request will not resolve it.

Acceleration limits

Google’s 429 troubleshooting documentation also notes that projects can hit acceleration limits after a sharp increase in usage, and advises ramping traffic gradually and maintaining consistent usage patterns.

This one is invisible in a dashboard and catches launches specifically. A project that has averaged five requests a minute for a month and jumps to five hundred on release day can be throttled at a level well below its nominal quota, because the platform is protecting itself against a shape of traffic it has no history of. If a launch is scheduled, generate realistic load in the days before it rather than discovering this at peak.

Custom endpoints fail differently

Everything above concerns Google’s hosted models. A 429 from your own deployed endpoint is a different animal: it means the replicas behind that endpoint are saturated and the request could not be queued. There is no per-model quota involved; the constraints are --max-replica-count, the autoscaling signal, and whether the region has accelerator capacity to scale into. Raising the replica ceiling is the fix, and accelerator quota is what stops you raising it.

The diagnostic that separates the two in ten seconds is the endpoint in the URL. A request to publishers/google/models/MODEL:generateContent hit a foundation-model quota; a request to endpoints/ENDPOINT_ID:predict hit your own capacity. Different cause, different dashboard, different fix, identical status code — and an on-call engineer who does not know this will spend the first twenty minutes on the wrong one.

What to actually change

1. Retry with exponential backoff and full jitter. Not fixed delays — synchronised clients retrying at the same interval reproduce the burst that caused the 429. Cap total attempts so a sustained outage does not turn into an unbounded queue.
2. Bound concurrency at the client. A semaphore sized below the concurrent-request quota converts a storm of failures into slightly slower success. This is the single most effective change for long-generation workloads.
3. Separate the traffic that can wait. Move bulk scoring onto batch prediction, which draws on different capacity and is cheaper, so interactive traffic is not competing with it.
4. Spread across regions. Named quotas are per region. Two regions is two allowances, at the cost of managing residency and two sets of latency.
5. Buy the capacity. Where the workload has a floor it cannot drop below, Provisioned Throughput replaces contention with a commitment.