Google Cloud Tasks for Rate-Limited Model API Calls

Every team that calls a model API eventually writes a token bucket. Cloud Tasks already has one, it is enforced outside your processes, and it survives an autoscaler doubling your worker count — which the in-process version does not.

The queue is the rate limiter

An application-level limiter has a structural problem: it limits one process. Run ten replicas and you have ten limiters, so your real rate is ten times the number you configured, and it changes whenever the autoscaler acts. Fixing that properly means shared state, usually Redis, and now the limiter has its own availability story.

Cloud Tasks inverts the arrangement. The queue holds the work and calls your handler at a rate it enforces, so the throttle lives outside every replica by construction. Your handler contains no limiting logic at all; it does one unit of work and returns a status code.

The second property that matters here is that a task is addressed. Unlike a pull queue, an HTTP target task names a URL, so a single queue can be the throttle in front of an endpoint rather than in front of a worker pool. That is the right granularity when the thing being protected is a provider quota.

Dispatch rate, burst and the token bucket

Google documents three rate-limit fields on a queue. max_dispatches_per_second is the dispatch rate, described as the rate at which tokens in the bucket are refreshed. max_concurrent_dispatches is the maximum number of tasks that can run at once. max_burst_size controls how much the queue can spike above the sustained rate, and is documented as calculated by the system from the value you set for the dispatch rate rather than set directly.

The defaults are permissive. Google’s documented example output shows maxDispatchesPerSecond of 500.0, maxConcurrentDispatches of 1000 and maxBurstSize of 100. A queue you create and do not configure is not protecting anything.

gcloud tasks queues create model-calls \
  --max-dispatches-per-second=8 \
  --max-concurrent-dispatches=12 \
  --max-attempts=5 \
  --min-backoff=2s \
  --max-backoff=120s \
  --max-doublings=4

Why concurrency is the setting that binds

This is the part that is not obvious and that makes the difference on a model workload. Dispatch rate governs starts per second. Concurrency governs how many can be in flight. When each task finishes in milliseconds the two are nearly interchangeable, because tasks retire as fast as they start.

A model call does not retire quickly. At 8 dispatches per second and an average call of 20 seconds, the steady-state in-flight count is 8 × 20 = 160 concurrent calls — far past most providers’ concurrency allowance, and reached about twenty seconds after you turn the queue on. The concurrency setting is what stops that, and the arithmetic that gives you the right value is Little’s law: concurrency equals arrival rate times service time.

So set max_concurrent_dispatches to the number of simultaneous calls the provider will accept, and derive the dispatch rate from it rather than the other way round: rate = concurrency divided by your p95 call duration. With a concurrency of 12 and a 20-second p95, a dispatch rate above 0.6 per second only queues work at the dispatcher instead of at the provider — which is still better than a 429, but pretending otherwise leads people to raise the wrong number.

Retries, 429s and the deadline

Cloud Tasks retries any task whose handler does not return a 2xx, using the queue’s retry configuration. Google documents the fields as max_attempts (including the first), max_retry_duration, min_backoff, max_backoff and max_doublings, with example defaults of 100 attempts, a minimum backoff of 0.100s, a maximum of 3600s and 16 doublings.

A hundred attempts is a great default for a webhook and a poor one for a billed model call. Set it down. And be careful what you return: if your handler catches a provider 400 and returns 500, Cloud Tasks will retry an invalid request until it exhausts the attempts. Return 2xx for anything permanently unprocessable after recording the failure, and reserve non-2xx for genuinely transient conditions.

Provider throttling is exactly such a condition, and the right handler returns a non-2xx so the queue backs off rather than sleeping inside the handler. Sleeping holds a concurrency slot for the whole backoff, which reduces your throughput by the thing you were trying to avoid.

Finally, the dispatch deadline. Google documents a default timeout of 10 minutes for HTTP target task handlers, with a maximum of 30 minutes. That is comfortable for a single model call and is worth lowering rather than raising: a task that has hung for ten minutes has held a concurrency slot for ten minutes.

Building it

1. Find the provider’s documented concurrent-request allowance for your account and model. That number, not a guess, is your max_concurrent_dispatches.
2. Measure p95 call duration and set the dispatch rate to concurrency divided by that duration.
3. Create the queue with those two values plus a retry config with a sane --max-attempts — five, not a hundred.
4. Create tasks with an httpRequest naming url, httpMethod, headers, body and an oidcToken with a serviceAccountEmail, so the handler can stay private.
5. Make the handler return 2xx on permanent failure and non-2xx only on transient failure, and never sleep in it.
6. Watch the provider’s 429 rate rather than the queue’s. The queue will report itself as healthy while the provider rejects everything; only one of those two numbers tells you the settings are right.