Pub/Sub Triggering a Cloud Run Worker for Model Calls

A push subscription turns your worker into an HTTP server that Pub/Sub calls. That is convenient right up to the moment the work takes longer than the acknowledgement deadline, at which point Pub/Sub starts delivering the same message to a second instance while the first is still talking to the model.

Push, and what it means for your concurrency

With a pull subscription your worker decides how much work to take. With a push subscription Pub/Sub decides, and your only levers are the service’s own maximum instance count and per-instance concurrency. For a model-calling worker this matters more than it does for a normal web service, because the thing you are protecting is not your CPU — it is a provider quota that is shared across every instance you run.

Cloud Run’s default per-instance concurrency is high, which is right for a service that spends its time waiting on I/O and wrong if each in-flight request holds a provider slot. Set --concurrency to the number of simultaneous model calls one instance should make, and --max-instances to bound the total. The product of the two is your real request rate against the provider, and it is worth writing that multiplication down somewhere.

The envelope your handler receives

Pub/Sub wraps the message. Your handler gets a JSON body with a message object containing base64-encoded data, a messageId, a publishTime, optional attributes, an optional orderingKey, and — if you have configured a dead letter topic — a deliveryAttempt counter. Google documents both camelCase and snake_case spellings of the id and timestamp fields, so read defensively rather than assuming one.

The deliveryAttempt field is the one worth using. It is the cheapest signal you have that this message has been tried before, which on this workload means the previous attempt may already have made a billed call.

import base64, json, os
from flask import Flask, request

app = Flask(__name__)

@app.post("/")
def handle():
    envelope = request.get_json(silent=True) or {}
    message = envelope.get("message", {})
    payload = json.loads(base64.b64decode(message["data"]).decode())
    attempt = envelope.get("deliveryAttempt", 1)

    if already_done(payload["job_id"]):
        return "", 204                    # ack: nothing to redo

    try:
        write_result(payload["job_id"], call_model(payload["prompt"]))
    except RetryableError:
        return "", 500                    # nack: Pub/Sub redelivers
    return "", 204

The status codes are not a convention, they are the protocol. Google documents that returning 102, 200, 201, 202 or 204 acknowledges the message, and that any other status code is a negative acknowledgement that causes redelivery. A 404 from a mis-routed path is a nack. A 403 because the invoker permission is missing is a nack, forever, at the full retry rate — which is how a permissions mistake turns into a bill.

Ack deadline against Cloud Run’s request timeout

Two independent clocks are running on the same request and both can end it. Google documents the subscription acknowledgement deadline as defaulting to 10 seconds, with a minimum of 10 and a maximum of 600 seconds, and notes that you cannot modify the deadline of an individual message received through a push subscription — the per-message extension trick available to pull subscribers is not available to you here. Separately, Cloud Run’s request timeout defaults to 5 minutes (300 seconds) and can be extended to 60 minutes (3600 seconds).

So the ceiling for a push-delivered model call is 600 seconds, whatever you set on Cloud Run. Raising the Cloud Run timeout past ten minutes on a push-triggered service buys you nothing except a longer window in which Pub/Sub has already given up and redelivered. Set the ack deadline to comfortably above your worst-case call, set the Cloud Run timeout slightly above the ack deadline, and treat 600 seconds as a hard architectural limit: past it, the answer is a pull subscription or Cloud Tasks, not a bigger number.

Two more defaults are worth setting rather than inheriting. Message retention defaults to 7 days, with a documented range of 10 minutes to 31 days — a week of accumulated model jobs replayed after an outage is its own incident. And a dead letter topic defaults to 5 delivery attempts, configurable to any number between 5 and 100 inclusive, which is the setting that stops a permanently poisonous message being retried until the retention window expires.

Authenticating a private service

Deploy the service with --no-allow-unauthenticated. Then give the push subscription an oidcToken naming a service account, grant that service account the Cloud Run Invoker role on the service, and verify the resulting bearer token in front of your handler — or let Cloud Run’s own IAM check do it, which is the reason to keep the service private in the first place.

There is one grant people miss: Pub/Sub’s own service agent needs permission to mint tokens as the service account you named, which means the Service Account Token Creator role on that account. The agent’s address is project-specific; read it from the IAM page for your project with Google-managed accounts shown, rather than copying an address out of a blog post.

Building it

1. Create the topic, then deploy the Cloud Run service with --no-allow-unauthenticated, an explicit --concurrency, an explicit --max-instances, and a --timeout above your worst-case model call.
2. Create a service account for push, grant it roles/run.invoker on the service, and grant the Pub/Sub service agent the Service Account Token Creator role on that service account.
3. Create the push subscription with --push-auth-service-account, an --ack-deadline sized to the call, and a dead letter topic with --max-delivery-attempts.
4. Publish a message and confirm one result row appears. Then publish one that sleeps past the ack deadline and confirm you can see the duplicate delivery in your own logs — you want to have watched this happen once deliberately.
5. Add the idempotency check keyed on messageId or your own job id, and confirm the duplicate now costs nothing.

gcloud run deploy inference-worker \
  --image europe-docker.pkg.dev/PROJECT/repo/worker:1 \
  --no-allow-unauthenticated --concurrency 4 --max-instances 20 --timeout 420

gcloud pubsub subscriptions create inference-push \
  --topic inference-jobs \
  --push-endpoint https://inference-worker-xxxx.europe-west1.run.app/ \
  --push-auth-service-account [email protected] \
  --ack-deadline 400 \
  --dead-letter-topic inference-dead \
  --max-delivery-attempts 5