Private Google Access for Vertex AI Endpoints

Removing a VM’s external IP is the easiest security win in a GCP network and it immediately breaks every call to aiplatform.googleapis.com. Private Google Access fixes that without giving the VM a route to the internet, and it does so with three pieces that must all be present.

Why a private VM cannot call the API

Google Cloud APIs are reached over public IP addresses. A VM with an external IP, or behind Cloud NAT, has a path to them like any internet host. Strip the external IP and remove the NAT, and the VM has no route to any public address — including Google’s own. The symptom is a connection timeout on the SDK call rather than an authentication error, which sends people to check credentials for an hour.

Private Google Access changes that at the subnet level. When it is on, traffic from instances in that subnet destined for Google API addresses is delivered through Google’s network rather than requiring an internet path. The VM still has no external IP and still cannot reach a third-party host; what it gains is one specific destination class.

For inference work this is close to the default correct posture. An instance whose only outbound need is Vertex AI, Cloud Storage and Logging has no business being able to reach the open internet, and removing that ability removes an entire category of exfiltration and dependency-confusion risk in one setting.

Enabling it on the subnet

It is a property of the subnet, not of the VM, and it can be turned on for a live subnet without recreating anything:

gcloud compute networks subnets update inference-subnet \
  --region=us-central1 \
  --enable-private-ip-google-access

# confirm
gcloud compute networks subnets describe inference-subnet \
  --region=us-central1 \
  --format='value(privateIpGoogleAccess)'

For a new subnet the same flag applies at creation time. Note the scope carefully: every instance in that subnet gains the access, and instances in a sibling subnet of the same VPC do not. A workload that “sometimes works” across a fleet is usually a fleet spread over two subnets where only one has the flag.

The route, which is the part people forget

Enabling the flag is necessary and not sufficient. The VM still needs a route toward the destination, and in a network where you deleted the default internet gateway route as part of locking things down, there is none. Add a route for the specific Google address range you intend to use, with the default gateway as next hop — the gateway is the route target even though there is no internet access, because the delivery happens inside Google’s network.

gcloud compute routes create restricted-googleapis \
  --network=inference-vpc \
  --destination-range=199.36.153.4/30 \
  --next-hop-gateway=default-internet-gateway

And the firewall has to permit it. A VPC with a default-deny egress policy blocks this as readily as it blocks anything else:

gcloud compute firewall-rules create allow-restricted-googleapis \
  --network=inference-vpc \
  --direction=EGRESS \
  --action=ALLOW \
  --rules=tcp:443 \
  --destination-ranges=199.36.153.4/30 \
  --priority=1000

DNS: private versus restricted

Google publishes two special domains for this, and choosing between them is the real decision on this page. Per Google’s Private Google Access configuration documentation, private.googleapis.com resolves to 199.36.153.8/30 and restricted.googleapis.com resolves to 199.36.153.4/30.

• private.googleapis.com reaches most Google APIs, including ones that VPC Service Controls does not support. Use it when you want private connectivity and are not running a perimeter.
• restricted.googleapis.com reaches only APIs that support VPC Service Controls. That restriction is the feature: a workload pointed at it cannot reach an unsupported API even by accident, which closes the gap where a perimeter protects the services you listed and an unlisted service quietly does not.

If you are running the perimeter from VPC Service Controls around Vertex AI, use the restricted VIP. The two go together and using the private one with a perimeter leaves a path the perimeter cannot see.

Then override DNS so that aiplatform.googleapis.com and its regional forms resolve to those addresses, using a private Cloud DNS zone for googleapis.com:

gcloud dns managed-zones create googleapis-private \
  --dns-name=googleapis.com. \
  --networks=inference-vpc \
  --visibility=private \
  --description="Restricted VIP override"

gcloud dns record-sets create restricted.googleapis.com. \
  --zone=googleapis-private --type=A --ttl=300 \
  --rrdatas=199.36.153.4,199.36.153.5,199.36.153.6,199.36.153.7

gcloud dns record-sets create '*.googleapis.com.' \
  --zone=googleapis-private --type=CNAME --ttl=300 \
  --rrdatas=restricted.googleapis.com.

The wildcard CNAME is what makes this work for Vertex AI specifically. Vertex AI is called at regional hostnames such as us-central1-aiplatform.googleapis.com, and pinning only the apex would leave those resolving publicly. The wildcard catches every regional endpoint, in every region, without you having to enumerate them.

Verifying it from the VM

# on the VM, with no external IP
dig +short us-central1-aiplatform.googleapis.com
# expect one of 199.36.153.4 .. .7 via restricted.googleapis.com

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/endpoints

# and confirm there is genuinely no general internet path
curl -sS --max-time 5 https://example.com || echo "no internet — correct"

If dig returns a public Google address, the DNS zone is not attached to this VPC or the wildcard is missing. If DNS is right and curl hangs, it is the route or the egress firewall rule. If both are right and you get a 403 mentioning VPC Service Controls, the network is working correctly and the perimeter is doing its job — a different problem, and a much better one to have.

One consequence to plan for: with the restricted VIP and no NAT, the VM cannot install packages from PyPI or pull an image from a public registry. Mirror what you need into Artifact Registry, which is reachable through the same path, and treat that as part of the design rather than as a thing to discover during an incident.

What it does not cover

Private Google Access is a subnet property that applies to traffic from instances with internal IP addresses in that subnet. Three adjacent cases look like they should be covered and are not, and each has a different answer.

Serverless workloads. A Cloud Run service or a 2nd gen function does not sit in your subnet by default; its egress leaves through Google’s infrastructure and the subnet flag has nothing to apply to. If you need those calls to originate inside your VPC — because a perimeter, a firewall rule or an on-premises destination requires it — you attach the service to the network explicitly with Direct VPC egress or a Serverless VPC Access connector, and route its egress through that. Only then does the subnet’s Private Google Access setting govern the traffic.

On-premises hosts over Cloud VPN or Interconnect. Google supports this, but it is a separate configuration rather than a side effect of the subnet flag. Your on-premises network needs a route to the chosen VIP range advertised over the tunnel by a Cloud Router, and it needs DNS resolution that returns those addresses — typically an on-premises forwarder pointed at a Cloud DNS inbound forwarding entry, so the same wildcard override applies to hosts that are not in the VPC at all. Miss the DNS half and the route works while every lookup still resolves to a public address the route does not cover.

A stable private IP of your own. Private Google Access gives you Google’s VIP ranges, which are shared and outside your address space. Where you need the API to answer on an address you allocated — because an on-premises firewall keys on it, or because 199.36.153.4/30 overlaps something you already use — Private Service Connect is the mechanism that gives you an endpoint with an IP from your own subnet. It is more setup and it is the only option that puts you in control of the address.

A fourth case is worth naming because it is a security assumption rather than a connectivity one. Private Google Access reaches all projects’ resources for the services it exposes, not only yours. A VM with no external IP can still authenticate to a bucket in an unrelated project if it holds a credential for one. That is the gap a service perimeter closes, which is why VPC Service Controls around Vertex AI is the companion to this page rather than an alternative to it.