VPC Service Controls Around Vertex AI

IAM answers who may call Vertex AI. A service perimeter answers a different question: where the data may go afterwards. They are independent controls and a perimeter is the one that survives a stolen credential.

What a perimeter actually blocks

VPC Service Controls draws a boundary around a set of projects and declares a set of Google APIs restricted within it. Requests to a restricted service that originate outside the perimeter are refused, and — this is the half people underrate — requests from inside the perimeter to a resource of that service outside it are refused too.

That second direction is the exfiltration control. A compromised service account key with roles/aiplatform.user is useless from the internet if the perimeter refuses off-perimeter callers. And a process running legitimately inside the perimeter cannot write your prompt corpus to a Cloud Storage bucket in an attacker’s project, because that bucket is outside the perimeter and Cloud Storage is restricted.

Google’s supported products list confirms the Vertex AI API can be protected by VPC Service Controls, under the service name aiplatform.googleapis.com. What a perimeter does not do is any of IAM’s job: inside the perimeter, a principal with too many permissions still has too many permissions.

Choosing what goes inside

The most common way to make a perimeter that fails is to restrict aiplatform.googleapis.com and nothing else. A Vertex AI workflow almost never touches only Vertex AI, and every adjacent service you leave unrestricted is a path out of the boundary you just drew.

• aiplatform.googleapis.com — the API itself. Training, endpoints, batch prediction, the generative surface.
• storage.googleapis.com — where training data, model artifacts and batch inputs and outputs live. Leaving this out is the single largest hole, because it is the natural egress route for everything Vertex AI touches.
• bigquery.googleapis.com — and, if you use remote models, bigqueryconnection.googleapis.com alongside it, since the connection is a separate service.
• artifactregistry.googleapis.com — custom containers are pulled from somewhere, and an unrestricted registry is both an ingress path for code and an egress path for anything a build can write into a layer.
• logging.googleapis.com and monitoring.googleapis.com — logs contain prompts far more often than anyone intends. Related: what to log from an LLM call.

Include the projects, not just the services. A perimeter is a set of project resources plus a set of restricted services, and a project you forgot is a project that can read across.

Build it in dry-run first

Do not create an enforcing perimeter around a working system. Google documents dry-run perimeters as logging violations as though the services were protected, without preventing access — which is exactly the tool for finding out what you are about to break.

1. Find your organisation’s access policy, which the perimeter hangs off:

   gcloud access-context-manager policies list \
     --organization=ORGANIZATION_ID --format='value(name)'

2. Create the perimeter in dry-run mode. Project resources are named by number, not by ID:

   gcloud access-context-manager perimeters dry-run create ai_perimeter \
     --policy=POLICY_NUMBER \
     --perimeter-title="Vertex AI production" \
     --perimeter-type=regular \
     --perimeter-resources=projects/123456789012,projects/210987654321 \
     --perimeter-restricted-services=aiplatform.googleapis.com,\
   storage.googleapis.com,bigquery.googleapis.com,\
   bigqueryconnection.googleapis.com,artifactregistry.googleapis.com

3. Let it run for at least a full business cycle. A week catches the daily jobs; a month catches the monthly retrain, the quarterly audit export and the one analyst who works from a different network.
4. Read the violations out of the audit logs. Dry-run violations appear with the perimeter marked as dry-run, so they are separable from real denials:

   gcloud logging read \
     'protoPayload.metadata.dryRun="true"
      protoPayload.metadata."@type"="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"' \
     --limit=100 --freshness=7d \
     --format='table(protoPayload.methodName,
                     protoPayload.authenticationInfo.principalEmail,
                     protoPayload.metadata.violationReason)'

Every line in that output is something that will stop working. Decide for each one whether it is a legitimate caller that needs an access level, a project that should be inside the perimeter, or exactly the kind of access the perimeter exists to stop.

Enforcing, and letting the right callers in

Legitimate callers from outside — a CI runner, an on-call laptop on a corporate VPN — get in through an access level, which is a condition on the caller rather than a hole in the perimeter. Access levels can match on IP range, device policy, or identity.

# ci-egress.yaml
- ipSubnetworks:
  - 203.0.113.0/24

gcloud access-context-manager levels create ci_network \
  --policy=POLICY_NUMBER \
  --title="CI egress range" \
  --basic-level-spec=ci-egress.yaml

gcloud access-context-manager perimeters dry-run update ai_perimeter \
  --policy=POLICY_NUMBER \
  --add-access-levels=ci_network

# only when the violation log is clean:
gcloud access-context-manager perimeters dry-run enforce ai_perimeter \
  --policy=POLICY_NUMBER

An IP-based access level is the weakest of the three, because an IP range is not an identity. Prefer an identity-scoped ingress rule where the caller is a known service account, and treat an IP subnet as a transitional measure rather than the design.

What breaks, and how it looks when it does

A perimeter denial does not look like a permission error, which is why the first one costs an afternoon. The API returns a PERMISSION_DENIED with a request identifier and a message about VPC Service Controls, and it does so regardless of how much IAM the caller has — granting Owner changes nothing. The audit log entry carries a violationReason, and that field is the actual diagnosis; the client-side message is deliberately thin because a detailed one would leak the perimeter’s shape.

Three things reliably break on the day of enforcement. Anything reached from a machine with a public IP over the ordinary googleapis.com endpoints, which is why a perimeter is normally deployed alongside Private Google Access and the restricted VIP. Cross-project reads that nobody documented, particularly a shared Cloud Storage bucket owned by a data team in a project outside the list. And any managed service that calls Vertex AI on your behalf using a Google-managed service agent, where the agent’s own project is outside the perimeter — those show up in the dry-run log as principals you do not recognise, and they are worth reading carefully rather than adding blanket exceptions for.

The debugging loop for a live denial is worth knowing before you need it. The client-side error carries a unique identifier, and that identifier appears in the corresponding audit log entry — so the workflow is to capture the string from whoever hit the error and look it up, rather than to reason about what might have been blocked:

gcloud logging read \
  'protoPayload.metadata."@type"="type.googleapis.com/google.cloud.audit.VpcServiceControlAuditMetadata"
   protoPayload.metadata.vpcServiceControlsUniqueId="UNIQUE_ID_FROM_ERROR"' \
  --limit=1 --format=json

That entry names the principal, the method, the perimeter and the violation type, which between them tell you whether the fix is an access level, a project added to the perimeter, or nothing at all. Resist adding an ingress rule as the reflex: about half of first-week denials are a workload that genuinely should not have been reaching across, and an exception written in a hurry becomes permanent.

Keep the dry-run configuration after enforcing, too. Perimeter changes can be tested the same way, and a change to a live perimeter is exactly as capable of taking production down as the first enforcement was.