Fixing DNS Resolution Failing for a Model Endpoint Inside a VPC

Name resolution failures inside a VPC produce four or five distinct error strings and people treat them as one problem. They are not: a permanent failure, an intermittent failure and an NXDOMAIN have different causes and different fixes, and the string in your logs already narrows it to one or two.

The errors, and what each one rules out

# Python / botocore — resolution failed outright
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL:
  "https://bedrock-runtime.us-east-1.amazonaws.com/model/.../converse"
socket.gaierror: [Errno -2] Name or service not known
socket.gaierror: [Errno -3] Temporary failure in name resolution

# Node
Error: getaddrinfo ENOTFOUND api.anthropic.com
Error: getaddrinfo EAI_AGAIN api.anthropic.com

# shell
curl: (6) Could not resolve host: api.anthropic.com
;; connection timed out; no servers could be reached

Read them like this:

• ENOTFOUND / Errno -2 / curl: (6) — a resolver answered and said the name does not exist. Something is resolving; it just returned NXDOMAIN. Go to cause 2.
• EAI_AGAIN / Errno -3 / dig timing out — no answer came back at all. Either nothing is listening where the client is asking, or the query was throttled. Causes 1, 3 and 4.
• Resolution succeeds and the connection times out — not a DNS problem. That is routing or a security group; see reaching a model provider from a private subnet.

The one command that separates these definitively, run from inside the subnet:

# ask the VPC resolver directly (VPC CIDR base + 2)
dig +short bedrock-runtime.us-east-1.amazonaws.com @10.0.0.2

# and the link-local alias, which is identical but reachable from any CIDR
dig +short api.anthropic.com @169.254.169.253

Cause 1: the VPC DNS attributes

Two boolean attributes on the VPC decide whether the Amazon-provided resolver exists for your instances at all: enableDnsSupport and enableDnsHostnames. Route 53 documents that to use private hosted zones you must set both to true — see AWS on considerations for private hosted zones — and AWS PrivateLink documents the same requirement for private DNS on an interface endpoint.

aws ec2 describe-vpc-attribute --vpc-id vpc-0abc --attribute enableDnsSupport
aws ec2 describe-vpc-attribute --vpc-id vpc-0abc --attribute enableDnsHostnames

aws ec2 modify-vpc-attribute --vpc-id vpc-0abc --enable-dns-support
aws ec2 modify-vpc-attribute --vpc-id vpc-0abc --enable-dns-hostnames

A default VPC has both enabled. A VPC created by a Terraform module, CloudFormation template or CDK construct may not, because the underlying CreateVpc API does not enable enableDnsHostnames by default. This is why the failure overwhelmingly shows up in a purpose-built VPC and never in the one somebody tested in.

With enableDnsSupport false, the resolver at the base of the VPC range plus two does not answer, which is a timeout rather than an NXDOMAIN — the EAI_AGAIN signature. Note that flipping it does not require a restart of anything, but a long-running process that has already cached a negative result may need one.

Cause 2: a private hosted zone swallowing the name

This is the cause that produces the most confusing outage, because the name being resolved is a perfectly ordinary public one and the answer is a confident “does not exist”.

Route 53 resolves against private hosted zones associated with the VPC before it goes to public DNS, matching on the most specific suffix. And AWS states the consequence explicitly: if there is a matching private hosted zone but no record matching the name and type, “VPC Resolver doesn’t forward the request to a public DNS resolver. Instead, it returns NXDOMAIN.”

So a private hosted zone for example.com, created for internal service discovery, captures every query for every subdomain of example.com — including models.example.com, which lives in public DNS and which nobody thought to copy into the private zone. It fails with ENOTFOUND, permanently, while the same name resolves fine from a laptop. The fix is to add the record to the private zone, or to narrow the zone to the subdomain it was actually meant to serve.

# which private zones could be capturing this name?
aws route53 list-hosted-zones-by-vpc --vpc-id vpc-0abc --vpc-region us-east-1

# associate a zone that exists but was never attached to this VPC
aws route53 associate-vpc-with-hosted-zone \
  --hosted-zone-id Z0123456789ABCDEFGHIJ \
  --vpc VPCRegion=us-east-1,VPCId=vpc-0abc

The mirror image of this is a private hosted zone that exists but was never associated with the VPC doing the querying — common when the zone lives in a shared networking account. Then an internal model endpoint name resolves in one account and not another, which people reliably misdiagnose as a permissions problem.

One more precedence rule worth holding: AWS documents that if you have both a private hosted zone and a Route 53 Resolver rule for the same domain name, the Resolver rule wins. A forwarding rule sending example.com to an on-premises resolver silently overrides the private zone you have been editing.

Cause 3: a custom DHCP option set

A VPC with a custom DHCP option set hands your instances the DNS servers named in that set instead of the Amazon resolver. If those are on-premises servers reached over a VPN or Direct Connect, then every name resolution in the VPC now depends on that link, and a link problem presents as a DNS problem.

aws ec2 describe-vpcs --vpc-ids vpc-0abc \
  --query 'Vpcs[].DhcpOptionsId' --output text

aws ec2 describe-dhcp-options --dhcp-options-ids dopt-0abc \
  --query 'DhcpOptions[].DhcpConfigurations'

If the answer is anything other than AmazonProvidedDNS, those servers must be able to answer for everything you need — including AWS service endpoints and any private hosted zone. AWS’s guidance for custom DNS servers is that they must forward private queries to the Amazon-provided DNS server for the VPC, at “the IP address at the base of the VPC network range plus two” — so for a 10.0.0.0/16 VPC, 10.0.0.2. A custom resolver that does not forward is the reason an interface endpoint’s private DNS name resolves from one subnet and not another.

Cause 4: the 1,024 packet-per-second limit

If resolution works most of the time and fails under load, stop looking at configuration. AWS documents a hard quota: “Each EC2 instance can send 1024 packets per second per network interface to Route 53 Resolver” — the .2 address and 169.254.169.253 — and states plainly that “this quota cannot be increased.” It is on the Amazon VPC quotas page.

A high-concurrency inference worker is a good way to reach it without meaning to. Every new outbound connection is a resolution, and an SDK configured with a short DNS cache — or a runtime that does not cache at all, which is the default in several Python HTTP stacks — turns a few thousand requests per second into a few thousand DNS queries per second on one network interface. The excess is dropped, and a dropped query is EAI_AGAIN: intermittent, load-correlated, and impossible to reproduce in a quiet environment.

• Cache. Run a local caching resolver on the host, or set the JVM’s networkaddress.cache.ttl and its Node or Python equivalents to something above zero. This is usually the entire fix.
• Reuse connections. A pooled HTTP client resolves once and reuses the socket. Creating an SDK client per request resolves per request.
• Spread across interfaces. The quota is per network interface, so more, smaller tasks hit it later than one large one — a reason the failure appears after a consolidation rather than after a growth.

Because the quota is not adjustable, this is one of the few AWS limits where the only answer is to make fewer queries. Confirm it before you rebuild anything: if the failure rate tracks request rate rather than time of day or deployment, it is this.