Fixing a Timeout Calling a Model API From Inside a Locked-Down VPC

The service worked in staging and hangs in the private VPC. The request does not fail fast, it sits there until your client gives up. That symptom — silence rather than refusal — is itself the most useful piece of evidence you have, and it rules out about half of the things people check first.

Which timeout you actually have

A packet that reaches something which does not want it comes back as a TCP reset, and your client raises a connection-refused error almost immediately. A packet that is dropped — by a route that goes nowhere, by a security group, by a network ACL, or by a NAT device with no port to give it — produces nothing at all, and your client waits out its full connect timeout. So the first question is not “why is the model API slow” but which of these two you are looking at.

Establish it before you touch any configuration. From inside the workload’s own network namespace, not from your laptop:

# From the task, pod or instance itself.
# Separate DNS from connectivity: resolve first, then connect by name.
getent hosts api.openai.com
curl -sv --max-time 10 https://api.openai.com/v1/models -o /dev/null

If getent returns nothing, you have a resolution problem and not a routing problem, and the rest of this page does not apply — go to VPC DNS resolution for a model endpoint. If it resolves and curl hangs at Trying 104.18.x.x:443... with nothing after it, the SYN is leaving and no SYN-ACK is coming back. That is the case worth the ordering below. If instead you see the TLS handshake complete and then a stall, you have a read timeout, which is a different page entirely: the network is fine and the model is thinking.

1. The route table

Check this first because it is the cheapest check and the most common cause in a subnet somebody recently made private. A subnet is private because its route table has no 0.0.0.0/0 entry pointing at an internet gateway. If nothing replaced that entry, every packet destined for a public address is dropped by the VPC router with no reply.

SUBNET=subnet-0a1b2c3d4e5f67890

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=$SUBNET" \
  --query 'RouteTables[].Routes[?DestinationCidrBlock==`0.0.0.0/0`]'

You want exactly one default route, and its target tells you which world you are in. A nat- id means the subnet egresses through a NAT gateway; an igw- id in a subnet you believed was private means it is not private; and an empty array means there is no default route and the answer is already found. The other failure this catches is a NAT gateway that exists in the same private subnet it is meant to serve — a NAT gateway must live in a public subnet, because its own path out is an internet gateway.

One route-table trap deserves naming because it produces a timeout that moves around: subnets in a multi-AZ deployment often each get their own route table, and it is easy to point three of them at a NAT gateway and forget the fourth. The workload then fails only when the scheduler places it in one particular availability zone, which reads as an intermittent problem rather than a configuration one.

2. Security groups and network ACLs

Security groups are stateful, which is why they are second and not first: a security group that permits an outbound connection automatically permits the return traffic, so there is no separate inbound rule to get wrong. AWS documents that a newly created security group starts with an outbound rule allowing all traffic, so unless somebody deliberately replaced it, egress is open.

Somebody frequently has. A tightened baseline that allows egress only to an internal CIDR is the standard way a working service starts hanging on an external call after a security review. Read the actual rules rather than trusting the description:

aws ec2 describe-security-groups \
  --group-ids sg-0123456789abcdef0 \
  --query 'SecurityGroups[].IpPermissionsEgress'

If you need to write a rule rather than diagnose one, the minimal version is on security group rules for outbound HTTPS to a model provider.

Network ACLs are the opposite and this is where the subtle failure lives. They are stateless, they evaluate rules in numbered order, and the first match wins. An ACL that allows outbound TCP 443 but whose inbound rules only cover ports 80 and 443 will drop the response, because the response arrives on the ephemeral port your client chose, not on 443. The inbound rule you need is the ephemeral range — 1024–65535 — and its absence is invisible in an outbound rule review. Check both directions:

aws ec2 describe-network-acls \
  --filters "Name=association.subnet-id,Values=$SUBNET" \
  --query 'NetworkAcls[].Entries[].[RuleNumber,Egress,Protocol,PortRange,CidrBlock,RuleAction]' \
  --output table

3. NAT capacity and port allocation

This is the one that produces a timeout under load and never in a test, and it is the reason a model-serving workload hits it more than a typical web service does. A NAT gateway rewrites your source port, so it needs a free port per concurrent connection per destination. AWS documents that a NAT gateway supports up to 55,000 simultaneous connections to each unique destination, and that beyond that you get port allocation errors, visible as the ErrorPortAllocation CloudWatch metric on the gateway (AWS re:Post, resolving NAT gateway port allocation errors).

Read that limit carefully, because the per-destination part is what makes it bite here. A fleet of workers talking to thousands of different hosts spreads across many destination tuples. A fleet of inference workers all talking to one provider hostname on port 443 concentrates every connection into a single tuple, which is precisely the case the ceiling applies to. Long-lived streaming responses make it worse, because each held-open connection occupies its port for the whole generation rather than for a few hundred milliseconds.

The second metric to pull is IdleTimeoutCount. AWS documents that a NAT gateway drops connections idle for 350 seconds or more, and that a spike in this metric means the application is leaving connections open that the gateway has already discarded. The visible symptom is a request that hangs and then fails on a connection your client believed was healthy — a pooled keep-alive connection the NAT device stopped tracking. Both metrics come from the same namespace:

aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value=nat-0123456789abcdef0 \
  --start-time 2026-08-10T00:00:00Z \
  --end-time 2026-08-11T00:00:00Z \
  --period 300 \
  --statistics Sum

There are three real remedies and they are not equivalent.

• Set a connection-pool TTL below the idle timeout. Recycling idle connections before 350 seconds keeps your pool and the gateway’s translation table in agreement, and returns ports the gateway would otherwise hold.
• Cap concurrency deliberately. An unbounded worker pool against one provider hostname is what turns a per-destination ceiling into an outage. A bounded queue in front of the calls turns it into latency instead — see batching model requests through a queue.
• Add addresses or gateways. AWS announced in February 2023 that a NAT gateway can carry multiple IP addresses, up to eight, multiplying the per-destination ceiling accordingly (AWS, February 2023). Per-AZ gateways help too, and cut cross-AZ data charges as a side effect.

4. What is left after the first three

If the route exists, the filters allow it, and the NAT gateway is not exhausted, the remaining causes are narrow enough to enumerate. A transparent egress proxy or firewall appliance in the path that does TLS inspection will drop connections it cannot classify, and a model provider fronted by a CDN presents a rotating set of addresses that an appliance rule written against a fixed list will not match — which is the whole argument on allowlisting model provider IP ranges. An IPv6-enabled subnet whose only egress path is IPv4 will hang on any hostname with an AAAA record, because the client prefers IPv6 and the packets have nowhere to go; that one needs an egress-only internet gateway. And a VPC endpoint policy on an interface endpoint can deny a call that reached the right place, which fails at the application layer rather than the network one and usually returns an explicit authorization error rather than a hang.

Work the list in this order and each step eliminates the layer below it. The reason to resist starting at the NAT gateway — the most interesting cause — is that it is also the one that requires load to reproduce, so a morning spent there when the answer was a missing route in one availability zone is a morning that proves nothing.