Skip to content
Multigrid

Operating AI infrastructure

Queues, infrastructure as code, CI, spend control, private networking and secrets — the parts that decide whether a working deployment stays working.

A model call is an ordinary network request with two unusual properties: it can take a minute, and running it twice costs money. Almost everything awkward about operating this kind of system comes from those two facts colliding with infrastructure that was designed on the assumption that requests are fast and retries are free. A queue that redelivers after thirty seconds is correct behaviour for a thumbnail resizer and a duplicate invoice for a generation. A retry policy that is prudent in front of a database is a spend multiplier in front of a provider.

These pages work at the level of the API, the CLI and the manifest, because that is the layer that survives a console redesign. Where a number is a vendor default rather than a property of the world, it is dated and attributed in the sentence, and the page tells you which quota page to check rather than asking you to trust it.

SQS to Lambda for Asynchronous Model Inference

Wiring an SQS queue to a Lambda worker that calls a model and writes the result to DynamoDB, with the timeout arithmetic that makes it survive a slow call.

11 min read

Pub/Sub Triggering a Cloud Run Worker for Model Calls

A push subscription delivering to a private Cloud Run service that calls a model, with the ack deadline and request timeout set so a slow call is not redelivered.

11 min read

Celery and Redis for Queued Inference on Kubernetes

Running a Celery worker and a Redis broker as Kubernetes workloads for model calls, with the prefetch and visibility-timeout settings that make slow tasks behave.

11 min read

An Azure Service Bus Queue for Buffering Model Requests

Putting a Service Bus queue between an intake function and a worker that calls a model, with lock renewal for calls that outlast the five-minute maximum.

11 min read

Setting an SQS Visibility Timeout for Slow Model Calls

How to pick, extend and monitor the visibility timeout when the consumer's work is a model call whose duration you do not control.

11 min read

Batching Requests From a Queue Before Calling a Model API

Draining a queue into small windows and combining the work into one provider request, where the provider's API actually supports combining.

12 min read

Fan-Out From SNS to Multiple Model Processing Queues

Publishing one event to an SNS topic that several SQS queues subscribe to, so summarisation, embedding and moderation run in parallel with independent retries.

11 min read

Google Cloud Tasks for Rate-Limited Model API Calls

Using a Cloud Tasks queue's dispatch rate and concurrency settings as the throttle that keeps your workers inside a provider's rate limit.

11 min read

RabbitMQ for Queued Model Inference on Kubernetes

Deploying RabbitMQ with the cluster operator and a consumer that pulls model jobs, with the acknowledgement timeout that catches slow consumers.

11 min read

Scaling Celery Workers by Queue Length on Kubernetes

Wiring a KEDA RabbitMQ or Redis scaler to a Celery worker Deployment so replica count tracks real backlog rather than CPU.

11 min read

Idempotency Keys for a Queued Model Request That Might Retry

Making an at-least-once redelivery cost nothing, using a conditional write that reserves the key before the model call rather than after it.

11 min read

Long Polling and Push Delivery for a Queue Feeding a Model Pipeline

What you actually trade between polling a queue and having it call you, when the consumer's work is a slow model request.

10 min read

A Terraform Module for a Bedrock-Backed Lambda Function

A reusable module that provisions the function, its execution role and the exact Bedrock invoke permissions together, including the region-scoped ARN most policies get wrong.

10 min read

Provisioning a Vertex AI Endpoint With Pulumi

Creating a Vertex AI endpoint in a Pulumi program, and handling the fact that the endpoint resource does not deploy a model into itself.

10 min read

Terraform State Locking for a Shared GPU Cluster Config

Setting up remote state with S3-native locking so two engineers resizing the same GPU node pool cannot corrupt each other's state.

9 min read

Managing Azure OpenAI Deployments With Terraform

Defining the Cognitive Services account and its model deployments as Terraform resources, including what the capacity field actually means.

10 min read

Pulumi for Cloudflare Workers AI Infrastructure

Provisioning a Worker and its AI and Vectorize bindings in Pulumi, including the index the provider has no resource for.

10 min read

A Terraform Module for an ECS GPU Task

A module that provisions the GPU task definition, the service and the capacity provider backing it, with the agent and AMI settings that make GPUs visible.

11 min read

Provisioning a GKE GPU Node Pool With Terraform

Defining a GKE node pool with an accelerator config and the driver installation block that GKE will not do for you by default.

10 min read

Terraform Import for a Manually Created Bedrock Resource

Bringing a console-created Bedrock guardrail or provisioned throughput under Terraform management without destroying and recreating it.

10 min read

Managing Secrets Manager Entries for Model API Keys With Terraform

Provisioning the secret and its read policy in Terraform while keeping the key value itself out of the state file.

10 min read

Pulumi Stacks for Separate Dev and Prod Model Endpoints

Splitting one Pulumi program into per-environment stacks so a dev model endpoint and a prod one cannot share a resource by accident.

10 min read

Adding a GPU Runner to a GitHub Actions Workflow

Registering a self-hosted GPU runner, labelling it so only the right jobs land on it, and the security constraint that decides whether you can do this at all.

10 min read

Self-Hosted GPU Runners for GitLab CI

Registering a GPU-equipped GitLab Runner with the current authentication-token flow and tagging jobs so only the ones that need a GPU reach it.

10 min read

Running Model Eval Tests on a GPU in CI

Wires a GPU-backed CI job that runs an evaluation suite and fails the build when a score drops below a committed baseline.

10 min read

Caching Model Weights Between CI Runs

Builds a CI cache keyed on the resolved model revision so weights are downloaded once rather than on every run.

9 min read

The Cost of a GPU CI Runner Per Build Minute

Works a per-build cost for a GPU CI job from published per-minute runner rates, and the break-even point against a self-hosted instance.

9 min read

Spinning Up a Spot GPU Instance as an Ephemeral CI Runner

Provisions a spot GPU instance that registers itself as a just-in-time runner, executes one job, and terminates.

10 min read

Setting a Hard Spend Cap on AWS Bedrock Usage

Combines an AWS Budgets action with an IAM deny policy so Bedrock invocation is revoked once a spend threshold is crossed.

10 min read

Tagging Cloud Resources to Attribute AI Spend by Team

Sets a consistent tagging scheme across Lambda, Bedrock and S3 so a cost report splits cleanly by team.

10 min read

Why an Idle GPU Node Pool Costs More Than It Looks Like

Works the monthly cost of a minimum-size GPU node pool sitting idle from published hourly rates, including the charges that are not the GPU.

9 min read

Cost Management Alerts for Container Apps Running Inference

Scopes an Azure Cost Management budget and alert to a Container Apps environment running model-calling workloads.

9 min read

Reading a GCP Billing Export to Find Vertex AI Spend by SKU

Queries the BigQuery billing export filtered to Vertex AI SKUs to produce a per-model cost breakdown.

10 min read

Right-Sizing a SageMaker Instance Type for Cost

Reads endpoint invocation and utilization metrics to pick a smaller instance type without breaching the latency SLA.

10 min read

Cleaning Up Idle Cloud Run Revisions That Still Bill

Finds old Cloud Run revisions still holding minimum instances and removes the ones no longer receiving traffic.

9 min read

Setting a Per-Request Cost Ceiling Before Calling a Model API

Estimates a request's worst-case token cost before sending it and rejects the request if it exceeds a configured ceiling.

10 min read

Handling Spot Interruption for a Fargate-Backed Batch Job

Run batch inference on Fargate Spot and survive the two-minute interruption warning, including the stopTimeout default that gives you thirty seconds rather than a hundred and twenty.

10 min read

Estimating Egress Cost for Streaming Model Responses at Scale

A per-million-request egress figure derived from server-sent-event framing overhead, AWS data-transfer rates and NAT gateway processing, with every assumption labelled.

10 min read

Setting Per-Environment Budgets for Dev, Staging and Prod Model Calls

Tag model spend by environment using Bedrock application inference profiles, then filter an AWS Budget on that tag so a staging leak alerts on its own.

10 min read

Why a Forgotten SageMaker Endpoint Is a Common Surprise Bill

The line item is an instance-hour charge for provisioned capacity that bills from InService to DeleteEndpoint, whether or not anything ever invoked it.

9 min read

Reaching a Model Provider From a Private Subnet With No Internet Gateway

Why a private subnet has no route to a public model API, the three exits available, and how to tell which one your case actually needs.

9 min read

Setting Up a NAT Gateway for Outbound Calls to a Model API

Provision a public NAT gateway and route table entry, confirm outbound HTTPS reaches a model provider, and understand the port-allocation ceiling before you hit it.

10 min read

PrivateLink to Reach a Model Endpoint Without a Public IP

Create an interface VPC endpoint for Bedrock so model traffic never crosses the public internet, including the DNS and security group prerequisites that cause most failures.

10 min read

Allowlisting a Model Provider's IP Ranges Through a Firewall

Where the major providers publish their IP ranges, how to turn a published list into a firewall rule that updates itself, and when to use a hostname rule instead.

10 min read

VPC Peering Between an Inference Service and a Vector Database

Peer two VPCs so an inference service reaches a managed vector database privately, and understand the CIDR, DNS and transitivity limits before you commit to it.

10 min read

Fixing DNS Resolution Failing for a Model Endpoint Inside a VPC

From getaddrinfo ENOTFOUND and EAI_AGAIN to the specific cause: a disabled VPC attribute, an unassociated private hosted zone, a suffix match returning NXDOMAIN, or the 1,024 packet-per-second resolver limit.

10 min read

An Egress-Only Internet Gateway for a Model-Calling Service

Provision an egress-only internet gateway so an IPv6 workload can call out but never accept inbound, and handle the case where the provider has no AAAA record.

9 min read

Fixing a Timeout Calling a Model API From Inside a Locked-Down VPC

A connect timeout from a private subnet has four possible causes; this is the order that finds them fastest, ending at NAT port exhaustion.

10 min read

Security Group Rules for Outbound HTTPS to a Model Provider

Replace the default allow-all egress rule with a single HTTPS rule, and understand why the destination cannot usefully be narrower than a CIDR.

9 min read

Private Service Connect to Vertex AI From Another VPC

Create a Private Service Connect endpoint for Google APIs, point DNS at it, and reach Vertex AI from a VPC with no external addresses.

10 min read

Automating Model API Key Rotation With a Secrets Manager Trigger

Wire a rotation Lambda against a provider API key, including the part the database templates hide: a provider key cannot be changed in place.

11 min read

Injecting Secrets Into a Kubernetes Pod From a Cloud Secret Manager

Install the External Secrets Operator, sync a provider API key from AWS Secrets Manager into a native Secret, and make the pod actually pick up a change.

10 min read

Sealed Secrets for a GitOps-Managed Model API Key

Encrypt a provider API key so the ciphertext can live in git, and understand the scope rules and key custody that decide whether it stays safe.

10 min read

Fixing an API Key Leaked Into Serverless Function Logs

You found a provider key in CloudWatch. Revoke first, then find the line that printed it — usually an error object, not a deliberate log statement.

10 min read

Using HashiCorp Vault for Model Provider Credentials

Store a provider key in Vault's KV v2 engine, authenticate a workload with the Kubernetes auth method, and be clear about what is and is not short-lived.

11 min read

Scoping a Secrets Manager Read Policy to One Secret

Write an IAM policy that reads one named secret, including the six-character ARN suffix that makes the obvious wildcard wrong.

11 min read

Why a Secret Baked Into a Docker Image Never Really Goes Away

Deleting a credential in a later layer hides it from the filesystem view and leaves it intact in the layer below, where anyone with the image can read it.

10 min read

Provisioning a GPU Node Pool With Autoscaling on Amazon EKS

Create a managed node group on an accelerated AMI, install the device plugin the AMI omits, and add the tags that let the autoscaler scale it from zero.

11 min read

Other topics