Skip to content
Multigrid

Running AI workloads on AWS

Bedrock, SageMaker and the serverless plumbing around them — what each service actually does to a model call, and which of them you do not need.

AWS sells at least four different things that will run a language model for you, and the documentation for each is written as though the other three do not exist. Bedrock is a managed API over models you do not host. SageMaker real-time endpoints are instances you rent by the second with a container on them. SageMaker serverless is a different service wearing the same API. And underneath all of it is the ordinary AWS machinery — IAM, S3, quotas, Marketplace subscriptions — which is where most of the failures actually come from.

These pages are written against the API, the CLI and the resource shapes rather than the console, because consoles get redesigned and CreateModelInvocationJob does not. Where a number moves — a quota, a price, a region list — the page says so and points at the page that holds the current value, rather than pretending to be it.

Calling Claude on Bedrock With the Converse API

The Converse request and response shape end to end — messages, inferenceConfig, tool use and every stopReason value — instead of the per-provider InvokeModel body.

10 min read

Setting Up Cross-Region Inference on Bedrock

How an inference profile changes which quota your traffic is measured against, and the IAM statement people miss when they switch to one.

9 min read

Fixing AccessDeniedException When You Request Bedrock Model Access

The three causes of a 403 on your first Bedrock call now that model access is granted automatically, and the one API call that tells you which you have.

9 min read

Bedrock Provisioned Throughput: Working Out Whether It Pays

The break-even calculation for Provisioned Throughput, derived as a formula with every assumption labelled, plus the two numbers only your AWS account team can give you.

9 min read

Running a Batch Inference Job on Bedrock

A batch job against an S3 JSONL manifest, from input record format through to reading the output records and manifest.json.out.

10 min read

Building a Bedrock Knowledge Base Over S3 Documents

Creating a knowledge base over an S3 prefix, with the chunking configuration decided deliberately and the ingestion job actually run.

10 min read

Building a Denied-Topics Policy With Bedrock Guardrails

Writing a topic policy, versioning it, and testing it against real prompts with ApplyGuardrail before it ever touches a model call.

10 min read

Wiring an Action Group to Lambda With Bedrock Agents

The exact event contract between a Bedrock agent action group and its Lambda function, and how to read a trace — for the people who already own one.

10 min read

Fixing ThrottlingException on Amazon Bedrock

Which of the two per-model quotas you hit, why a 429 is not a 503, and the four fixes in the order they are worth trying.

9 min read

Reading Amazon Bedrock’s On-Demand Token Rates

How the Bedrock price list is structured, which four things change a rate, and how to pull the current numbers programmatically instead of copying them.

9 min read

Deploying a Container to a SageMaker Real-Time Endpoint

The three-resource deployment from container image to a signed invocation, and the container contract that decides whether it comes up at all.

10 min read

SageMaker Serverless Inference: Setup and Its Real Limits

The six memory sizes, the concurrency arithmetic, and the documented exclusions — no GPU, no VPC, no multi-model — that decide this for most workloads.

9 min read

Building a Multi-Model Endpoint on SageMaker

Put many model artifacts behind one SageMaker endpoint with Mode MultiModel, and understand what dynamic loading costs the first caller of a cold model.

10 min read

Autoscaling a SageMaker Endpoint

Register an endpoint variant with Application Auto Scaling and put a target-tracking policy on InvocationsPerInstance, including why that metric reacts slowly.

9 min read

Async Inference on SageMaker for Long-Running Requests

Queue a large payload through S3, get a 202 with an output location, and poll or subscribe for the result when the work outlasts a synchronous call.

10 min read

Deploying a JumpStart Foundation Model on SageMaker

Deploy a JumpStart model card to an endpoint, accept the EULA correctly, and read back the image, instance type and payload format JumpStart chose for you.

9 min read

Why a SageMaker Endpoint Costs More Than Expected

A real-time endpoint bills instance-hours from InService to deletion regardless of traffic; here is how to confirm that is what happened and what to do about it.

9 min read

Fixing ModelError on a SageMaker Endpoint

ModelError means your container returned a 4xx or 5xx, and the exception carries the log stream that explains it; here is how to read both.

9 min read

Streaming Responses From an AWS Lambda Function URL

Set a function URL to RESPONSE_STREAM, write a streamifyResponse handler, and read the bytes as they arrive instead of waiting for the whole payload.

10 min read

Packaging a Model Call in a Lambda Container Image

Build, test and push a Lambda container image sized for a model SDK's dependencies, using the AWS Python base image and the runtime interface emulator.

10 min read

Lambda Provisioned Concurrency for AI Endpoints

Allocate pre-initialised execution environments to an alias, verify they are being used, and autoscale them — plus when the cold start was not the problem.

10 min read

Fixing “Task Timed Out After N Seconds” on AWS Lambda

The message means the function hit its configured timeout, which is rarely the same problem as the model API call that was slow inside it.

9 min read

Lambda Memory Size and AI Workload Performance

Memory is the CPU dial: raising it raises the vCPU share proportionally, which matters for some parts of a model-calling function and not at all for others.

9 min read

Running a GPU Task on Amazon ECS

Writing a task definition that reserves a physical GPU, and getting the cluster underneath it into a state where ECS will actually place the task.

9 min read

Deploying Inference on AWS Fargate

Running an inference container on Fargate with no instances to manage, and sizing it honestly against the CPU and memory combinations Fargate actually offers.

9 min read

Autoscaling an ECS Service by Queue Depth

Building a backlog-per-task target tracking policy from an SQS queue and an ECS running-task count, and deriving the target value instead of guessing it.

9 min read

WebSocket Streaming Through AWS API Gateway

Relaying a streamed model response to a browser over an API Gateway WebSocket API, and handling the connection lifecycle that the relay depends on.

10 min read

Throttling Limits on AWS API Gateway

The four throttle levels API Gateway applies, the order it applies them in, and how to tell which one returned the 429 your AI endpoint just produced.

9 min read

Private Integration Between API Gateway and a VPC Model Endpoint

Wiring a VPC link so a public API Gateway route reaches a load-balanced model server that has no public IP, without a NAT gateway or a public listener.

9 min read

Human-in-the-Loop Approval in a Step Functions Workflow

Pausing a model workflow on a task token until a person signs off, and bounding the wait so a forgotten approval does not leave an execution open for a year.

9 min read

Retries and Error Handling in AWS Step Functions for Model Calls

Configuring per-state retriers and Catch branches around a flaky model API, including which errors are worth retrying and which will never succeed.

10 min read

Triggering an Embedding Pipeline on S3 Object Upload

Wiring an S3 event notification to a function that chunks and embeds a newly uploaded document, without the self-trigger loop or duplicate embeddings.

9 min read

Generating Presigned S3 URLs for a RAG Upload Flow

Issuing a time-limited upload URL so a browser can push a document straight into an ingestion bucket, and constraining what it is allowed to put there.

9 min read

Amazon S3 Vectors for Storing Embeddings

Creating a vector bucket and index with the s3vectors API, writing embeddings into it, and running a filtered similarity query.

9 min read

Writing an IAM Role Scoped to a Single Bedrock Model

A resource-scoped Bedrock policy that permits one named model, and the second statement an inference profile forces you to add.

10 min read

Fixing AccessDeniedException Calling Bedrock From Lambda

The exception has two distinct wordings with two unrelated causes; reading which one you have narrows the fix to a single check.

10 min read

Setting Up a VPC Endpoint for Amazon Bedrock

Creating an interface endpoint so Bedrock calls never traverse the internet, and the DNS resolution that proves it.

10 min read

Storing a Model Provider API Key in AWS Secrets Manager

Putting a provider key in Secrets Manager, reading it from Lambda without the SDK, and rotating it without touching the function.

9 min read

Setting a Bedrock Spend Alert With AWS Budgets

A cost budget filtered to Bedrock, with forecast and actual thresholds, and an honest account of how late the alert arrives.

9 min read

Reading Bedrock Usage From AWS Cost Explorer

Separating Bedrock spend from the rest of an AWS bill, and decoding the usage-type string that says which model and which token type you paid for.

9 min read

When SageMaker Makes Sense Instead of a Hosted Model API

The choice is not about features; it is about whether you need control of the serving process, and whether you can keep an instance busy.

10 min read

Why the Model API Call, Not Lambda Init, Dominates a Cold Path

Where the time actually goes in a Lambda that calls a model provider, and why optimising init is usually optimising the smaller number.

9 min read

AWS Batch for Long-Running Model Jobs

Submitting a containerised model job to a compute environment sized for work that outlasts Lambda's fifteen-minute ceiling.

10 min read

An EventBridge Rule That Triggers a Scheduled Model Pipeline

Wiring a cron schedule to a batch model job, and handling the overlap that a scheduler will never handle for you.

9 min read

Cross-Account Access to a Shared Bedrock Model

Letting a second AWS account invoke a model held by a first, using role assumption because a foundation model has no resource policy to attach.

9 min read

Other topics