Lambda Provisioned Concurrency for AI Endpoints

Provisioned concurrency is a number of execution environments that Lambda initialises ahead of time and keeps ready. AWS describes it as designed to make functions available with double-digit millisecond response times. It is also the most reliably over-bought Lambda feature there is, so this page spends as much time on verifying it as on setting it.

What it removes, and what it does not

A cold invocation has two phases: Init, where the runtime starts and your module-scope code runs, and Invoke, where your handler runs. Provisioned concurrency moves Init off the request path entirely — AWS documents that initialisation code runs during allocation, ahead of any request.

It does nothing at all to Invoke. For a function whose job is to call a model API, the handler is dominated by the model’s own latency, and no amount of provisioned concurrency touches that. This is the single most important thing to internalise before buying any: if your p99 is 4 seconds because generation takes 4 seconds, removing a 600 ms cold start changes p99 by nothing you will notice.

Where it does pay is when Init is genuinely large. A container image with a heavy dependency tree, an SDK client whose construction reads config and resolves credentials, a warm-up call to fetch a secret — all of that is Init work, and all of it happens on the first request of every new environment without provisioned concurrency. Read the Init Duration field in your logs before deciding; AWS reports it in a platform-initReport event in JSON log format at INFO level or above.

Configuring it

One constraint governs the whole setup: AWS documents that provisioned concurrency cannot be configured on $LATEST. It attaches to a published version or, better, an alias.

1. Publish a version and point an alias at it. The alias is what your trigger will invoke and what the configuration attaches to, so it survives every subsequent deployment.

   VERSION=$(aws lambda publish-version \
     --function-name model-caller --query Version --output text)
   
   aws lambda create-alias --function-name model-caller \
     --name LIVE --function-version "$VERSION"

2. Allocate. AWS’s configuring provisioned concurrency page shows the call and the response, which returns Status: IN_PROGRESS with Allocated ProvisionedConcurrentExecutions at zero — allocation is not instant.

   aws lambda put-provisioned-concurrency-config \
     --function-name model-caller \
     --qualifier LIVE \
     --provisioned-concurrent-executions 20

3. Repoint every trigger at the alias. This is the step that gets missed, and the symptom is paying for provisioned concurrency while still seeing cold starts — AWS calls it out specifically for API Gateway integrations still pointing at $LATEST.

   aws lambda get-provisioned-concurrency-config \
     --function-name model-caller --qualifier LIVE

How much to allocate: AWS gives the formula as concurrency = (average requests per second) × (average request duration in seconds), estimated from the Invocations and Duration metrics, plus a suggested 10% buffer. For model-calling functions the duration term dominates and is highly variable, so use a high percentile of duration rather than the mean — a function averaging 2 seconds but reaching 20 on long generations needs concurrency sized nearer the tail.

Two account-level constraints. You can configure up to your unreserved account concurrency minus 100, the remaining 100 being reserved for functions without reserved concurrency. And allocating to one function reduces what every other function in the account can scale to, whether or not the allocation is used.

Verifying it is actually being used

Two CloudWatch metrics answer this, and checking them costs nothing.

• ProvisionedConcurrencyInvocations — a non-zero value confirms invocations are landing on initialised environments. If this is zero while you are being billed, your triggers are pointing at the wrong qualifier and the money is doing nothing.
• ProvisionedConcurrencySpilloverInvocations — non-zero means all provisioned capacity was in use and some invocations took a cold start anyway. A steady trickle is fine and expected. A large fraction means the allocation is too small.

From inside the function, the AWS_LAMBDA_INITIALIZATION_TYPE environment variable is either provisioned-concurrency or on-demand, is immutable for the life of the environment, and can be emitted as a log field. That gives you per-request attribution without a metric query:

import os
INIT_TYPE = os.environ.get("AWS_LAMBDA_INITIALIZATION_TYPE", "unknown")

def handler(event, context):
    ...
    print(json.dumps({"init_type": INIT_TYPE, "request_id": context.aws_request_id}))

There is one ceiling AWS documents that is rarely repeated anywhere else, and it can invalidate an otherwise correct calculation: functions with provisioned concurrency have a maximum rate of 10 requests per second per unit of provisioned concurrency. A function configured with 100 units handles 1,000 requests per second; above that, cold starts occur regardless. For a short-duration, high-frequency function this rate ceiling binds long before the concurrency calculation does.

Autoscaling the allocation

A fixed allocation sized for peak is paid for at 3am. Application Auto Scaling handles both the scheduled and the reactive case; the resource ID is function:NAME:ALIAS and the scalable dimension is lambda:function:ProvisionedConcurrency.

aws application-autoscaling register-scalable-target \
  --service-namespace lambda \
  --resource-id function:model-caller:LIVE \
  --scalable-dimension lambda:function:ProvisionedConcurrency \
  --min-capacity 5 --max-capacity 100

aws application-autoscaling put-scaling-policy \
  --service-namespace lambda \
  --resource-id function:model-caller:LIVE \
  --scalable-dimension lambda:function:ProvisionedConcurrency \
  --policy-name pc-utilization \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration \
    '{ "TargetValue": 0.7, "PredefinedMetricSpecification": { "PredefinedMetricType": "LambdaProvisionedConcurrencyUtilization" }}'

AWS documents the accepted target range as 10% to 90%, and notes that the scale-in alarm fires at 90% of the target — so a target of 0.7 scales in below 63% utilisation. Three cautions from the same page, all of which cost money when ignored:

• Set an initial provisioned concurrency value before registering the scalable target. Without one, Application Auto Scaling may not scale the function properly.
• The ProvisionedConcurrencyUtilization metric is only emitted while the function is active. During idle periods the alarms enter INSUFFICIENT_DATA and cannot scale down — AWS says plainly that this “might lead to unexpected billing”. A scheduled scale-down covers the quiet window.
• Both alarms use the Average statistic by default and Application Auto Scaling needs the load sustained for at least three minutes, with three datapoints, before provisioning more. For bursty traffic AWS suggests the Maximum statistic instead.

Whether it is worth it for a model call

Work through it in this order and you will usually reach a cheaper answer than allocation.

1. Read Init Duration. If it is 100 ms, there is nothing here to buy. If it is 3 seconds, there is.
2. Shrink Init first. AWS’s own guidance cuts both ways: move initialisation out of the handler when you have provisioned concurrency, because it runs free at allocation time; but for on-demand functions, defer work you do not always need, so that the cold path is cheaper. A container image is often where the time is — see packaging a model call in a container image.
3. Try memory first. Raising memory raises CPU proportionally and speeds up Init as well as Invoke. It is often cheaper than provisioned concurrency and it helps both phases — Lambda memory size and AI workload performance has the mechanism.
4. Then allocate, and measure the same percentile. Compare p99 before and after on the same traffic. If it did not move, the cold start was not your problem — cold starts on a function calling an external API covers the case where the latency is in the call, not the container.