Rate Limiting an AI Endpoint You Expose Publicly

An AI endpoint without a limiter is a form on the internet that spends your money. Unlike a normal API, where abuse costs you CPU you already paid for, every request here has a direct marginal cost — and the cost per request varies by two orders of magnitude, which is why counting requests is not enough.

The failure, priced

Start with the endpoint as it usually ships. It authenticates nothing and limits nothing.

// app/api/summarise/route.ts — the version that gets found
export async function POST(request: Request) {
  const { text } = await request.json();

  const res = await fetch("https://api.multigrid.ai/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + process.env.LLM_API_KEY,
    },
    body: JSON.stringify({
      model: "openai/gpt-4o",
      messages: [{ role: "user", content: "Summarise: " + text }],
    }),
  });

  return Response.json(await res.json());
}

Now price the abuse, with the arithmetic in front of you. Assume a model at $2.50 per million input tokens and $10.00 per million output tokens — substitute your own figures; the structure is what matters.

One abusive request, using the endpoint as designed but at maximum size:

  input   100,000 tokens × $2.50 / 1,000,000  =  $0.250
  output    4,000 tokens × $10.00 / 1,000,000 =  $0.040
                                                 -------
  per request                                     $0.290

A single script, 10 requests per second, unattended for one hour:

  10 × 3600 = 36,000 requests × $0.29         =  $10,440

Left running overnight, eight hours:                $83,520

And there is no cap on max_tokens in that handler, no ceiling on the
input length, and no way to attribute a single dollar of it to anyone.

Nothing exotic is required for this. The endpoint is discoverable in the network tab of your own site, and the request is a curl command. The three missing controls are an input ceiling, a max_tokens ceiling, and a limiter — and the first two are two lines each.

// The two lines that bound a single request, before any limiter exists.
if (typeof text !== "string" || text.length > 20_000) {
  return Response.json({ error: "too long" }, { status: 400 });
}
// ... and in the model request body:
max_tokens: 500,

Requests are the wrong unit

A conventional limiter counts requests. For an AI endpoint that is a weak proxy for what you are trying to bound, because the requests are not comparable.

Two users, both inside a limit of 20 requests per minute:

  User A: 20 requests × 500 input tokens, 100 output tokens
          = 10,000 in, 2,000 out       ≈ $0.045

  User B: 20 requests × 100,000 input tokens, 4,000 output tokens
          = 2,000,000 in, 80,000 out   ≈ $5.80

Same request count. 129× the cost.

So the limiter has to count something proportional to spend. The two workable units are tokens and money, and money is better because it survives a model change: a limit expressed in tokens silently becomes a different limit the day you switch to a model that costs four times as much.

A limiter that is actually atomic

First the mechanics, because most hand-rolled limiters have a race in them. The broken version:

// BROKEN: read, decide, write. Two concurrent requests both read the
// same count, both decide they are under the limit, and both write.
const count = Number(await redis.get(key)) || 0;
if (count >= limit) return deny();
await redis.set(key, count + 1, { ex: 60 });

Under concurrency — which is exactly the condition a limiter exists for — that lets more through than the limit allows. The fix is to do the whole read-decide-write on the server, in one operation. A Lua script in Redis is executed atomically, so nothing can interleave with it.

-- sliding-window.lua
-- KEYS[1]   the limiter key
-- ARGV[1]   now, in milliseconds
-- ARGV[2]   window size, in milliseconds
-- ARGV[3]   maximum weight allowed inside the window
-- ARGV[4]   the weight of this request
-- ARGV[5]   a unique member id, so identical weights do not collide

local now    = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local limit  = tonumber(ARGV[3])
local weight = tonumber(ARGV[4])
local member = ARGV[5]

-- Drop everything older than the window.
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, now - window)

-- Sum the weights still inside it. Weights are stored in the member string
-- as "weight:id", so we total them rather than counting entries.
local entries = redis.call('ZRANGE', KEYS[1], 0, -1)
local used = 0
for i = 1, #entries do
  local w = string.match(entries[i], "^(%d+)")
  used = used + (tonumber(w) or 0)
end

if used + weight > limit then
  local oldest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
  local retry = window
  if oldest[2] then retry = (tonumber(oldest[2]) + window) - now end
  return { 0, used, retry }
end

redis.call('ZADD', KEYS[1], now, weight .. ':' .. member)
redis.call('PEXPIRE', KEYS[1], window)
return { 1, used + weight, 0 }

// limiter.ts
import { Redis } from "ioredis";
import { readFileSync } from "node:fs";

const redis = new Redis(process.env.REDIS_URL!);
const script = readFileSync("sliding-window.lua", "utf8");

export type LimitResult = {
  allowed: boolean;
  used: number;
  limit: number;
  retryAfterMs: number;
};

export async function consume(
  key: string,
  limit: number,
  windowMs: number,
  weight = 1,
): Promise<LimitResult> {
  const member = crypto.randomUUID();

  const [ok, used, retry] = (await redis.eval(
    script,
    1,
    key,
    String(Date.now()),
    String(windowMs),
    String(limit),
    String(Math.max(1, Math.round(weight))),
    member,
  )) as [number, number, number];

  return {
    allowed: ok === 1,
    used,
    limit,
    retryAfterMs: Math.max(0, retry),
  };
}

A sliding window rather than a fixed one, deliberately. A fixed window resets on the minute, so a caller can send a full allowance at 11:59:59 and another at 12:00:00 — double the intended rate across a two-second span, which is precisely the burst you were limiting. The sorted set costs a little more memory and does not have that edge.

Limiting on cost, not count

The unit problem now has a solution: pass a weight. The weight is an estimate of what the request will cost, computed before the call, in a currency small enough to be an integer — tenths of a cent works well.

// cost.ts
// Prices per million tokens. Keep them in one place; they change.
const PRICES: Record<string, { in: number; out: number }> = {
  "openai/gpt-4o":      { in: 2.50, out: 10.00 },
  "openai/gpt-4o-mini": { in: 0.15, out: 0.60 },
};

/** Tenths of a cent, as an integer, so the limiter can add them up. */
export function estimateWeight(
  model: string,
  inputChars: number,
  maxTokens: number,
): number {
  const price = PRICES[model] ?? PRICES["openai/gpt-4o"];

  // ~4 characters per token for English. Deliberately an over-estimate for
  // other scripts, because under-estimating is the direction that costs money.
  const inputTokens = Math.ceil(inputChars / 4);

  // Charge for the ceiling, not the expectation: max_tokens is what the
  // request could cost, and the limiter must bound the worst case.
  const usd = (inputTokens * price.in + maxTokens * price.out) / 1_000_000;

  return Math.max(1, Math.ceil(usd * 1000));   // tenths of a cent
}

// The endpoint, with all of it in place.
import { auth } from "@/lib/auth";
import { consume } from "@/lib/limiter";
import { estimateWeight } from "@/lib/cost";

const MODEL = "openai/gpt-4o-mini";
const MAX_TOKENS = 500;
const MAX_INPUT_CHARS = 20_000;

export async function POST(request: Request) {
  // 1. Identity first. Everything downstream is keyed on it.
  const session = await auth();
  if (!session) {
    return Response.json({ error: "sign in" }, { status: 401 });
  }

  // 2. Bound the single request before anything else.
  const { text } = await request.json();
  if (typeof text !== "string" || text.length > MAX_INPUT_CHARS) {
    return Response.json({ error: "too long" }, { status: 400 });
  }

  // 3. Cost-weighted limit: $2.00 per account per hour, in tenths of a cent.
  const weight = estimateWeight(MODEL, text.length, MAX_TOKENS);
  const spend = await consume("spend:" + session.accountId, 2000, 3_600_000, weight);

  if (!spend.allowed) {
    return Response.json(
      { error: "hourly usage limit reached" },
      {
        status: 429,
        headers: { "Retry-After": String(Math.ceil(spend.retryAfterMs / 1000)) },
      },
    );
  }

  // 4. A burst limit as well, so one account cannot open 200 connections.
  const burst = await consume("burst:" + session.accountId, 10, 10_000, 1);
  if (!burst.allowed) {
    return Response.json(
      { error: "slow down" },
      {
        status: 429,
        headers: { "Retry-After": String(Math.ceil(burst.retryAfterMs / 1000)) },
      },
    );
  }

  // 5. The ceiling is enforced in the request itself, not only in the estimate.
  const res = await fetch("https://api.multigrid.ai/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: "Bearer " + process.env.LLM_API_KEY,
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: "user", content: "Summarise: " + text }],
      max_tokens: MAX_TOKENS,
    }),
    signal: AbortSignal.timeout(30_000),
  });

  return Response.json(await res.json());
}

The estimate is deliberately pessimistic — it charges the full max_tokens even though most answers are shorter. That is the right direction to be wrong in: a limiter that under-estimates lets through more spend than it was configured for, which defeats the purpose. If you want accuracy, reconcile afterwards by returning the unused weight once the real usage comes back.

Three layers, in order

1. Per-IP, before authentication. The only thing you have about an anonymous caller. Keep it loose — IPs are shared by offices, universities and mobile carriers, so a tight per-IP limit blocks real users in groups. Its job is to stop trivial floods, not to be precise.
2. Per-account, after authentication. The main control. Cost-weighted, over a window of an hour or a day. This is where the real budget lives, and it is only possible because every request has a payer attached.
3. Global, for the whole endpoint. A circuit breaker. If total spend across all users in an hour exceeds a threshold, shed load and alert — because the failure mode you cannot enumerate is the one that gets you, and a global ceiling bounds it without needing to know what it was.

Anonymous endpoints deserve one extra note. If a feature must work before sign-in, give anonymous users a small allowance keyed on a signed cookie plus IP, cap the model and the token ceiling hard, and treat the total anonymous spend as its own global budget with its own alarm. A free demo is a legitimate thing to want and it is also the single most commonly drained endpoint on the internet.

Telling the client, and what not to tell it

return Response.json(
  { error: "rate limited" },
  {
    status: 429,
    headers: {
      "Retry-After": String(Math.ceil(result.retryAfterMs / 1000)),
      "X-RateLimit-Limit": String(result.limit),
      "X-RateLimit-Remaining": String(Math.max(0, result.limit - result.used)),
      "X-RateLimit-Reset": String(Math.ceil((Date.now() + result.retryAfterMs) / 1000)),
    },
  },
);

Retry-After is the one that matters: without it, a client’s retry is a guess, and on a tight window guessing wrong burns the next window too. Send it on every 429 you emit, including the ones from the burst limiter.

What not to expose: the internal cost weight, the model name, the remaining dollar budget, or anything that lets a caller reverse-engineer the pricing model to find the cheapest way to consume the most. Report limits in units meaningful to the user — requests, or a plain “monthly usage” percentage — and keep the cost arithmetic on the server. The general treatment is in rate limiting as a security control.