Rate Limiting as a Security Control

A rate limit is a bound on how fast something bad can happen. It is not a decision about whether it may happen, and treating it as one is how endpoints end up with a limiter and an unbounded bill.

What a rate limit is for

Rate limiting appears in the OWASP list under LLM10, Unbounded Consumption, and its security value is specific. It converts attacks whose cost is a function of volume into attacks that take longer than the attacker is willing to spend, and it gives you time to notice.

• Credential stuffing and brute force against your own auth, unchanged from any other API.
• Cost exhaustion. The wallet attack is a volume attack; a per-key token budget is the direct control.
• Model extraction. Extraction needs a large number of queries, so a limit raises the attacker’s cost and elapsed time substantially.
• Automated jailbreak search. Adversarial optimisation requires many attempts against the same target. A limit turns a twenty-minute search into a multi-day one that your anomaly detection has a chance to see.
• Bulk scraping of an expensive capability — using your summariser as free inference at scale.

Requests per minute is the wrong unit

On a conventional API, requests are roughly interchangeable. On an inference endpoint they differ by orders of magnitude in cost: a twenty-token question and a 200,000-token document summarisation are both one request. A limiter counting requests permits an attacker to take the entire budget within a compliant request count.

Limit on tokens, and ideally on money. That has an awkward property — you do not know the output size until the response finishes — which is solved by reserving and reconciling: charge an estimate against the bucket before the call, based on input tokens plus max_tokens, then settle to the true usage when the response completes. Reserving the worst case is what stops a single streaming request from overrunning a budget that looked fine when it started.

Keep a request-count limit as well. It is the cheapest defence against a flood of tiny requests, and it can be enforced before any tokenising happens.

Choosing the dimension

Layer them. A single dimension is always evadable by moving along another one, and the layers should fail independently so that a misconfiguration in one does not remove the others.

Add a concurrency limit alongside the rate limit, because they bound different things. A rate limit constrains work per unit time; a concurrency limit constrains how many requests are in flight at once, and it is the one that protects you from a caller who opens two hundred simultaneous long-context streams. Since inference requests are long-lived compared with ordinary API calls, in-flight count is often the binding resource — and a queue with a bounded depth and a fast rejection past it degrades far more gracefully than an unbounded one that converts overload into timeouts you still pay for.

An implementation

A token bucket, sized in model tokens rather than requests, with the reserve-and-settle behaviour that streaming requires:

// Token bucket over model tokens. Refills continuously; capacity is the
// burst allowance. Stored per key -- in Redis if you have more than one
// process, because a per-instance limiter multiplies by your replica count.

type Bucket = { tokens: number; updatedAt: number };

const CAPACITY = 200_000;        // burst
const REFILL_PER_SEC = 2_000;    // sustained

function refill(b: Bucket, now: number): Bucket {
  const elapsed = (now - b.updatedAt) / 1000;
  return {
    tokens: Math.min(CAPACITY, b.tokens + elapsed * REFILL_PER_SEC),
    updatedAt: now,
  };
}

/** Reserve the worst case before the call. */
export function reserve(b: Bucket, inputTokens: number, maxOutput: number, now = Date.now()) {
  const cost = inputTokens + maxOutput;
  const next = refill(b, now);
  if (next.tokens < cost) {
    const waitSec = (cost - next.tokens) / REFILL_PER_SEC;
    return { ok: false as const, bucket: next, retryAfter: Math.ceil(waitSec) };
  }
  return { ok: true as const, bucket: { ...next, tokens: next.tokens - cost }, reserved: cost };
}

/** Settle once usage is known: hand back what was not used. */
export function settle(b: Bucket, reserved: number, actual: number): Bucket {
  return { ...b, tokens: Math.min(CAPACITY, b.tokens + Math.max(0, reserved - actual)) };
}

Three details that decide whether it works in production. Return 429 with a Retry-After header, so well-behaved clients back off instead of hammering. Make the state shared — a limiter held in process memory silently multiplies its limit by the number of replicas, which is the most common way a correct algorithm ships as an incorrect control. And fail closed on an infrastructure error, or at minimum degrade to a much tighter static limit, because a limiter that opens when Redis is unavailable is a limiter an attacker can remove.

What it does not stop

This is the part that belongs in the design document, because a rate limit invites the belief that the endpoint is protected.

• Prompt injection. One request. Nothing about a rate limit is relevant to it.
• Any single expensive request. Unless you bound context length and max_tokens per request, the first call can be the whole budget.
• Distributed abuse. A thousand accounts each staying under the per-account limit sum to a large number. That is what abuse detection is for: correlation across identities, not enforcement within one.
• Low-and-slow extraction. A patient attacker under the limit for months is invisible to a limiter and visible to aggregate volume analysis.
• Insider and compromised-key use that looks like normal traffic, because it is under a legitimate key at a legitimate rate.

Write that list into the design document next to the limiter configuration. The realistic failure here is not a badly tuned bucket; it is a review that sees “rate limited: yes” on a checklist and stops asking questions. A limiter bounds the rate of a known quantity, and it says nothing at all about the two threats on this page that arrive in a single, entirely compliant request.