Detecting Automated Abuse of an AI Endpoint

Rate limits enforce a ceiling. Detection is the part that notices the traffic staying carefully underneath it, and on an inference endpoint the giveaway is usually the shape of the prompts rather than the request pattern.

Define abuse before detecting it

“Automated” is not the definition. Plenty of legitimate traffic is automated: a customer’s batch job, a CI pipeline, an integration. Detecting automation and calling it abuse produces a system that punishes your best customers.

Write down the behaviours you actually want to stop, because each one has a different signature: reselling your capability as a general model; systematic extraction of your data or your model’s behaviour; automated search for a policy bypass; cost exhaustion; and scraped-credential use. A detector built for the wrong one of these will be both noisy and blind.

Decide the privacy question in the same sitting, because it constrains which signals you may build. Some of the strongest ones read prompt content, and prompt content is frequently the most sensitive data in the system. The workable compromise is to derive features rather than retain text — a normalised structural hash, a token count, a language code, a similarity score against the key’s recent history — and to keep raw prompts only under a short retention window with access logged. Write that down before building, since a detector that quietly depends on warehousing customer prompts is one legal review away from deletion.

Signals that actually separate

Prompt-shape signals

• Template similarity. Scripted traffic reuses a skeleton with a slot filled. Hash the prompt with the variable spans normalised — or compare embeddings — and a high rate of near-duplicates with one changing field is characteristic. Note that this also describes your own product’s templated features, so compute it per key against that key’s own history.
• Distribution drift. A key whose prompts suddenly change topic, language or length distribution is either a new feature shipping or a key that changed hands.
• Parameter fingerprints. Real applications settle on a small set of sampling parameters. Traffic sweeping temperature, or requesting log probabilities it never used before, is doing something other than serving users — the log-probability case in particular is a model-extraction signal.
• Systematic coverage. Prompts that walk an enumerable space — every product id, every letter pair, every entity in a list — indicate harvesting rather than use.

Timing and session signals

• Inter-arrival regularity. Humans are bursty and irregular. Very low variance in the gap between requests, or perfectly periodic arrival, is a scheduler. Beware: a queue-driven legitimate workload looks the same, which is why this signal needs company.
• No think time after a response. A new request arriving faster than a human could have read the last one, sustained over a session.
• Round-the-clock uniformity. Human traffic has a diurnal shape tied to a timezone. Flat traffic for days is a machine.
• Retry-on-refusal. The strongest single signal for policy-bypass search: a refusal followed immediately by a near-identical prompt, repeatedly. Legitimate users rephrase once or twice and give up; a search process does not.

Identity and infrastructure signals

• Requests from hosting-provider ASNs on a consumer product.
• Many accounts sharing an IP, a device fingerprint, or a payment instrument.
• Accounts that reach maximum usage within minutes of signup, with no exploration first.
• Disposable-domain email at signup correlated with immediate heavy usage.

Combining signals without a model

Resist building a classifier first. A weighted score over individually-explainable signals is easier to tune, easier to defend to a customer whose account you throttled, and easier to debug when it misfires:

// Each signal returns 0..1 and is computed against the KEY'S OWN baseline,
// not a global one. Weights are policy; keep them in config, not in code.

const SIGNALS = [
  { name: "template_similarity", weight: 3, fn: templateSimilarity },
  { name: "retry_after_refusal",  weight: 4, fn: retryAfterRefusal },
  { name: "arrival_regularity",   weight: 2, fn: arrivalRegularity },
  { name: "no_diurnal_pattern",   weight: 1, fn: diurnalFlatness },
  { name: "logprob_requests",     weight: 3, fn: logprobUsage },
  { name: "hosting_asn",          weight: 1, fn: hostingAsn },
];

export function abuseScore(w: Window): Scored {
  const parts = SIGNALS.map((s) => ({ name: s.name, v: s.fn(w) * s.weight }));
  const score = parts.reduce((a, p) => a + p.v, 0);
  return {
    score,
    // Always keep the breakdown. An unexplainable score is one an operator
    // will learn to ignore, and one you cannot justify to a customer.
    reasons: parts.filter((p) => p.v > 0).sort((a, b) => b.v - a.v),
  };
}

Compute the signals over a rolling window per key and per account, and baseline each key against itself. Global thresholds are what generate the false positives, because the batch customer and the reseller look identical in absolute terms and completely different relative to their own histories.

A graded response ladder

Binary blocking is the wrong response for a probabilistic signal. Escalate, and make each step reversible:

• Observe. Score and record. Most alerts should end here, and the volume at this level tells you whether the thresholds are sane.
• Throttle. Tighten the limit for that key. Costs a legitimate customer some latency and costs an abuser their economics.
• Add friction. Require a re-authentication, a verified payment method, or a support contact for continued volume.
• Restrict scope. Remove access to the expensive models or the long-context routes while leaving the cheap path open. Frequently the right permanent answer.
• Suspend, with a fast, staffed appeal path. If you cannot reverse a suspension within hours, you cannot afford aggressive thresholds.

Living with false positives

Every detector has both error types and you are choosing between them, not eliminating them. Two habits keep the choice deliberate. Sample your enforcement actions weekly and have a human review them against the account’s history — the review is how you learn that one signal is carrying all the false positives. And instrument the appeal path as a metric, because the rate of successful appeals is the closest thing you have to a false-positive rate on real traffic.

Publish the boundary you enforce, too. An acceptable-use policy that states volume expectations converts an argument about fairness into a reference to a document, and it makes legitimate heavy users tell you in advance rather than surprise you.