Provider preferences

Order, exclude, sort, split and pin the providers behind one model id.

6 min read

Why this exists

An open-weights model is usually sold by several hosts at very different prices and speeds. meta/llama-3.3-70b-instruct is served here by DeepInfra, Groq and Together, and the completion price across those three differs by more than a factor of three for the same weights. One public model id, several real choices — this object is how you make them.

Request body

{
  "model": "meta/llama-3.3-70b-instruct",
  "provider": {
    "order": ["groq", "deepinfra"],
    "ignore": ["together"],
    "sort": "price",
    "allow_fallbacks": true,
    "weights": { "groq": 90, "deepinfra": 10 }
  }
}

The provider object

Parameter	Type	Required	Description
order	string[]	Optional	Provider slugs to try first, in this order. A preference, not a filter.
ignore	string[]	Optional	Never use these. This is the filter.
allow_fallbacks	boolean	Optional	Defaults true. `false` pins the request to the first usable route — no second provider, and no second model from `models` either, since a flag that only half applied would be worse than none.
sort	"price" \| "latency" \| "throughput"	Optional	`price` sorts on completion price, with prompt price as the tiebreak — completion tokens dominate almost every real bill. `latency` and `throughput` use measured rankings from our own traffic; providers we have not measured sort last but keep their catalogue order among themselves, because unmeasured is not the same as slow.
weights	{ [slug]: number }	Optional	A traffic split. `{ "groq": 90, "deepinfra": 10 }` sends roughly a tenth of requests to DeepInfra. The draw decides which route is tried first; the rest keep their order behind the winner, so a split still fails over.

Provider slugs are the ones GET /providers returns. That endpoint also reports available, which tells you whether this deployment holds a key for it at all — a provider you order first but have no key for is simply skipped.

Order is not a filter

Naming a provider in order does not exclude the others

A provider you did not name still gets a turn once the ones you did have failed, unless you turn fallbacks off or list it in ignore. Conflating the two is the classic way to strand a request that could have been served — so they are two separate fields, and only one of them removes anything.

Weights are the same shape of decision: a provider with no weight cannot win the draw, but it is not dropped. Naming one provider in weights does not silently disable every provider you did not mention.

Suffixes on the model id

Three routing strategies can be written into the model id itself, because the model id is often the only part of a request that is already an environment variable — and changing a deployment’s routing should not need a code change.

Suffix	Description
:floor	Cheapest route for this model. Equivalent to `sort: "price"`.
:nitro	Fastest measured route. Equivalent to `sort: "latency"`.
:free	Only routes that cost you nothing — your own provider key, or a route genuinely priced at zero. If none exists the request is refused with `no_free_route`; it never quietly falls back to a paid one, since charging you is the outcome the suffix was ruling out.

An explicit provider.sort wins over a suffix: someone who wrote both said the specific thing twice, and reversing a field they set is how a routing layer becomes impossible to trust. Only those three suffixes are recognised, so a real model id containing a colon — several vendors ship names like llama-3:8b — is not mangled into a 404.

Something here disagrees with what the API actually did? That is a bug in this page, and worth reporting.

Report it