LiteLLM as a Provider Abstraction Layer

LiteLLM translates one request shape — the OpenAI chat completion — into whatever a hundred providers expect, and translates the responses back. For the common fields it does this well enough to be boring, which is the correct outcome. The interesting part of the page is the fields where translation is impossible, because that is where the abstraction stops being free.

Two products, one name

People argue past each other about this library because they are using different halves of it.

They solve different problems. The SDK solves “my Python code should not care which provider”. The proxy solves “my organisation should have one endpoint, one set of credentials and one place where spend is visible”. Teams that adopt the SDK and then find themselves building key management around it wanted the proxy.

The SDK: one function, many providers

The core surface is small and has been stable for years: a completion function, an async variant, an embedding function, and a model string that carries the provider as a prefix.

from litellm import completion

# The provider is part of the model string; credentials come from
# provider-specific environment variables.
resp = completion(
    model="anthropic/<model-id>",
    messages=[{"role": "user", "content": "..."}],
)

print(resp.choices[0].message.content)
print(resp.usage.total_tokens)

# Same call, different provider, nothing else changes:
resp = completion(model="openai/<model-id>", messages=[...])
resp = completion(model="ollama/<model-id>", messages=[...])

The response is normalised into the OpenAI shape regardless of source, which is the property that makes the abstraction useful: choices[0].message.content and a usage object with token counts exist for every provider. Anything you build on those two fields — logging, cost attribution, caching keys — works everywhere without a conditional.

What maps cleanly

The fields below translate across essentially every provider, and they are enough for most applications. If your usage stays inside this list, provider swaps really are a string change.

• Messages with system, user and assistant roles. Including the awkward case of providers that take the system prompt as a separate parameter rather than as a message.
• Temperature, top-p, max output tokens, stop sequences. Names differ per provider; meanings are close enough that the translation is honest.
• Streaming. Delivered as OpenAI-shaped chunks regardless of the provider’s native event format, which is a genuine saving — the native formats differ considerably.
• Tool calling. Tool definitions and tool-call responses map for providers with native support. Where support is absent the abstraction is thinner than it looks; see below.
• Token usage. Prompt, completion and total, in one place, which is the foundation for cost attribution.

What does not map

This is the section that justifies the page. Every abstraction over heterogeneous providers has a lowest common denominator, and knowing where it sits is what stops you from being surprised in production.

The practical rule: keep provider-specific fields in one small per-provider configuration module rather than sprinkled through call sites. Then “what breaks if we switch” has an answer you can read, which is the same discipline described in normalising API errors.

When to run the proxy instead

Four situations where the proxy is the right half and the SDK is not.

1. More than one language in the estate. A Python library does nothing for the TypeScript service. An HTTP endpoint serves both.
2. Credentials must not be in application processes. Applications hold a virtual key to the proxy; the real provider keys live in one place with one rotation procedure. This is the strongest argument for the proxy and it is a security argument, not a convenience one — see API key security.
3. Spend needs a per-team or per-customer limit. Budgets enforced at the gateway are enforced; budgets enforced in library code are enforced until someone writes a script.
4. You want one place to change routing. Failover, model aliasing and load balancing configured centrally means an incident is a config change rather than a deploy of every service.

The cost of the proxy is that it is a component in the request path: it needs deploying, monitoring, scaling and a plan for what happens when it is down. That is a real operational obligation, and it is the honest reason to compare running one against using a hosted gateway that already has those properties.

Failure modes worth knowing

Silently dropped parameters. Passing a parameter a provider does not support may result in it being dropped rather than raising. The symptom is a setting that appears to have no effect. When a parameter seems ignored, check the provider actually supports it before assuming the model is disobedient.

Model strings drift from reality. The mapping from a model string to a provider endpoint lives in the library and is updated as providers change. A new model can be unavailable through the abstraction until the library catches up, which is a dependency on somebody else’s release cadence — the standard tax of a translation layer.

Cost figures are a lookup table. Any cost number the library reports comes from a bundled price map, not from your invoice. It is useful for relative comparison and it is not an accounting record. Reconcile against the provider’s own usage reporting before anyone builds a chargeback process on it.