What Is an LLM Gateway? How It Works, What to Look For, and When You Need One
By TrueFoundry · Published August 8, 2024 · Updated November 16, 2026
TL;DR
- An LLM gateway sits between a team's apps and every model provider, and handles routing, auth, budgets, guardrails, failover and logging in one place.
- It becomes worth having once more than one team, model or production app is involved.
- In 2026 it's increasingly governing MCP tools and agents too, not just model calls.
- A well-built gateway adds a few milliseconds per request. TrueFoundry's adds about 3–4 ms.
- Below: how it works, what to check when choosing one, and how the main options compare.
Ask five platform engineers what an LLM gateway is and there will be five slightly different answers. The short version: it's a layer that sits between a team's apps and the model providers they call, whether that's OpenAI, Anthropic, Google, Bedrock or something the team hosts itself. Every request goes through it. So instead of each team wiring up its own SDKs, keys and retry logic, teams get one API, one place to set budgets and guardrails, and one log that shows who called which model and what it cost.
Most teams don't go looking for a gateway. They end up needing one, usually after a rough week. A provider goes down for an hour and takes the support bot with it. Someone's test script loops over a weekend and quietly eats a month of budget. Or security asks for a list of every app sending customer data to an LLM, and nobody can answer.
All three of those problems happen in the same place: the request path. That's exactly where a gateway sits.
Why do teams need an LLM gateway?
Calling a provider directly is fine when a team has one app. It gets messy around the third or fourth, and a few things tend to break first.
Every provider speaks its own dialect. OpenAI, Anthropic, Bedrock and a vLLM server all format requests, errors and streaming a little differently, so each app ends up carrying its own adapter code.
Then there are the keys. They live in env files, CI secrets, a notebook someone forgot about. Rotating one turns into a scavenger hunt.
Cost is the one finance notices. The invoice is a single number. It won't say which team or feature spent it, and by the time it arrives, the money's already gone.
Outages hurt too. If a team's code calls one model directly, that model's bad hour is the team's bad hour, unless every team writes its own retry and fallback logic. Most don't.
And in a regulated industry, sooner or later someone will ask a team to show what data went to which model. Logs scattered across ten services won't get a team there.
None of this can really be fixed inside a single app. These are shared problems, so they need a shared layer.
How does an LLM gateway work?
Under the hood, the flow is fairly simple. A request comes in, gets checked, gets routed to a model, and gets logged on the way out. In a bit more detail:
- The app sends the request to the gateway instead of straight to the provider.
- The gateway checks who's calling and whether they're allowed to, then applies rate limits and budgets. If a team is over budget, the request stops right here, before it costs anything.
- Input guardrails run next, such as PII masking, prompt-injection checks or a team's own rules.
- The LLM router picks a model based on weights, latency, cost or how hard the task looks. If that model errors out, it tries the next one.
- The gateway translates the request into that provider's format, then turns the response back into the shape the app expects.
- Tokens, latency, cost and the full trace get recorded against whoever made the call.
For developers, switching over is usually a one-line change. Most gateways speak the OpenAI API, so developers keep their existing SDK, point it at a different base URL, and pick models by name:
from openai import OpenAI
# Point the SDK at the gateway, not the provider
client = OpenAI(
base_url="{GATEWAY_BASE_URL}",
api_key="your_truefoundry_api_key", # one key per team or app
)
# Switch providers by changing the model string
response = client.chat.completions.create(
model="provider-account/model-name",
messages=[{"role": "user", "content": "Summarize this ticket"}],
)
What about latency? A good gateway adds a few milliseconds, which disappears next to a model response that takes hundreds. In TrueFoundry's own benchmark, its gateway added about 3–4 ms and handled 350 requests per second on a single CPU core (full numbers here).
What features should an LLM gateway have in 2026?
For teams comparing gateways, these are the things TrueFoundry would check first. The first five matter for almost every production team; the rest depend on a team's setup.
| Feature | What it does | Must-have? |
|---|---|---|
| Unified API | One OpenAI-compatible endpoint for every provider and self-hosted model | Yes |
| Routing + fallbacks | Routes by weight, latency, cost or complexity; fails over automatically when a provider errors | Yes |
| Budgets + rate limits | Hard caps per team, app or user, enforced before the request runs | Yes |
| Guardrails | PII masking, prompt-injection defense, content moderation, custom policies | Yes for regulated teams |
| Observability + cost attribution | Full traces, token and cost per request, export to a team's stack via OpenTelemetry | Yes |
| Identity + access | SSO through a team's IdP, SCIM, role-based access, scoped keys | Yes for enterprises |
| Deployment + data residency | Runs in a team's VPC or on-prem, keeps logs in the team's region | Depends on compliance needs |
| Caching | Exact and semantic caching to cut repeat costs | Nice to have |
| MCP + agent governance | Controls which tools and agents can be called, and by whom | Yes if a team runs agents |
LLM gateway vs API gateway vs LLM proxy vs LLM router
People use these terms interchangeably, which causes real confusion on vendor calls. Roughly speaking, a proxy forwards traffic, a router chooses the model, an API gateway manages REST APIs, and an LLM gateway does the first two plus token-level control.
| Aspect | LLM proxy | LLM router | API gateway (Kong, Apigee) | LLM gateway |
|---|---|---|---|---|
| Main job | Forward requests to a provider | Choose the best model per request | Manage REST/GraphQL APIs | Govern all LLM traffic |
| Understands tokens and cost | Basic | Partly | No | Yes |
| Fallbacks across providers | Rarely | Yes | No | Yes |
| Guardrails + PII | No | No | Through plugins | Built in |
| Budgets per team | No | No | Request-count limits only | Yes, in tokens and dollars |
| Governs MCP tools + agents | No | No | No | Yes (modern gateways) |
Want more depth? TrueFoundry has written separately about LLM proxies, AI gateways vs API gateways and MCP gateways vs proxies vs routers.
Do you actually need an LLM gateway?
Not always, and it's worth saying that plainly. One app, one provider, a small bill? Call the provider directly and come back to this later. A gateway is one more piece of infrastructure, and it should earn its place.
It's probably time if a couple of these sound familiar:
- More than one team is building with LLMs
- The team uses, or wants to try, more than one model provider
- Finance asked which team is driving the LLM bill and nobody could say
- A provider outage has already hit a customer-facing feature
- Security or compliance wants an audit log of prompts and responses
- Developers are pointing coding agents like Claude Code or Cursor at company systems
| Aspect | Direct API calls | LLM gateway |
|---|---|---|
| Setup | Fast for the first app | One integration, then every app reuses it |
| Switching models | Code change in every app | Change a config value |
| Outages | The app goes down with the provider | Automatic fallback to another model |
| Cost visibility | One invoice, no breakdown | Per team, app and user, in real time |
| Security | Keys and policies managed per app | Central keys, guardrails and audit trail |
What does this look like at scale?
A Fortune 50 healthcare company runs an AI agent handling 500 million IVR calls on TrueFoundry. FloQast governed 4M+ AI requests across 6 model providers in 90 days, with a 98.9% success rate.
From LLM gateway to AI gateway: MCP tools and agents
Here's what changed over the last year or so. Agents don't just call models anymore. They call tools, usually through the Model Context Protocol (MCP), and those tools are a company's CRM, its database, its ticketing system. Each of those calls needs the same treatment a model call gets: who's allowed, with which permissions, and a record of what happened.
So the category is stretching. When vendors say "AI gateway" now, they usually mean one control plane for three kinds of traffic: model calls, tool calls through an MCP gateway, and the agents themselves through an agent gateway, which handles agent identity, a shared registry and human approval for risky actions.
The shift shows in the market, too. Gartner is putting out its first Magic Quadrant for AI gateways this year, and in April Palo Alto Networks said it would buy Portkey (TrueFoundry's take on that deal). A team choosing a gateway today should make sure it can grow into MCP and agent governance. Otherwise the team will probably be shopping again within a year.
Popular LLM gateway options in 2026
Which one fits depends on what a team cares about most: running open source in-house, enterprise governance, or quick access to lots of models. Here's how TrueFoundry sees the main options, trade-offs included:
| Option | Type | Best for | Watch out for |
|---|---|---|---|
| TrueFoundry AI Gateway | LLM + MCP + agent gateway; runs in a team's VPC or on-prem | Enterprises that need governance, data residency and agent control | More than a solo developer needs |
| LiteLLM | Open-source proxy | Teams happy to self-host and maintain it | You own patching, scaling and security; its PyPI package was hit by a supply-chain attack in March 2026 |
| Portkey | Managed gateway | Teams wanting a hosted gateway with observability | Palo Alto Networks plans to acquire it, which may shift its roadmap toward security |
| Kong AI Gateway | API gateway with AI plugins | Teams already running Kong for APIs | Built for REST first; token-level budgets and MCP governance are add-ons |
| OpenRouter | Hosted model marketplace | Fast access to hundreds of models with one key | Traffic runs through a third party; limited team-level governance |
| Cloudflare AI Gateway | Edge proxy | Caching and basic analytics for web apps | Light on guardrails, budgets and self-hosting |
For a longer comparison covering latency, pricing and enterprise features, see TrueFoundry's best LLM gateways guide.
Best practices for rolling out an LLM gateway
A few lessons that save a lot of pain later.
Start with whichever app is noisiest, the one with the biggest bill or the most incidents. A visible win there makes it far easier to bring other teams along.
Once apps have moved over, revoke the direct provider keys. A gateway people can route around doesn't govern anything.
Put budgets in on day one, for every team and environment. Adding limits after people are used to having none is a much harder conversation.
Give every production model a fallback from a different provider, and actually test the failover before it's needed.
Finally, send traces to the tools the team already watches, like Datadog or Grafana, through OpenTelemetry instead of adding yet another dashboard.
The bottom line
An LLM gateway turns a pile of scattered model calls into something a team can actually govern. A team that runs one small app can wait. Once several teams are shipping LLM features, or agents start touching company systems, the gateway is where cost control, reliability and security really get enforced. Just pick one that handles MCP tools and agents as well, because that's where the traffic is heading.