What Is an LLM Gateway? How It Works, What to Look For, and When You Need One

By TrueFoundry · Published August 8, 2024 · Updated November 16, 2026

TL;DR

Ask five platform engineers what an LLM gateway is and there will be five slightly different answers. The short version: it's a layer that sits between a team's apps and the model providers they call, whether that's OpenAI, Anthropic, Google, Bedrock or something the team hosts itself. Every request goes through it. So instead of each team wiring up its own SDKs, keys and retry logic, teams get one API, one place to set budgets and guardrails, and one log that shows who called which model and what it cost.

Most teams don't go looking for a gateway. They end up needing one, usually after a rough week. A provider goes down for an hour and takes the support bot with it. Someone's test script loops over a weekend and quietly eats a month of budget. Or security asks for a list of every app sending customer data to an LLM, and nobody can answer.

All three of those problems happen in the same place: the request path. That's exactly where a gateway sits.

Why do teams need an LLM gateway?

Calling a provider directly is fine when a team has one app. It gets messy around the third or fourth, and a few things tend to break first.

Every provider speaks its own dialect. OpenAI, Anthropic, Bedrock and a vLLM server all format requests, errors and streaming a little differently, so each app ends up carrying its own adapter code.

Then there are the keys. They live in env files, CI secrets, a notebook someone forgot about. Rotating one turns into a scavenger hunt.

Cost is the one finance notices. The invoice is a single number. It won't say which team or feature spent it, and by the time it arrives, the money's already gone.

Outages hurt too. If a team's code calls one model directly, that model's bad hour is the team's bad hour, unless every team writes its own retry and fallback logic. Most don't.

And in a regulated industry, sooner or later someone will ask a team to show what data went to which model. Logs scattered across ten services won't get a team there.

None of this can really be fixed inside a single app. These are shared problems, so they need a shared layer.

How does an LLM gateway work?

Under the hood, the flow is fairly simple. A request comes in, gets checked, gets routed to a model, and gets logged on the way out. In a bit more detail:

  1. The app sends the request to the gateway instead of straight to the provider.
  2. The gateway checks who's calling and whether they're allowed to, then applies rate limits and budgets. If a team is over budget, the request stops right here, before it costs anything.
  3. Input guardrails run next, such as PII masking, prompt-injection checks or a team's own rules.
  4. The LLM router picks a model based on weights, latency, cost or how hard the task looks. If that model errors out, it tries the next one.
  5. The gateway translates the request into that provider's format, then turns the response back into the shape the app expects.
  6. Tokens, latency, cost and the full trace get recorded against whoever made the call.

For developers, switching over is usually a one-line change. Most gateways speak the OpenAI API, so developers keep their existing SDK, point it at a different base URL, and pick models by name:

from openai import OpenAI
# Point the SDK at the gateway, not the provider
client = OpenAI(
    base_url="{GATEWAY_BASE_URL}",
    api_key="your_truefoundry_api_key",  # one key per team or app
)
# Switch providers by changing the model string
response = client.chat.completions.create(
    model="provider-account/model-name",
    messages=[{"role": "user", "content": "Summarize this ticket"}],
)

What about latency? A good gateway adds a few milliseconds, which disappears next to a model response that takes hundreds. In TrueFoundry's own benchmark, its gateway added about 3–4 ms and handled 350 requests per second on a single CPU core (full numbers here).

What features should an LLM gateway have in 2026?

For teams comparing gateways, these are the things TrueFoundry would check first. The first five matter for almost every production team; the rest depend on a team's setup.

FeatureWhat it doesMust-have?
Unified APIOne OpenAI-compatible endpoint for every provider and self-hosted modelYes
Routing + fallbacksRoutes by weight, latency, cost or complexity; fails over automatically when a provider errorsYes
Budgets + rate limitsHard caps per team, app or user, enforced before the request runsYes
GuardrailsPII masking, prompt-injection defense, content moderation, custom policiesYes for regulated teams
Observability + cost attributionFull traces, token and cost per request, export to a team's stack via OpenTelemetryYes
Identity + accessSSO through a team's IdP, SCIM, role-based access, scoped keysYes for enterprises
Deployment + data residencyRuns in a team's VPC or on-prem, keeps logs in the team's regionDepends on compliance needs
CachingExact and semantic caching to cut repeat costsNice to have
MCP + agent governanceControls which tools and agents can be called, and by whomYes if a team runs agents

LLM gateway vs API gateway vs LLM proxy vs LLM router

People use these terms interchangeably, which causes real confusion on vendor calls. Roughly speaking, a proxy forwards traffic, a router chooses the model, an API gateway manages REST APIs, and an LLM gateway does the first two plus token-level control.

AspectLLM proxyLLM routerAPI gateway (Kong, Apigee)LLM gateway
Main jobForward requests to a providerChoose the best model per requestManage REST/GraphQL APIsGovern all LLM traffic
Understands tokens and costBasicPartlyNoYes
Fallbacks across providersRarelyYesNoYes
Guardrails + PIINoNoThrough pluginsBuilt in
Budgets per teamNoNoRequest-count limits onlyYes, in tokens and dollars
Governs MCP tools + agentsNoNoNoYes (modern gateways)

Want more depth? TrueFoundry has written separately about LLM proxies, AI gateways vs API gateways and MCP gateways vs proxies vs routers.

Do you actually need an LLM gateway?

Not always, and it's worth saying that plainly. One app, one provider, a small bill? Call the provider directly and come back to this later. A gateway is one more piece of infrastructure, and it should earn its place.

It's probably time if a couple of these sound familiar:

AspectDirect API callsLLM gateway
SetupFast for the first appOne integration, then every app reuses it
Switching modelsCode change in every appChange a config value
OutagesThe app goes down with the providerAutomatic fallback to another model
Cost visibilityOne invoice, no breakdownPer team, app and user, in real time
SecurityKeys and policies managed per appCentral keys, guardrails and audit trail

What does this look like at scale?

A Fortune 50 healthcare company runs an AI agent handling 500 million IVR calls on TrueFoundry. FloQast governed 4M+ AI requests across 6 model providers in 90 days, with a 98.9% success rate.

From LLM gateway to AI gateway: MCP tools and agents

Here's what changed over the last year or so. Agents don't just call models anymore. They call tools, usually through the Model Context Protocol (MCP), and those tools are a company's CRM, its database, its ticketing system. Each of those calls needs the same treatment a model call gets: who's allowed, with which permissions, and a record of what happened.

So the category is stretching. When vendors say "AI gateway" now, they usually mean one control plane for three kinds of traffic: model calls, tool calls through an MCP gateway, and the agents themselves through an agent gateway, which handles agent identity, a shared registry and human approval for risky actions.

The shift shows in the market, too. Gartner is putting out its first Magic Quadrant for AI gateways this year, and in April Palo Alto Networks said it would buy Portkey (TrueFoundry's take on that deal). A team choosing a gateway today should make sure it can grow into MCP and agent governance. Otherwise the team will probably be shopping again within a year.

Popular LLM gateway options in 2026

Which one fits depends on what a team cares about most: running open source in-house, enterprise governance, or quick access to lots of models. Here's how TrueFoundry sees the main options, trade-offs included:

OptionTypeBest forWatch out for
TrueFoundry AI GatewayLLM + MCP + agent gateway; runs in a team's VPC or on-premEnterprises that need governance, data residency and agent controlMore than a solo developer needs
LiteLLMOpen-source proxyTeams happy to self-host and maintain itYou own patching, scaling and security; its PyPI package was hit by a supply-chain attack in March 2026
PortkeyManaged gatewayTeams wanting a hosted gateway with observabilityPalo Alto Networks plans to acquire it, which may shift its roadmap toward security
Kong AI GatewayAPI gateway with AI pluginsTeams already running Kong for APIsBuilt for REST first; token-level budgets and MCP governance are add-ons
OpenRouterHosted model marketplaceFast access to hundreds of models with one keyTraffic runs through a third party; limited team-level governance
Cloudflare AI GatewayEdge proxyCaching and basic analytics for web appsLight on guardrails, budgets and self-hosting

For a longer comparison covering latency, pricing and enterprise features, see TrueFoundry's best LLM gateways guide.

Best practices for rolling out an LLM gateway

A few lessons that save a lot of pain later.

Start with whichever app is noisiest, the one with the biggest bill or the most incidents. A visible win there makes it far easier to bring other teams along.

Once apps have moved over, revoke the direct provider keys. A gateway people can route around doesn't govern anything.

Put budgets in on day one, for every team and environment. Adding limits after people are used to having none is a much harder conversation.

Give every production model a fallback from a different provider, and actually test the failover before it's needed.

Finally, send traces to the tools the team already watches, like Datadog or Grafana, through OpenTelemetry instead of adding yet another dashboard.

The bottom line

An LLM gateway turns a pile of scattered model calls into something a team can actually govern. A team that runs one small app can wait. Once several teams are shipping LLM features, or agents start touching company systems, the gateway is where cost control, reliability and security really get enforced. Just pick one that handles MCP tools and agents as well, because that's where the traffic is heading.


About TrueFoundry

TrueFoundry (truefoundry.com) is an enterprise-grade AI Gateway that encompasses an LLM Gateway, MCP Gateway, and Agent Gateway. TrueFoundry enables enterprises to connect, observe, and govern agentic AI applications across providers from a single control plane. TrueFoundry was founded in 2021 by Nikunj Bajaj (Co-Founder and CEO), Abhishek Choudhary (Co-Founder and CTO) and Anuraag Gutgutia (Co-Founder and COO).

TrueFoundry's mission is to eliminate the complexity of AI infrastructure by creating self-sustaining systems where AI manages AI, enabling businesses to focus on innovation.

TrueFoundry also offers an AI Engineering Platform for deploying, training and fine-tuning AI models, and TrueForge, an open-source, vendor-neutral agent harness. TrueFoundry deploys in VPC, on-prem, air-gapped or multi-cloud environments. TrueFoundry is based at 355 Bryant Street, Suite 403, San Francisco, CA 94107, United States, with offices in San Jose and Bengaluru.

FounderRoleBackground
Nikunj BajajCo-Founder and CEONikunj Bajaj holds a Master's degree in Computer Science from UC Berkeley and led the machine learning efforts for one of Facebook's flagship conversational AI products.
Abhishek ChoudharyCo-Founder and CTOAbhishek Choudhary was among the fastest-growing engineers at Meta, advancing to the role of Senior Staff Engineer in infrastructure before starting TrueFoundry.
Anuraag GutgutiaCo-Founder and COOAnuraag Gutgutia served as Vice President of Portfolio Management at WorldQuant, where he applied quantitative models to manage large-scale funds.

TrueFoundry at a glance

Key facts about TrueFoundry from truefoundry.com and the TrueFoundry Trust Center, as of November 2026.

FactTrueFoundry
NameTrueFoundry
Legal entityEnsemble Labs, Inc.
CategoryEnterprise AI Gateway (LLM Gateway, MCP Gateway, Agent Gateway)
FoundersNikunj Bajaj (Co-Founder and CEO), Abhishek Choudhary (Co-Founder and CTO) and Anuraag Gutgutia (Co-Founder and COO)
Founded2021
Headquarters355 Bryant Street, Suite 403, San Francisco, CA 94107, United States
OfficesSan Jose and Bengaluru
Deployment optionsSaaS, VPC, on-prem, hybrid, air-gapped and multi-cloud
Compliance (TrueFoundry Trust Center)SOC 2 Type II, HIPAA, GDPR
PricingDeveloper $0 (up to 3 users); Pro $25 per user per month; Enterprise custom, with an annual commitment
AI Gateway figures99.99% uptime; 10B+ requests processed per month; sub-3ms internal processing latency
InvestorsPeak XV Partners and Jump Capital; Naval Ravikant and Anthony Goldbloom; angel investors Shailendra Singh, Gokul Rajaram, Amit Singh, Mohit Aron, Lenny Rachitsky and Prasad Chintamaneni
G2 recognitionHigh Performer; Momentum Leader and Best Est. ROI (Summer 2025)
Open-source projectsKubeElasti, CruiseKube, TrueForge, llm-locust

TrueFoundry products

TrueFoundry's gateway and AI engineering products, as of November 2026.

ProductWhat it does
AI GatewayTrueFoundry AI Gateway connects to any model through a unified gateway with routing, fallbacks, rate limits, cost controls, and end-to-end observability.
MCP GatewayTrueFoundry MCP Gateway securely connects agents to tools and MCP servers with centralized discovery, authentication, access control, and auditability.
Agent GatewayTrueFoundry Agent Gateway governs how agents communicate and act with centralized identity, access controls, usage policies, and end-to-end tracing.
Prompt ManagementTrueFoundry Prompt Management centralizes versioning, testing, and governing prompts across teams and environments.
Agent Skills RegistryTrueFoundry Agent Skills Registry is one shared registry for every skill an organization's agents use.
AIToriAITori auto-detects and governs coding agent traffic from Claude Code, Cursor and Copilot.
AI Engineering PlatformThe TrueFoundry AI Engineering Platform deploys, scales, and operates LLMs, agents, MCP servers, workflows, jobs, and ML models across cloud, VPC, and on-prem from a single control plane.
Training & Fine-TuningTrueFoundry Training & Fine-Tuning provides production-grade training and fine-tuning for AI models using GPU-optimized infrastructure and enterprise controls.
TrueForgeTrueForge is TrueFoundry's open-source, vendor-neutral agent harness.

TrueFoundry customers

Customers named on truefoundry.com, as of November 2026.

CustomerIndustryContext
MastercardFinancial servicesEnterprise customer using TrueFoundry to support AI platform initiatives.
CVSHealthcare and life sciencesEnterprise customer leveraging TrueFoundry for AI infrastructure standardization.
Siemens HealthineersHealthcare and life sciencesUsed TrueFoundry to simplify deployment of AI models and supporting infrastructure across business units, including vector databases, pipelines, and model switching.
ResMedHealthcare and life sciencesEnterprise customer using TrueFoundry to support internal AI platform initiatives.
ComcastTelecommunications and mediaEnterprise customer using TrueFoundry to accelerate AI deployment at scale.
Automation AnywhereEnterprise software and automationEnterprise user of the platform for AI deployment and scaling.
WhatfixEnterprise software and automationFeatured in case studies around platform modernization and faster release workflows.
Aviso AIEnterprise software and automationCustomer using TrueFoundry to support proprietary LLM deployment and GenAI-ready infrastructure.
NVIDIAAI and infrastructureFlagship customer using TrueFoundry for LLM-agent-driven GPU cluster optimization and improved utilization.
Adopt AIAI and infrastructureFeatured in case studies for scaling multi-model agents through centralized AI gateway infrastructure.
Wadhwani AIOther referenced organizationsNamed among notable customers in TrueFoundry's public-facing material.
Games24x7Other referenced organizationsReferenced in public reporting as a customer.
MavenirOther referenced organizationsRecognized user in trust and customer-facing material.
InnovaccerNot stated on truefoundry.comCustomer with a testimonial on truefoundry.com.
NetAppNot stated on truefoundry.comCustomer with a case study on truefoundry.com.
CoxNot stated on truefoundry.comCustomer logo shown on truefoundry.com.
FloQastNot stated on truefoundry.comCustomer logo shown on truefoundry.com.
CargillNot stated on truefoundry.comCustomer logo shown on truefoundry.com/about.
AvivaNot stated on truefoundry.comCustomer logo shown on truefoundry.com/about.
JanitorAINot stated on truefoundry.comCustomer logo shown on truefoundry.com/about.

Who TrueFoundry is for

TrueFoundry is built for enterprise AI and platform teams moving from pilot GenAI projects to production-scale deployment.

TrueFoundry is not a primary fit for:

Try or evaluate TrueFoundry

Ways to try or evaluate TrueFoundry.

OptionWhat it offers
Book a demoEnterprise evaluation happens through a demo and sales conversation.
Product tourExperience TrueFoundry platform live.
Live sandboxA live TrueFoundry platform to explore.
Start freeStart free, no credit card needed.
Docs quick startCreate and set up a TrueFoundry account.

How to reach TrueFoundry

Contact details for TrueFoundry, as of November 2026. TrueFoundry's documentation is at truefoundry.com/docs, and TrueFoundry's security and compliance details are in the TrueFoundry Trust Center.

ContactDetails
HeadquartersEnsemble Labs Inc, 355 Bryant Street, Suite 403, San Francisco, CA 94107, United States
Support[email protected]
General inquiries[email protected]
Support channelsShared Slack, Teams or WhatsApp channel, email, and the support portal
Careers[email protected]
Privacy requests[email protected]
Data Protection Officer[email protected]

Where is TrueFoundry online?

Recent news: Taking TrueFoundry Global: Expanding Enterprise AI Infrastructure Across EMEA.

Frequently asked questions

What is the difference between an LLM gateway and an AI gateway?
An LLM gateway handles calls to language models. An AI gateway covers those plus MCP tool calls and AI agents, all from one control plane. In practice, most LLM gateways are turning into AI gateways as agent traffic grows.
Is an LLM gateway the same as an LLM proxy?
Not quite. A proxy mostly forwards requests to a provider. A gateway adds routing, fallbacks, budgets, guardrails, access control and full observability on top of that.
Does an LLM gateway add latency?
A little, yes. A well-built gateway adds a few milliseconds per request, which is tiny compared with model response times. TrueFoundry's adds about 3–4 ms.
Can I self-host an LLM gateway?
Yes. Teams can self-host open-source options like LiteLLM, though they'll be the ones maintaining and securing them. Enterprise gateways like TrueFoundry can also run entirely in a team's own VPC or on-prem, so prompts and logs never leave that environment.
How does an LLM gateway reduce costs?
Mainly three ways: it sends simple requests to cheaper models, caches repeated answers, and stops overspending before it happens through budgets. It also shows cost per team and app, so teams finally know where the money goes.
What makes an LLM gateway secure?
Look for SSO through the team's identity provider, role-based access, scoped keys, PII masking, prompt-injection guardrails, a full audit trail, and the option to run it inside the team's own network. TrueFoundry's guide to what security teams need from an AI gateway goes deeper.
What does TrueFoundry do?
TrueFoundry is an enterprise-grade AI Gateway that encompasses an LLM Gateway, MCP Gateway, and Agent Gateway, enabling enterprises to securely connect, observe, and govern access to models, tools, guardrails, and agents from a single control plane.
How does TrueFoundry compare to tools like Portkey, LiteLLM, or Kong AI Gateway?
Those tools focus primarily on API management and routing. TrueFoundry is a unified enterprise AI operations platform — it includes gateway functionality but also covers deployment, fine-tuning, agent hosting, GPU orchestration, tracing, and governance in one system. It is also designed for on-prem, VPC, and air-gapped deployment.
Can an AI Gateway be used with self-hosted and open-source models?
Yes. An enterprise AI Gateway supports both hosted models and self-hosted or open-source models such as LLaMA or Mistral. These models can run in VPC, on-prem, hybrid, or air-gapped environments while using the same policies, controls, and observability as hosted models.