What is the difference between an LLM router and an LLM gateway?

In practice the terms are used interchangeably. Where people do draw a line, routing emphasizes choosing between models on cost, latency or capability, while gateway emphasizes the control functions around the call: authentication, limits, filtering and audit. Most products in the category do both.

Why not call the provider API directly?

Direct calls mean every agent holds provider keys, model choice is hard-coded, an outage at one provider is an outage for you, and spend is visible only as one large bill per provider rather than per agent or per team.

Does an LLM router govern tool calls?

Usually not. LLM routers govern the model channel. Tool calls, which is where an agent takes actions with real-world consequences, typically pass through a separate MCP gateway with its own rules and its own logs.

Is Dome an LLM router?

Dome's Model Broker is an LLM router, and is part of the overall Dome platform. It accepts OpenAI- and Anthropic-compatible calls, routes to a provider or pool, and injects credentials at egress. It also governs tool calls, agent identity and the caller an agent acts for, under one rule language and one audit trail.

Which LLM router should I choose?

LLM routers tend to have similar capabilities but targeted at specific users and focus only on model routing. Dome is for enterprise AI platform teams building agent services for the rest of the business: dozens of agents across many teams, model spend that has to be attributed to the agent rather than the key, tool calls governed by the same rules, and a security function that expects one answer to who did what, on whose behalf, and under which rule.

What is an LLM router?

An LLM router is a single endpoint that sits between agents and the model providers they call. It presents one API shape, chooses which model serves each request, injects the provider credential, and reports usage consistently across providers.

Read the field guide See Dome’s LLM router Set up a model broker

Why use Dome for a model router?

If one application needs to reach several models, a router is the right size of answer. Dome is for the enterprise AI platform team that owns model access for the whole business, and has to account for every token.

Which means the model is not the only thing under governance. Spend attributes to the agent rather than the key. The person the agent acts for is part of the authorization decision. Tool calls run under the same rules and land in the same audit trail. One object model, one rule language, one record. How that works.

One provider is a decision you make once

Calling a provider API directly is the right first move. It stops being right at a predictable point.

The first pressure is availability. A provider rate limit becomes your outage, and the fix has to live somewhere other than in every agent. The second is cost: token spend arrives as one bill with no way to attribute it to the team, the agent, or the workflow that caused it.

The third is choice. Models change every few months, and the good answer differs by task. A router turns each of those from a code change into a configuration change, and gives you one place to see what is actually being spent.

What an LLM router does

Provider abstraction

One request shape in front of many providers. The agent speaks a single API and the router translates, so changing provider is a configuration change rather than a code change.

Routing strategy

Choosing which model serves a request: cheapest that will do, fastest available, pinned to a region, or escalating to a stronger model only when the cheap one fails.

Failover

Providers have outages and rate limits. Grouping models into a pool lets a request fall through to the next candidate without the agent knowing anything happened.

Credential isolation

Provider API keys live at the router and are injected at egress. Agents never hold them. Rotation is one change in one place.

Spend control

Token and cost ceilings, ideally per agent and per workspace rather than per API key. This is where most of an agent estate's cost actually accrues.

Normalized telemetry

Token counts and latency reported the same way regardless of provider, so usage across a mixed estate can be compared and charged back.

Where LLM routers differ

The capability lists converge. The differences that matter are underneath them.

Identity-scoped or key-scoped limits

Most routers cap spend against an API key. A key is not an agent, and it is certainly not a team. If a budget cannot bind to the identity that made the call, attributing or containing spend means reconciling two systems by hand.

Policy per call, or configuration per key

The useful question is not whether a model can be disabled, but whether a rule can say which agents may call which models, under which conditions, and be evaluated on every single request.

Whether tool traffic is governed too

Model calls are half of what an agent does. The other half is tool calls, which are where the actions with real-world consequences happen. A router that governs only models leaves the rest to a second system.

What the audit record contains

Usage metering is not an audit trail. Attributed audit records which agent called which model, on behalf of whom, under which rule, with what outcome, in the same vocabulary as everything else the agent did.

Caching and its trade-offs

Exact-match caching is safe and boring. Semantic caching is neither: it can serve a response generated for a different caller, which is a governance question and not only a performance one.

Where it runs

A router sits in the path of every prompt and response. Whether it is hosted or self-managed decides what crosses your boundary, and that constraint rarely comes from engineering.

Who builds LLM routers

A short, honest map of the category. Products are described by shape rather than by feature list, because shape is what stays true.

Product	What it is	Best suited to
LiteLLM	The open-source default. Broad provider coverage, library-shaped, self-hostable.	Teams who want provider abstraction in their own stack and are happy to operate it.
Prisma AIRS	Formerly Portkey, now part of Palo Alto Networks. A routing layer inside a security suite.	Organizations consolidating on Palo Alto for AI runtime security.
Kong AI Gateway	API-management lineage extended to model traffic, on the same routes and plugin chain.	Organizations already running Kong who want models on that fabric.
OpenRouter	Aggregation-first. One account and one endpoint in front of a very wide model catalog.	Reaching many models quickly without negotiating provider contracts.
Dome	An agent operations platform. The LLM router is one control point of three, alongside tool traffic and agent identity.	Enterprise platform teams running agent services for the whole business: many agents, many teams, models and tools both in scope, one audit trail.

Last updated 11 August 2026, based on publicly available information about these products. Spot something out of date? Tell us and we will correct it.

Questions worth asking

1Can a spend cap bind to an agent, or only to an API key?
2Can a rule say which agents may call which models, evaluated per call?
3Who governs the tool calls the same agents are making?
4Does the audit record name the agent and the person it acted for?
5On failover, is the fallback model authorized in its own right?
6If semantic caching is on, whose response can another caller receive?

Routing in service of governance

Most products in this category began as routing and added governance afterward. Dome began at the other end, and it changes what the router is for.

A router that exists to route treats spend and access as settings on a key. A router that exists to govern treats the model as one object among several that the same rules have to reach. That is the difference between metering usage and being able to say which agent spent it, on whose behalf, and under which rule.

So Dome’s LLM router, the Model Broker, is one control point of three. Spend attributes to the agent, not the key. Every failover candidate is authorized in its own right. The same rules that decide which model an agent may call decide which tools it may reach. The agent’s call shape never changes. What changes is that somebody can finally answer for it.

Quotas bind to identity

Spend and token ceilings attach to the agent and the workspace, not to an API key. Attribution and containment stop being a reconciliation exercise across two systems.

Failover is re-authorized

Each candidate in a pool is authorized in its own right rather than inheriting the permission of the first choice. A fallback cannot quietly widen what an agent may reach.

One record with tool calls

Model calls and tool calls share an audit vocabulary, so the question of what an agent did has one answer in one place rather than two partial ones.

Read the field guide Explore the Model Broker Set up a model broker Get started

Agents Need to Talk to Models is a 20-minute field guide to the Model Broker: connecting providers, building pools, expressing model choice as routing policy, and capping spend. No form, no gate.

Common questions

What is an LLM router?: An LLM router is a single endpoint that sits between applications or agents and the model providers they call. It presents one API shape, chooses which provider and model serves each request, injects the provider credential, handles failover, and reports usage consistently across providers.
What is the difference between an LLM router and an LLM gateway?: In practice the terms are used interchangeably. Where people do draw a line, routing emphasizes choosing between models on cost, latency or capability, while gateway emphasizes the control functions around the call: authentication, limits, filtering and audit. Most products in the category do both.
Why not call the provider API directly?: Direct calls mean every agent holds provider keys, model choice is hard-coded, an outage at one provider is an outage for you, and spend is visible only as one large bill per provider rather than per agent or per team.
Does an LLM router govern tool calls?: Usually not. LLM routers govern the model channel. Tool calls, which is where an agent takes actions with real-world consequences, typically pass through a separate MCP gateway with its own rules and its own logs.
Is Dome an LLM router?: Dome's Model Broker is an LLM router, and is part of the overall Dome platform. It accepts OpenAI- and Anthropic-compatible calls, routes to a provider or pool, and injects credentials at egress. It also governs tool calls, agent identity and the caller an agent acts for, under one rule language and one audit trail.
Which LLM router should I choose?: LLM routers tend to have similar capabilities but targeted at specific users and focus only on model routing. Dome is for enterprise AI platform teams building agent services for the rest of the business: dozens of agents across many teams, model spend that has to be attributed to the agent rather than the key, tool calls governed by the same rules, and a security function that expects one answer to who did what, on whose behalf, and under which rule.