AI gateway vs. LLM gateway vs. MCP gateway

An AI gateway sits between an AI application and the services it calls. The term is broad: an LLM gateway handles requests to language models, while a Model Context Protocol (MCP) gateway handles protocol traffic, such as discovering and calling tools.

Two products called “AI gateways” may do different jobs. One might route prompts to a model, while another also checks which tools an agent can call.

Cloudflare's AI Gateway documentation uses the term for model traffic. Dome documents separate paths for tool and model requests. When comparing gateways, start with the requests your application needs to send.

Explore Dome: Dome Platform · Docs: Tools gateway.

Follow a model request and a tool call

Suppose a support assistant sends prompts through an LLM gateway but calls a ticket service directly. The gateway can apply checks to the model requests, but it doesn't receive the ticket calls.

Model brokering manages access to models and selects where to send each request. It can be part of an LLM gateway or a separate service.

The MCP entry describes how hosts, clients, and servers work together. Model routing explains how a system chooses which model receives a request.

Explore Dome: Model Broker · Docs: LLM gateway.

Check what the gateway does with each request

For a gateway handling ticket calls, check how it identifies the caller, decides whether to allow the operation, and handles the credentials used to connect to the ticket service. For the model call, check what it inspects in the prompt and response and how it limits usage. Translating a request into a provider's API format is a separate task from checking permissions, inspecting content, or limiting usage.

For guardrails, check whether the gateway inspects requests, responses, or both. Check which content formats it supports and what happens when a check fails. For audit logs, look for the attempted operation and its outcome. For example, can you tell whether the assistant's ticket update was allowed and whether it succeeded?

Keep the answers for model requests and tool calls separate. That shows which checks apply to each part of the application.

Dome's Tool Gateway and Model Broker documentation describes the paths for tool and model requests, including the protocols and controls supported on each path.

Explore Dome: Tool Gateway · Docs: Guards.

Check retries and calls that bypass the gateway

Follow one model request and one ticket call through the application. Note where each goes and which credential it uses. If the application retries a failed request through another connection, check whether that attempt still passes through the intended gateway.

A provider key stored in the application may let it call a model directly. The ticket tool may also call other services. Those calls may not pass through the same gateway. Include those calls when you map where requests go.

For each route, test a request that should be allowed and one that should be denied. Confirm where the denied request stops and what the logs record. Runtime policy enforcement explains how to check that a policy decision actually changes what runs.

Use that request map when choosing a gateway or reviewing a deployment. It should show which calls pass through the gateway, which checks apply to them, and which calls need controls elsewhere.

Explore Dome: Authorization · Docs: Authorization model.