LLM routing and model routing policy

LLM routing, also called model routing, selects a model or serving endpoint for a request. A model routing policy defines which destinations are eligible and how the system chooses among them.

A model router can compare cost, latency, or load after checking requirements such as approved providers, allowed data destinations, and required capabilities.

Explore Dome: Model Broker · Docs: Configure pools.

Decide which destinations are allowed before choosing one

Suppose an application summarizes internal engineering reports. It has three possible model connections. One is approved for the reports, another lacks the required context capacity, and a third is outside the approved data destination.

In this hypothetical example, only connection A is eligible. A lower price or faster response would not make B or C acceptable for these reports.

Use a trusted source for each report's data classification, and decide what happens if that classification is missing. Don't let an unverified caller-supplied label determine where the report can be sent.

Explore Dome: Authorization · Docs: Authorization model.

Routing strategies optimize different things

Once unsuitable destinations are excluded, routing can serve different goals:

• Model load balancing distributes requests across serving capacity.
• Capability routing selects a model expected to support the task.
• Cost-aware routing uses price or estimated cost to choose a destination.

Implementations may use ordered preference, weights, request attributes, or suitability estimates to make that choice.

These strategies do not establish equivalent behavior between models. A model may accept the same API shape while differing in tool-call behavior, output style, context limits, or provider-specific options. Multi-provider AI explains the compatibility differences to check.

Dome's pool and model-gateway documentation describes its supported routing and authorization behavior. Check the available strategies against your requirements; a router does not necessarily predict answer quality.

For the engineering-report summary, test whether the selected model produces an acceptable summary, not just whether the request returns successfully.

Explore Dome: Model Broker · Docs: LLM gateway.

Record enough to explain the choice

Record both the requested destination and the connection that actually received the report, along with the policy and any failed attempts. This lets an operator trace where the data went, including after a route change.

Test the routing policy with cases that distinguish eligibility from preference:

• The preferred destination is available and permitted.
• It is available but explicitly forbidden.
• Required classification context is missing.
• Another permitted destination is cheaper.
• No destination satisfies the hard constraints.

Define the expected outcome for each case before deployment. If no destination meets the requirements, the system should follow an agreed procedure for failing the request or escalating it. It should not try any endpoint that happens to respond.

Dome documents authorization against the actual model connection it dispatches. See failover and fallback for what changes when the selected destination fails.

Explore Dome: Model Broker · Docs: Configure pools.