Guide
When a model pool should fail over, skip a candidate, or stop
The primary model is unavailable. One fallback is permitted and compatible; another is healthy but prohibited for this workload. A healthy model isn't necessarily one this agent is allowed to use.
When a request fails, should the pool try another model, skip one, or stop? Those are different decisions. The table below helps you keep a provider outage separate from a policy refusal.
Check which destinations are approved
The model-pool foundation defines membership, compatibility, and routing preference. Here, assume the organization has already reviewed the candidate destinations and configured the relevant restrictions.
Dome's LLM gateway authorizes the actual upstream selected for dispatch. Each fallback candidate receives its own authorization check.
Keep destination labels and approvals up to date. A policy can enforce the supplied model attributes; it does not independently verify a provider's contractual data treatment.
Use a decision table before configuring retries
| Condition | Documented behavior to preserve | Operating implication |
|---|---|---|
| Primary authorization denies a non-streaming request | The denial is terminal | Do not treat policy refusal as a provider outage |
| Primary encounters an eligible transient failure | Further eligible members may be attempted within the configured limit | Recovery remains inside the approved candidate set |
| A fallback candidate is refused by policy | Skip that candidate rather than dispatching to it | A refusal must not become a successful route-around |
| One member's model quota is exhausted | Pool resolution can skip that member if another eligible one remains | Distinguish a member budget from an aggregate ceiling |
| An applicable workspace, agent, pool, or gateway quota is exhausted | Affected calls are refused | Changing the member does not remove the broader budget |
| A streaming failure occurs before the first response byte | Failover can still be eligible | Preserve the configured attempt limit and authorization checks |
| A streaming failure occurs after bytes have reached the caller | The stream terminates | The application must handle partial output; do not splice a new completion into it |
| No eligible member remains | The request cannot be fulfilled through the pool | Return the defined failure instead of broadening eligibility |
Quota behavior is described in the quota documentation. The table deliberately separates refusal from availability failure.
Follow a request through three candidates
Consider a hypothetical non-streaming request to a pool containing A, B, and C.
A is authorized, but the upstream encounters a failure eligible for fallback. B is then considered and refused by policy. B must not receive the request. If C is eligible, permitted, and within the configured attempt budget, it may serve the request.
B's denial still holds: B never gets the request. C serves it under its own permission, not by overriding the restriction on B.
Now change just the first condition: authorization denies A before dispatch. The documented non-streaming behavior stops there. The caller should not automatically turn that policy decision into a search for a more permissive pool.
Finally, consider a streaming response from A that has already delivered content. A later upstream failure terminates that stream. Even a compatible C cannot make the already-delivered partial response disappear.
Count the application's retries too
Dome's pool reference defines a zero failover limit as primary-only, and a numeric limit as that many additional members after the primary.
The application or SDK may also retry. Review the combined behavior so one user request does not generate an unexpected series of pool attempts. Decide which error classes justify another application attempt and which require an operator or a different user action.
For workflows that act on model output, partial-output handling also needs a clear contract. The client should not treat an interrupted response as a complete instruction simply because some bytes arrived.
Write down what your application should do
For each row, write the application's expected response, the operator who owns recovery, and the evidence that distinguishes the outcome. Use synthetic data when rehearsing the cases.
Before release, your team should be able to explain why a timeout can lead to another attempt, a denied primary stops the request, and a broken stream leaves the caller with partial output.