LLM failover vs. fallback
LLM failover moves a request to another serving destination after a defined failure. LLM fallback uses a configured alternative when specified conditions are met, which may include failure or a reduced-service choice. Vendors use these terms differently, so the implementation's trigger and behavior matter more than the label.
A retry repeats an attempt. A failover can be a retry against another destination. Before retrying, check that the alternative is permitted, supports the request, and will not repeat an action that already completed.
Explore Dome: Model Broker · Docs: LLM gateway.
Define the trigger before choosing the alternative
Suppose a document-summary service has a primary model connection and a backup. “Use the backup if the primary fails” leaves several important cases unspecified.
| Condition | Decision to define |
|---|---|
| Temporary provider unavailability | Whether another eligible connection may be tried |
| Invalid request shape | Whether to correct the request rather than repeat it |
| Authorization refusal | Whether to stop instead of seeking a more permissive route |
| Usage limit reached | Whether another independently permitted budget exists |
| Partial streamed output already delivered | Whether the client must treat the response as incomplete |
For the summary service, an outage might justify trying the approved backup. An authorization refusal should not send the same document to a more permissive destination. Verify how the selected implementation handles each condition.
Check every alternative against the same destination requirements used for the first choice. Model routing explains how to define those requirements.
Explore Dome: Authorization · Docs: Authorization model.
Set one retry budget for the whole request
A retry budget caps the number of attempts or the time spent retrying. Consider retries in the client, gateway, and provider library together. If each layer retries independently, the combined number of attempts may be much larger than an operator expects.
A circuit breaker is a related reliability mechanism that temporarily stops attempts to an unhealthy dependency and later probes for recovery. Its purpose differs from choosing a suitable alternative model.
For the summary service, set a maximum total time, decide which failures allow a retry, and specify what to record about each attempted destination. Also include cost: unsuccessful or abandoned attempts may still consume billable resources depending on provider behavior.
When retries run out, report the summary as incomplete rather than making the application infer completion from missing output.
Explore Dome: Model Broker · Docs: Configure pools.
Check for partial output and actions already taken
Once a response has begun reaching the caller, replacing it with another model's output can break the application's expectations. Dome's LLM gateway documentation limits streaming failover to the period before the first response byte reaches the caller. Other implementations may handle streaming failures differently.
For an agent that also calls tools, record which proposed calls the application actually executed and what the backend returned. A retry must account for actions already accepted, not just the model request that failed.
For a hypothetical booking assistant, check the booking service before retrying a booking whose result is unknown. If you use idempotency to prevent duplicate bookings, check that it covers this operation and its retries.
Build a failure table that records the trigger, permitted alternatives, attempt limits, handling of partial output, and stop conditions. See multi-provider AI for the behavioral differences to check between destinations.
Explore Dome: Model Broker · Docs: LLM gateway.