A fallback connection returns a successful response during setup. That establishes connectivity. It does not show what happens when the primary fails halfway through a stream or when the only healthy fallback is forbidden.
Suppose the application needs a complete classification result. A fallback that returns half a response hasn't recovered that service. Test those failure cases before the real outage, and keep the results in a run sheet the on-call team can use.
What counts as a recovered request?
Use the pool contract to name the input, required output, approved members, and acceptable failure behavior. Use the failover decision table for the conditions under which another attempt is appropriate.
For a hypothetical classification service, recovery means the client receives a complete result satisfying its schema within its agreed deadline. A healthy provider response with a missing classification field is not service recovery.
Choose an isolated rehearsal environment, synthetic data, a cost allowance, and a stop condition. Do not inject failures into a shared production route as an improvised test.
Prepare a run sheet
Record the client version and retry configuration, gateway and pool, member list, strategy, failover limit, applicable rules and quotas, and the application deadline.
Then run one changed condition at a time. The following is a proposed protocol; its observations remain to be collected.
| Rehearsal | Change introduced | Observe at the client | Evidence to retain |
|---|---|---|---|
| Baseline | No induced failure | Complete valid output | Requested pool and served connection |
| Primary unavailable | Controlled failure before a usable response | Approved fallback or defined terminal failure | Attempt sequence and final upstream |
| Forbidden fallback | Candidate is reachable but policy-refused | No dispatch to that candidate | Refusal and any later permitted route |
| Early stream failure | Failure before the first response byte | Eligible recovery within the budget | Time and source of first delivered output |
| Late stream failure | Failure after output begins | Explicit partial-response failure | Delivered prefix, terminal error, application handling |
| Member removal | Remove the selected test member from the pool | New requests stop choosing it after configuration is applied | Observed routes and configuration-change time |
| Return to service | Restore the approved member configuration | Intended distribution resumes | Served connections and application-valid outputs |
Change one thing per run. If you remove a member, change a rule, and upgrade the client together, you won't know which change caused the result.
Does the fallback return something the application can use?
Run representative application cases against every fallback. If the client depends on provider-specific options, test the actual request shape through the intended route.
Dome's LLM Gateway documentation distinguishes same-dialect forwarding from cross-dialect rebuilding, where provider-specific extensions can be lost. A successful translated request does not establish that the option the application needed survived.
Keep output validation in the run sheet. Routing success and application acceptance should be separate recorded observations.
Practice removing a member
The current pool documentation says the stored member enabled field is ignored during dispatch. It directs operators to remove a member to stop routing to that connection through the pool.
Use the documented effective operation in the rehearsal and verify new requests. Removing a connection from one pool does not disable it everywhere else. Account for other pools and direct routes when the incident objective is broader.
Do not mistake a saved configuration for the observed end of traffic. Record when the change is made and what subsequent requests actually do.
Count retries in the client as well as the pool
The pool failover limit bounds additional member attempts. It does not, by itself, describe the application's complete retry policy or service deadline.
Inspect the SDK's retry behavior alongside the pool configuration. The application may initiate a new request after the broker has already exhausted its candidate set. Set a limit for those client retries too, then count the total attempts during the test.
For streaming, stop interpreting recovery as transparent once content has reached the caller. The application must know what to do with an incomplete response.
Keep failures and unanswered questions in the report
For every row, record expected behavior, actual behavior, application-valid result, served destination, errors, and unresolved questions. Retain failed and inconclusive runs.
Promote the configuration only after it meets the application's stated requirements. The run sheet should also become the incident procedure: which member to remove, what a partial failure looks like, and which evidence confirms that service has recovered.