Explainer
Design a model pool as an application contract
Suppose a support application asks for a model named “support-summary.” The platform team wants to switch providers behind that name without changing the application.
That can work, but both providers need to handle the same data and produce a summary the application can use. Before adding a model to the pool, agree on those requirements with the application owner.
What must every model in the pool do?
For a hypothetical support-summary workload, the contract might look like this:
| Contract item | Example agreement | Who maintains the evidence |
|---|---|---|
| Purpose | Summarize a ticket; do not issue tool calls or perform customer actions | Application owner |
| Input | Approved classes of support data, with the required regional treatment | Data owner and platform team |
| Output | A structured summary with mandatory fields and an explicit failure path | Application owner |
| Compatibility | Each member passes the same representative request and response checks | Application test owner |
| Selection | Prefer the primary tier; use only approved fallback members | Pool operator |
| Change | Re-run compatibility checks before adding or replacing a member | Pool operator and application owner |
| Investigation | Identify the pool requested and the connection that actually served it | Platform operations |
Use this as an agreement with the application owner, not as fields to paste into Dome. Your team still approves where data may go and tests whether each model's output is good enough.
What happens when the application names the pool?
Dome's Model Broker brings model connections and pools onto the same access, authorization, and audit layer used for governed tool traffic. An agent can request a pool by name while operators change the connections behind it.
Application requests "support-summary"
|
Gateway admission and pool resolution
|
Select an eligible member
|
Authorize that upstream before dispatch
|
Provider response and audit evidenceMembership and permission remain separate. Being in a pool does not excuse a model from rules. The LLM gateway evaluates the actual upstream before sending the request, including each fallback candidate.
The application can keep using the same name. The selected model still has to pass authorization.
Choose what routing should optimize
Dome documents four routing strategies:
| Strategy | Useful selection signal | Do not read it as |
|---|---|---|
priority_weighted | Preference tiers, with weighted selection within the lowest tier | A measured quality ranking |
round_robin | Rotation across members | Equal cost or equal response time |
least_loaded | Fewest in-flight requests | A latency benchmark |
most_quota_remaining | Remaining per-model cost-quota headroom | The cheapest model for the task |
Choose the signal that matches the operating objective. If quality is the deciding factor, establish it through application evaluation and member approval. Do not expect the routing strategy to infer it.
Dome also supports conditional pool selection. An exact pool or connection name takes precedence over conditional routes, so the caller's naming convention is part of the contract. Review direct-model exceptions deliberately; an application that names a connection may not exercise the pool behavior operators expect.
Check the response, not just the connection
Two providers may accept similarly shaped messages but differ in their supported options or outputs. Dome preserves provider-specific request options on same-dialect dispatch, while cross-dialect translation can lose extensions. That matters if the application depends on an option the common representation does not carry.
Keep the contract at the application boundary: required fields, tool behavior when applicable, error handling, and acceptable results. Test every member against it, including the member reached only during failure. A fallback that returns syntactically valid but unusable output has not restored the service.
Direct model selection can be the clearer choice for a provider-specific feature or a workload whose behavior has been validated against exactly one connection. Keep the exception visible instead of hiding it behind a pool that suggests interchangeability.
Compare the controls on both provider paths
Before adding the second provider, run the same harmless request through each connection. Keep the agent and required verified user context fixed. Check the actual selected destination against rules, the assigned request and response guards, and any applicable shared quota. Confirm that the record identifies which connection served the request.
Write the results beside each member's compatibility evidence: identity preserved, destination authorized, content handled as required, allowance enforced, output accepted. A route that passes permission checks can still fail the output test. Investigate the difference without weakening a rule to fix an incompatible request.
Review a member change like a service change
Before adding a member, attach its data approval, compatibility evidence, policy expectations, and removal procedure to the change. Preserve the previous membership so recovery does not depend on memory. Inspect the served connection in runtime evidence after rollout.
Keep the application requirements and member approvals together. That gives the next operator a basis for deciding whether a replacement model belongs in the pool. Use the pool configuration model to define one pool only after the application owner can state what every member must preserve.