Announcing Dome Managed Agents.

Try it now

Review a provider policy change separately from model quality

Review split between model evaluation results and changed provider treatment constraints.

A provider updates regional processing terms while model quality remains stable. The immediate question is usually framed as a configuration task. The safer question is what must be true before the next action is allowed, and what evidence will explain the result afterward.

None of this needs a second policy system. It is a small operating contract around one change: name the object, the decision point, the allowed effect, the owner, and the evidence that the check happened. The contract makes the change reviewable when the happy path succeeds and when it does not.

Start with the action, not the setting

Write down the action in a form a reviewer can test. For this example, the action is: review a provider policy change separately from model quality. Then separate three questions that are easy to collapse:

QuestionWhat to recordWhy it matters
Who or what is acting?Stable identity and accountable ownerA label or environment name can change without changing authority
What effect is proposed?Target, scope, and expected resultA tool name alone rarely captures the full consequence
What check admits it?Current policy, guard, quota, or review conditionA permitted call can still be filtered, delayed, or fail downstream

That separation keeps the review focused. A successful request does not prove that the right information was returned, that the backend completed the effect, or that a later retry will be safe. Conversely, a denial is useful only when the decision facts are available to the person who owns the next step.

Use a small evidence card

Keep one short record beside the implementation work:

  • Intent: A review split between model evaluation results and changed provider treatment constraints.
  • Scope: the smallest target set that exercises the change.
  • Owner: the team that can pause, reverse, or explain the change.
  • Decision evidence: the identifier, policy or control version, and outcome; do not copy secrets or unnecessary payloads.
  • Failure path: a defined response when the check cannot be completed or the downstream service refuses the request.

The card is deliberately smaller than a runbook. Its job is to make the boundary visible. If the action reaches a tool or model through Dome, confirm the current product behavior in the identity, authorization, guard, quota, and audit documentation.

Test the boundary with a contrasting case

One permitted example is not enough. Pair it with a nearby case that should change the decision: a different target, a changed owner, an expired approval, a larger effect, or a request that arrives after the original context is stale. The point is not to maximize negative tests. It is to make the condition that matters visible.

Finish with a reversible next step

Before widening the scope, confirm that the evidence card is complete and that the failure path has an owner. Run the smallest safe check, save the resulting decision reference, and review the downstream result separately from the authorization outcome. If the result is unclear, stop at the boundary instead of treating a timeout or partial response as success.

The useful result is not a prettier configuration. It is a change another operator can understand, test, and reverse without guessing what the agent was allowed to do.

More from the blog.

tenant northdata_class restrictedmodel_route private
system_prompt v12 → v13model_route unchangedreview separate
Primary modelAllowed fallback

Next perspective,straight to your inbox.

Subscribe

Each new perspective and field guide, the week it publishes. Nothing else.

Product updates

Release notes and platform changes live on the updates page.

Every agent you run,under one system of control.