An agent is repeatedly requesting an operation that policy refuses. The on-call engineer can suspend the identity, revoke it, or change the rules. Which action contains the problem without creating a larger one?
First check why the request failed. The rules may have correctly refused it, the application may be retrying the wrong operation, or the gateway may be unable to load the policy. Don't change permissions until you know which problem you're solving.
This runbook is a recommended operating procedure. Rehearse it against a controlled workload before relying on it during an incident.
Find the reason for the failure
Preserve the agent ID, relevant request and audit identifiers, timestamps, denial reason, and configuration context. Check whether the call was stopped before dispatch or encountered an execution problem later.
Dome's audit model separates decision result from execution error. Use that distinction to avoid treating every failed request as a permission problem.
If the request was correctly refused, investigate why the application keeps making it. Do not grant the operation merely to end the retry loop. If policy freshness or evaluation failed, restore the control's health instead of rewriting business permissions.
If a guard caused the refusal, check its direction. A request block stops forwarding; a response block may occur after the backend has acted. If represented identity could not be verified, repair that identity path rather than adding a permit. Record the finding and owner before choosing a containment action.
What do you need to stop?
| Situation | Candidate action | Recovery consequence |
|---|---|---|
| One operation is inappropriate, but other work should continue | Narrow or forbid that operation in rules | Verify the effective policy and the remaining legitimate work |
| Trust in the running agent is temporarily uncertain | Suspend the agent identity | Reversible; decide explicitly whether active descendants must also be suspended |
| One agent key is exposed, while the workload remains trusted | Revoke/rotate the affected key and inspect dependent runtimes | Replace credential copies; verify the old credential path stops working |
| Agent service is permanently retired or identity trust cannot be restored | Revoke the agent | Permanent and cascades to every descendant |
| A shared backend secret is compromised | Rotate/revoke at the backend and update affected connections | Includes users outside the one agent identity |
| Work already accepted by a backend may be harmful | Invoke the backend's cancellation or reconciliation procedure | Identity controls alone cannot undo accepted work |
Use the table to choose an initial response. If you don't yet know how far a compromise reaches, you may need to stop more than one operation or identity. Record what you need to establish before restoring access.
The operational inventory should identify the people who can carry out each action, including runtime and backend interventions outside Dome.
Check the child agents before suspending or revoking
Dome's agent lifecycle makes suspension reversible. A requested suspension cascade covers active descendants. Reactivation applies to the target identity alone; it does not automatically reactivate the whole suspended hierarchy.
Agent revocation is terminal and always reaches every descendant, regardless of state. Do not use it as a convenient temporary pause, and do not confuse it with revoking one API key.
Inspect the hierarchy before selecting the control. A parent-child record also does not establish that the parent granted all of the child's permissions.
Did new requests actually stop?
After the management action, test or observe a new request on the affected path. Record the result and the time. For a rules change, account for synchronization; a saved configuration is not evidence of instantaneous runtime enforcement.
Check other credentials or routes identified in the service record. If the suspected workload can still use a direct backend credential, an agent-identity suspension does not answer that separate path.
Also inspect work already in progress. Stopping future governed calls does not terminate the agent process or reverse a remote transaction. The incident owner needs a runtime stop or backend recovery action when the consequence requires it.
Decide what must be true before restarting
Before reactivation, name the condition that made the incident safe to resume: repaired code, replaced credential, corrected policy, or reconciled backend state.
Exercise one legitimate request and the forbidden request that mattered to the incident. Inspect descendants individually after a cascaded suspension. Avoid restoring access to every identity merely because the parent has recovered.
Close the incident with the containment evidence, residual effects, changed configuration, and an updated regression case. Use the agent management guide for the actual lifecycle and credential operations. Record which requests stopped, which work may already have completed, and who approved restarting.