Policy testing vs. policy simulation

Policy testing evaluates rules against selected inputs and expected outcomes. Policy simulation explores the decisions a policy would make without performing the protected operation. Historical simulation commonly compares a candidate policy with decisions recorded under an earlier policy.

Validation, testing, and simulation answer different questions. A policy file can pass validation and still allow or deny the wrong requests. A proposed policy can match past decisions while mishandling a case that has never occurred.

Explore Dome: Authorization · Docs: Simulate rules.

Choose the right test for the change

Suppose a team narrows an agent's access from every support queue to one assigned queue.

For the queue restriction, an expected-result test can confirm that the assigned queue is allowed and another queue is denied. Historical comparison shows which past requests would now receive a different decision. Open Policy Agent's testing guide illustrates explicit allow and deny assertions.

Dome's simulation documentation distinguishes testing a constructed request from replaying candidate rules against recorded authorization decisions. Check which policy and inputs a mode uses before interpreting its result.

Explore Dome: Authorization · Docs: Simulate rules.

Matching the old policy doesn't mean the new one is right

If the old policy allowed every queue, matching its decisions would preserve the very access you meant to remove. Review changed results against the new requirement, not just against the old policy.

For the queue restriction, a changed result from allow to deny may be the intended improvement. A changed result for the assigned queue may be a regression. An unchanged decision with missing queue context may indicate that both policies share the same faulty assumption.

Recorded traffic also has coverage limits. It may omit unusual callers, new resources, or cases excluded by earlier admission checks. Define the history window and input fields used by the replay.

A replay can use the queue owner recorded in an old request; it doesn't check today's ownership or execute the tool. Test those parts separately.

Explore Dome: Audit · Docs: Audit events.

Keep a small set of repeatable tests

For the hypothetical change, test the assigned queue, another team's queue, missing or malformed queue information, and an applicable rule inherited from a higher scope. Add an allowed request that represents real work.

For each test, record the trusted source of every decision input. If production accepts caller-supplied data where the test used a verified value, the rule may permit a request it should refuse.

Then retain the expected outcomes with the policy version. Future changes can run the same tests and add cases discovered in incidents or reviews. Historical comparisons complement this regression set by showing the effect on recorded traffic.

Dome supports synthetic evaluation and historical decision comparison for rules. Those tools are useful evidence within the broader policy-as-code change process. Human review still decides whether the expected outcomes match the business requirement.

Keep both the expected-result tests and the historical comparison with the proposed change. Before release, also check that the deployed system enforces the decisions.

Explore Dome: Authorization · Docs: Author rules.