The safest way to test AI agent guardrails is to replay what your agents actually did through the new rule before it goes live. If you're new to the idea, start with why prompt-based guardrails aren't limits. Unit tests catch typos; replay catches the rule that quietly doubles the finance team's approval queue.

Why new guardrails surprise people

A rule that looks sensible on paper can hold far more actions than anyone expected — or far fewer. Tighten a refund limit and the approval queue fills overnight. Loosen it and nobody notices until month-end. The problem isn't the rule's logic; it's that nobody saw its effect on real traffic.

Three levels of testing

  1. Unit tests — does amount: 150 return require_approval? Fast, and good for catching mistakes in rule logic.
  2. Scenario tests — a set of named examples ("bulk export", "VIP refund", "weekend production change") that must always produce the same answer. Run them on every change.
  3. Replay (dry-run) — run the new rule against a recent window of real agent activity and compare outcomes. This is the one that shows real impact.

How to run a dry-run

Before publishing a change, replay the last few weeks of agent actions through the draft version. Every past action gets a "would have been" outcome next to the one it actually got. See an example dry-run.

Read the differences, not the totals

Focus on what changed: actions that would now be held, now be denied, or now run automatically. Share that list with the process owner. It turns an abstract threshold into concrete examples they can judge — "these 14 refunds would now wait for you" is a decision people can actually make.

Publish with a record — and a way back

When the change goes live, record who proposed it, who approved it, and what the dry-run showed — alongside the audit trail of the decisions it will make. Keep the previous version one step away, so a rollback is a decision, not a project.

Testing is one part of getting ready to deploy AI agents to production, and it works best when rules are managed as policy as code.

Where Gaten fits

Gaten lets you dry-run any policy change against past agent activity, review the differences, and publish or roll back versions with a full record. Learn how policies work.

Frequently asked questions

How much history should a dry-run use?

Enough to include your normal peaks — often two to four weeks of activity. Include at least one month-end if the rule touches money.

Can I test guardrails without production data?

Yes, with scenario tests and synthetic examples. But only real activity shows how often a rule will actually fire.

Who should review dry-run results?

The owner of the process the rule protects — finance for refunds, security for access — alongside the engineer who wrote the change.