Protocol
Make the rollout stop itself on a known bad signal
A safety metric is more useful when it can pull the brake.
When it fits
- A progressive deployment can be observed automatically and waiting for a human response would increase impact.
When to avoid it
- Poor thresholds can cause harmful flapping or false rollback; automate only signals whose meaning and recovery action are understood.
Why it matters
Connect a small set of high-confidence failure signals to automatic halt or rollback behavior. Define the condition before rollout and test that the automation acts on the intended version. Keep a human override for ambiguous cases and record every automatic intervention for review.
Steps
- The stop signal maps to a real unacceptable outcome.
- The threshold is defined before deployment.
- The rollback or halt targets the correct change version.
- The automatic path has been tested safely.
- A human can inspect and override when the signal is ambiguous.
- Every automatic stop creates an auditable event.
An example
Pause a configuration rollout automatically when error rate or a critical data-integrity check crosses the predefined boundary.
Check your result
A known bad signal limits further exposure without depending on someone noticing a dashboard in time.
Keep this limit in mind
- Poor thresholds can cause harmful flapping or false rollback; automate only signals whose meaning and recovery action are understood.
Evidence and sources
AWS recommends automated rollback when predefined tests or desired-outcome thresholds indicate that a deployed change is unsuccessful.
Automatic rollback needs carefully chosen signals; a false alarm can itself create disruption.
OPS06-BP04 Automate testing and rollback · Pre-defined conditions and automated rollback