Protocol
Test the rollback before you need it
A rollback plan that exists only on paper is a hypothesis.
When it fits
- A release plan says 'rollback available' but nobody has recently executed the recovery path.
When to avoid it
- A rehearsal cannot reproduce every live dependency; data migrations and external side effects may require a different recovery design.
Why it matters
Exercise the rollback in a safe environment using the same artifacts, permissions and sequence intended for production. Verify that the old version or state actually returns, dependencies remain compatible and operators know what cannot be reversed. Record the measured recovery time and any manual step that still depends on memory.
Steps
- Execute the documented rollback with production-like artifacts and permissions.
- Verify the known-good state after rollback, not only that the command completed.
- Record irreversible parts and required fix-forward steps.
- Update the runbook when the rehearsal exposes missing access, timing or dependency assumptions.
An example
Before a high-risk configuration rollout, deploy the new version in staging, roll it back using the production runbook and verify the old behavior.
Check your result
Someone can demonstrate the recovery path and its limitations with evidence from a recent rehearsal.
Keep this limit in mind
- A rehearsal cannot reproduce every live dependency; data migrations and external side effects may require a different recovery design.
Connected ideas
Useful withWrite the rollback trigger before deployment
Evidence and sources
AWS recommends validating rollback procedures before live deployment rather than discovering the recovery path during failure.
A nonproduction rollback test may not reproduce every production dependency or data state.
OPS06-BP01 Plan for unsuccessful changes · Documented and tested recovery plan