Checklist
Write the rollback trigger before deployment
'We will know when to roll back' is not a rollback condition.
When it fits
- A change can degrade service and people may hesitate to roll back once effort has been invested.
When to avoid it
- Thresholds can be noisy or incomplete; allow human escalation when evidence is serious but the predefined metric misses it.
Checklist
- The rollback condition is observable during the rollout.
- The signal is tied to the change's real failure modes.
- An owner has authority to stop or reverse the rollout.
- The condition distinguishes rollback from a tolerable transient effect.
- The recovery path is known before the trigger fires.
Why it matters
Before deployment, define observable signals that mean the change should stop or reverse: error rate, failed reconciliation, latency, business invariant, unexpected side effect or another relevant threshold. Also define who can call the rollback. This converts a stressful judgment into a prepared decision and reduces the temptation to wait for certainty while impact grows.
An example
Rollback if more than 0.5% of updated records fail the post-write invariant or if replication backlog exceeds the agreed threshold for ten minutes.
Check your result
A responder can decide whether the trigger fired without inventing a new rule during the incident.
Keep this limit in mind
- Thresholds can be noisy or incomplete; allow human escalation when evidence is serious but the predefined metric misses it.
Connected ideas
Use beforeMake the rollout stop itself on a known bad signal
Evidence and sources
AWS recommends documenting rollback criteria and the rollback or fix-forward plan before deploying a change.
Rollback is not always safer than fixing forward, particularly for irreversible data migrations; the strategy must match the change.
OPS06-BP01 Plan for unsuccessful changes · Implementation guidance