Principle
Mitigate first when the incident is still hurting
The perfect explanation is not the first service you owe an active outage.
Before you use it
- Use when
- A live incident is causing material harm while a deeper diagnosis could take much longer.
Keep this limit in mind
- Do not use mitigation-first as permission for uncontrolled production changes or evidence destruction.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Separate the immediate question—how to reduce impact—from the later question—why the failure happened. Choose a reversible mitigation such as rollback, traffic shift, feature disablement or workload reduction when it is safer than continuing damage. Preserve evidence and record what the mitigation changed so the later investigation still has a trail.
An example
Route new work away from a failing integration before tracing every condition that caused the backlog.
Check your result
User or system impact is measurably reduced, or the proposed mitigation is rejected for an explicit safety reason.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Try it in a task guide
These guides use this edition of the method alongside other ideas for the same task.
Separate mitigation from root-cause diagnosis while impact continues.Debug an Intermittent Failure or Flaky Test
Connected ideas
You are readingMitigate first when the incident is still hurting
Evidence and sources
Google SRE's incident-response guidance recommends a mitigation-first response before deeper investigation consumes the incident.
Mitigation should not destroy evidence, violate change controls or create a larger safety problem.
Incident Response · Incident Response Training
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.