Principle

Mitigate first when the incident is still hurting

The perfect explanation is not the first service you owe an active outage.

About 1 min read

Before you use it

Use when
  • A live incident is causing material harm while a deeper diagnosis could take much longer.

Keep this limit in mind

  • Do not use mitigation-first as permission for uncontrolled production changes or evidence destruction.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Separate the immediate question—how to reduce impact—from the later question—why the failure happened. Choose a reversible mitigation such as rollback, traffic shift, feature disablement or workload reduction when it is safer than continuing damage. Preserve evidence and record what the mitigation changed so the later investigation still has a trail.

An example

Route new work away from a failing integration before tracing every condition that caused the backlog.

Check your result

User or system impact is measurably reduced, or the proposed mitigation is rejected for an explicit safety reason.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Try it in a task guide

These guides use this edition of the method alongside other ideas for the same task.

Separate mitigation from root-cause diagnosis while impact continues.
Debug an Intermittent Failure or Flaky Test

Connected ideas

You are readingMitigate first when the incident is still hurting

Evidence and sources
Supports

Google SRE's incident-response guidance recommends a mitigation-first response before deeper investigation consumes the incident.

Mitigation should not destroy evidence, violate change controls or create a larger safety problem.

Incident Response · Incident Response Training

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details