Collection
Find the fault without making the incident harder to understand
Turn vague technical failures into smaller tests, controlled incident work and system-level learning that another person can verify and reuse.
15 elements · Explore in any order.
1 checklist · 1 concept · 3 heuristics · 4 principles · 5 protocols · 1 template
15 elements
Build the smallest failure you can reproduce
A bug you can summon is easier to interrogate than a bug you can only describe.
Read the idea ProtocolProbe the boundary between two components
Do not interrogate the whole chain when one boundary can answer where the bad value first appears.
Read the idea HeuristicBisect a large failure domain
When the haystack has structure, stop inspecting every piece of hay.
Read the idea HeuristicStart with what changed, not with certainty about it
The last change deserves an interview, not an automatic conviction.
Read the idea PrinciplePrefer a test that kills hypotheses
A test that can only confirm your favorite story is a weak detective.
Read the idea HeuristicOrder troubleshooting tests by likelihood and risk
Do not reboot the world to answer a question a safe check could settle.
Read the idea TemplateKeep a hypothesis–test–result ledger
Without notes, troubleshooting quietly turns into reruns and folklore.
Read the idea PrincipleMitigate first when the incident is still hurting
The perfect explanation is not the first service you owe an active outage.
Read the idea ConceptChoose the incident channel before the incident
The worst time to invent the emergency room is after the alarm.
Read the idea ProtocolGive one person the incident command
Ten experts without coordination can create eleven incident plans.
Read the idea PrincipleSeparate fixing from explaining during an incident
Every status request steals the same attention you are asking to restore service.
Read the idea PrincipleGive production changes one operational lane
Parallel thinking helps. Parallel uncoordinated production changes often do not.
Read the idea ProtocolUpdate stakeholders on a deliberate cadence
Silence creates its own incident narrative.
Read the idea ChecklistGive every postmortem action a verifiable finish
'Improve reliability' is a wish wearing an action-item field.
Read the idea ProtocolMine postmortems for repeated failure patterns
One postmortem explains an event. A set of postmortems can reveal a system.
Read the idea