Debug an intermittent failure
Investigate a bug that appears only sometimes or a flaky test. Keep failed and successful runs, compare what changed and test competing explanations. Record what remains unknown.
Keep the failed run #
Keep the failed run and nearby successful runs. Record the input, version, timing, test order, concurrent work and dependency state you can observe.
Your first goal is to reproduce the same failure. If you cannot make it fail locally, preserve enough of the original run to test an explanation. A different local error may need a different fix.
Sources: Keep every rerun result when the failure is intermittent · Mozilla: Debugging Intermittent Test Failures
1. Check that it is the same failure #
Repeat the smallest useful test under controlled conditions. If it fails only in the full test suite or continuous integration (CI), keep that context: test order, shared state, resource pressure or the environment may be involved.
Match the new result to the original failure before trying a fix. Making a different error disappear does not resolve the reported one.
Sources: Keep every rerun result when the failure is intermittent · Mozilla: Debugging Intermittent Test Failures
2. Compare a pass with a failure #
Place a comparable successful run beside the failed one. Keep random seeds, test shards (groups of tests), worker counts, input identities, versions, timings and relevant external responses when available.
Change one condition at a time. Choose a test for which two plausible explanations predict different results, so the result can help you rule one out.
Sources: Keep every rerun result when the failure is intermittent
3. Use bisection only when it fits #
Bisection narrows an ordered range by testing a point within it. Start with a known-good point, a known-bad point and a range where the result lets you rule out half the candidates. Version history or an ordered data range may fit.
Do not use it when causes interact or you cannot reliably classify a run as passing or failing. A midpoint result then gives you no sound reason to discard half the range.
Sources: Bisect a large failure domain · Bisect the version history instead of inspecting every change
4. Inspect where data changes #
Inspect inputs and outputs where data is parsed, mapped, serialized, checked against a policy, queued or sent to a model. These are semantic boundaries: places where data changes form or moves between components. Check whether each side gives the data the same meaning.
Keep the capture bounded. Better observability is not permission to retain secrets or personal data that the diagnosis does not need.
5. Reduce harm and keep the evidence #
If the failure is harming users now, reduce its impact. A rollback, isolation or controlled fallback may be the right first step while you investigate.
Preserve enough state to continue the investigation. “Mitigate first” is not permission for uncontrolled changes that erase the evidence.
Choose the next test #
Use the comparison note below to record the failure, a comparable success and the observed differences. Keep a rival explanation beside your current hypothesis, then choose the smallest test that can help rule one out.
If you still cannot reproduce or explain the failure, write that down. A passing rerun or a recent change is not enough to establish the cause.
Sources: Keep every rerun result when the failure is intermittent · Capture semantic boundaries, not only raw logs
Use it now
Copy a pass/fail comparison note
Use this to keep a flaky failure tied to observable differences and one discriminating next test instead of another round of random reruns.
Failure signature: Nearest comparable passing run: Conditions that differed: Current hypothesis: Plausible rival: Next discriminating test: Semantic boundary to inspect if still unclear:
Use this with an AI assistant
Get help turning the guide into a plan, with practice tasks and a clear next check.
Debug an Intermittent Failure or Flaky Test