Protocol
Keep every rerun result when the failure is intermittent
Rerunning until green edits the story, not the software.
When it fits
- An unchanged test alternates between passing and failing.
When to avoid it
- No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk.
Why it matters
Preserve the sequence of outcomes and the conditions of each attempt. First establish that the result varies without the intended code change. Then investigate candidate dependencies such as concurrency, test order or external services instead of treating the final pass as a repair.
Steps
- Record version, input, environment, attempt and outcome together.
- Compare failing and passing attempts for a specific condition that changed.
- Test that condition deliberately in a safe environment and retain failures in the report.
An example
A batch test passes alone but fails after another test leaves shared state behind. An isolated green run did not cover that sequence.
Check your result
The evidence contains unsuccessful attempts and distinguishes a suspected condition from a confirmed cause.
Keep this limit in mind
- No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk.
Connected ideas
Use beforeShrink the example until the same failure is hard to hide
Evidence and sources
A flaky test can pass and fail on the same code, so one successful rerun does not establish that the underlying problem was fixed.
Repeated attempts reveal variability but do not by themselves identify its cause or certify reliability.
Where do our flaky tests come from? · Opening account of nondeterministic tests