Question
Ask what a nonsignificant study could have missed
An uncertain answer should not be promoted to a confident zero.
When it fits
- A report says there was no effect because a test did not cross a significance threshold.
When to avoid it
- Do not replace the analysis with post-hoc storytelling about power; examine the actual estimate, interval and design.
A question to ask
Which effect sizes remain compatible with the interval and assumptions? · Was the design capable of resolving the change that matters? · Was equivalence actually tested with justified bounds, or merely asserted after a nonsignificant result?
Why it matters
Inspect the effect estimate, interval and the study's ability to detect a meaningful difference. Power depends on a specified effect and design. A small or noisy study may leave both useful benefit and important harm unresolved even when its p-value is large.
An example
An imprecise comparison cannot establish that two tools perform equally just because neither is declared a winner.
Check your result
The conclusion distinguishes evidence of little difference from insufficiently precise evidence.
Keep this limit in mind
- Do not replace the analysis with post-hoc storytelling about power; examine the actual estimate, interval and design.
Evidence and sources
Statistical power concerns the probability of rejecting the null under a specified alternative and design.
Power is not a single property independent of the effect size, variability, sample size and analysis.
Quantitative Techniques · Power and type II error