Checklist
Count the comparisons behind the winning result
The winning p-value may have had a large audition pool.
When it fits
- One attractive result is selected from many tests, outcomes or subgroup comparisons.
When to avoid it
- No single correction is best for every problem. Define the question and testing family before choosing a method.
Checklist
- The relevant tested outcomes and comparisons are visible.
- The primary question is distinguished from exploratory follow-ups.
- Multiplicity is handled with an appropriate method or the exploratory limitation is stated clearly.
Why it matters
Find out how many opportunities there were to produce the highlighted result. Simultaneous inference needs an error-control plan appropriate to that family of questions. A table containing only the successful comparisons hides the selection process that shaped the conclusion.
An example
A campaign report that tests many audience slices should not present the best slice as though it was the only planned comparison.
Check your result
The interpretation accounts for the search that produced the selected result.
Keep this limit in mind
- No single correction is best for every problem. Define the question and testing family before choosing a method.
Connected ideas
Useful withTimestamp the analysis plan before seeing the answer
Evidence and sources
Supports
Simultaneous inference over several comparisons requires considering the joint error or coverage goal, not only each isolated comparison.
The appropriate multiplicity method depends on the scientific question and testing family.
Bonferroni's Method · Simultaneous confidence and comparisons