Collection
Check what a study can actually tell you
Inspect sampling, uncertainty and analysis choices before converting a statistical result into a decision.
15 elements · Explore in any order.
3 checklists · 4 concepts · 4 protocols · 3 questions · 1 template
15 elements
Check who could enter the sample
A million answers from the wrong doorway still came through that doorway.
Read the idea ChecklistTreat missing responses as an unanswered question
Silence is not a neutral rating waiting to be filled in.
Read the idea ConceptAsk what was randomized
A random draw into a group and a random allocation inside it solve different problems.
Read the idea ProtocolHide condition labels from the person judging the outcome
The evaluator should not have to fight the label before judging the work.
Read the idea ConceptDistinguish variation in cases from uncertainty in a mean
Customers can have wildly different experiences while the average is estimated very precisely.
Read the idea ConceptRead a confidence interval as a procedure with assumptions
The interval is useful because of how it was made, not because 95 is a reassuring number.
Read the idea ConceptKeep the condition attached to a p-value
The small number does not answer every question that can be phrased with 'chance'.
Read the idea TemplateName the smallest change worth acting on
Detectable is not the same as worth the migration.
Read the idea QuestionAsk what a nonsignificant study could have missed
An uncertain answer should not be promoted to a confident zero.
Read the idea ChecklistCount the comparisons behind the winning result
The winning p-value may have had a large audition pool.
Read the idea ProtocolTimestamp the analysis plan before seeing the answer
Memory is a generous editor of what you supposedly expected.
Read the idea ProtocolKeep a fresh dataset for the discovery's next test
Data that helped invent the rule cannot also be completely surprised by it.
Read the idea ProtocolPreserve the pair in a before-and-after comparison
Two columns may contain a relationship that their separate averages conceal.
Read the idea QuestionDo not mistake frequent readings for independent evidence
Recording the same slow-moving process every second does not create a new world every second.
Read the idea ChecklistKeep the ingredients of a ratio
The same ratio can hide two very different stories.
Read the idea