Protocol
Keep a fresh dataset for the discovery's next test
Data that helped invent the rule cannot also be completely surprised by it.
When it fits
- Exploration has produced a promising rule, segment or predictive pattern.
When to avoid it
- A later sample can differ for real contextual reasons. Separation reduces one problem, not every source of bias.
Why it matters
Use an untouched holdout or a genuinely new sample for a later check. Fix the rule and evaluation before examining that result. If you repeatedly tune against the holdout, acknowledge that it has become part of development and arrange a fresh evaluation.
Steps
- Separate exploratory data from a suitable later evaluation source.
- Specify the rule, outcome and success criterion before accessing the evaluation results.
- Preserve failures and avoid quietly retuning until the reserved data look favorable.
An example
A segment discovered in last quarter's records can be tested on a later eligible period, with time-related differences considered explicitly.
Check your result
The claimed confirmation uses data that did not guide the final rule's selection.
Keep this limit in mind
- A later sample can differ for real contextual reasons. Separation reduces one problem, not every source of bias.
Evidence and sources
Supports
Data reserved from exploration can provide a later check of a hypothesis formed on another portion of the data.
Repeatedly examining and tuning against the holdout removes the separation it was meant to provide.
Preregistration · Holdout data; confirmatory and exploratory research