Checklist
Define the forecast event before assigning a probability
A probability cannot rescue an event nobody defined.
When it fits
- People are debating whether something is 'likely' without agreeing on exactly what would count as occurring.
When to avoid it
- A precise event can still be based on poor evidence; resolution clarity improves evaluability, not forecast accuracy.
Checklist
- A third party can determine the outcome later without asking the forecaster what they originally meant.
Why it matters
Specify the event, observation window, resolution source and ambiguous edge cases before writing the probability. For continuous outcomes, define the threshold or interval being forecast. This makes later scoring possible and prevents forecasters from changing the meaning after seeing what happened.
An example
Replace 'the rollout will probably be stable' with 'there is a 70% chance that no Sev-1 rollback-triggering incident occurs during the first seven days after full rollout.'
Check your result
A third party can determine the outcome later without asking the forecaster what they originally meant.
Keep this limit in mind
- A precise event can still be based on poor evidence; resolution clarity improves evaluability, not forecast accuracy.
Connected ideas
Use beforeScore your probability forecasts instead of remembering the wins
Evidence and sources
Strictly proper scoring rules are designed so that a forecaster minimizes expected loss by reporting their actual probability belief rather than strategically distorting it.
A proper score rewards honest probabilistic reporting; it does not guarantee that the underlying belief is well informed.
Proper Scoring Rules for Estimation and Forecast Evaluation · Definition and motivation of proper scoring rules