Protocol
Score your probability forecasts instead of remembering the wins
A forecaster who remembers only being right has invented a very generous scoring rule.
When it fits
- You make recurring probabilistic judgments and want to improve calibration rather than collect anecdotes.
When to avoid it
- A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.
Why it matters
Record each binary forecast as a probability before the event resolves, then calculate a proper score such as the Brier score after resolution. Review a batch of forecasts rather than one dramatic miss. Keep the event definitions and timestamps so hindsight cannot edit the prediction.
Steps
- The probability is recorded before resolution.
- The event has a fixed binary resolution rule.
- The same scoring rule is used across comparable forecasts.
- Resolved and unresolved forecasts remain distinguishable.
- Review focuses on a batch, not one lucky or unlucky outcome.
An example
Record 60%, 80% and 30% probabilities for project milestones, then score them after outcomes instead of labeling each prediction simply right or wrong.
Check your result
Forecast performance is inspectable from the recorded probabilities and outcomes rather than memory.
Keep this limit in mind
- A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.
Connected ideas
Useful withSeparate calibration from sharpness
Evidence and sources
The Brier score is a strictly proper scoring rule for binary probabilistic forecasts based on squared distance between the forecast probability and the observed outcome.
One score summarizes performance and should be supplemented with diagnostics such as calibration when enough forecasts accumulate.
Proper Scoring Rules for Estimation and Forecast Evaluation · Brier score example