Principle
Separate calibration from sharpness
A forecast can be bold without being calibrated, and calibrated by being timid.
When it fits
- A forecaster looks impressive because predictions are confident or because most favored outcomes occur.
When to avoid it
- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.
Why it matters
Ask two different questions. Calibration: when you say 70%, do events in that class occur about 70% of the time? Sharpness or resolution: do your forecasts meaningfully distinguish higher-risk from lower-risk cases instead of clustering near the base rate? Improve confidence only while preserving calibration.
An example
Always saying 50% may be well calibrated in a balanced environment but tells the decision-maker almost nothing about which cases differ.
Check your result
Forecast review reports calibration and discriminatory sharpness as separate qualities rather than calling one number 'accuracy.'
Keep this limit in mind
- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.
Evidence and sources
Forecast evaluation distinguishes calibration, which concerns statistical compatibility between forecast probabilities and outcomes, from discrimination or resolution, which concerns separating cases with different outcomes.
Reliable calibration assessment needs enough comparable forecasts; small samples can look well or poorly calibrated by chance.
Proper Scoring Rules for Estimation and Forecast Evaluation · Scoring-rule decompositions
A 2025 interview study found that decision-makers want uncertainty information but differ in the level and form of detail they can use, and complex probabilistic communication can be hard to interpret.
The study is qualitative and context-dependent; it does not imply that probabilities should be avoided.
From scientific models to decisions: exploring uncertainty communication gaps between scientists and decision-makers · Abstract and uncertainty-communication findings