Principle
Separate confident probabilities from useful distinctions
The card groups sharpness and resolution together. That can hide an important difference: a probability can be confident without distinguishing cases well. State whether you are examining the forecasts alone or comparing them with what happened.
Before you use it
- Use when
- A forecaster looks impressive because predictions are confident or because most favored outcomes occur.
Keep this limit in mind
- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.
Current editorial reading
Name the property before judging it
Calibration asks whether events assigned a probability occur at about that rate over enough comparable cases. In the concentration-based definition used by Gneiting and colleagues, sharpness describes the forecast distribution alone: a binary 80% forecast is more concentrated than 50%. Resolution asks whether groups receiving different forecasts have different event rates. That needs outcomes. Terminology varies: Bröcker notes that the resolution term is also called a sharpness term. For a practical review, keep the definitions explicit rather than treating confidence and observed separation as the same test.
An 80% forecast can tell every case the same story
Imagine 100 comparable milestones, all forecast at 80%, with 80 eventually completed by the agreed deadline. The observed rate matches the forecast for this sample, and each forecast is more concentrated than 50%. But there is only one forecast group, so these predictions provide no between-group resolution. Now imagine 50 cases forecast at 60% with 30 completions, and 50 at 100% with 50 completions. The overall completion rate is still 80%, but the two groups have different observed rates. These invented counts illustrate the distinction; they do not establish future calibration or justify assigning certainty to real milestones.
Check the evidence before changing confidence
Review probabilities, group sizes and observed rates together. With few cases, or a changed mix of tasks, an apparent pattern may be unstable. Do not move probabilities away from 50% merely to make them look more informative. Ask whether relevant evidence distinguishes the cases, then evaluate later forecasts on comparable events. Keep the Brier score as an overall performance check; neither a confident forecast nor a pleasing calibration summary is sufficient on its own.
Take it with you
Use calibration for agreement with event rates, concentration-based sharpness for how definite the forecasts are, and resolution for differences in outcome rates between forecast groups. Name the convention, retain sample-size limits and test any change on later cases.
Sources and evidence (3)
What we checked
In the concentration-based definition, sharpness is a property of the forecasts alone, while calibration compares forecasts with outcomes.
Selected introduction, diagnostic and discussion sections inspected. The paper primarily treats continuous distributions; the binary concentration example is a direct mathematical illustration, not an observed learner result.
Probabilistic forecasts, calibration and sharpness · Introduction; Sections 3.3–3.4 and 5
For binary Brier-score decomposition, resolution depends on variation in outcome probabilities conditional on forecasts, not merely on the extremity of issued probabilities.
Section 2 was inspected in the author preprint, including the interpretation of equation (13). The hypothetical group counts are independent illustrations, not the paper’s empirical data.
Reliability, sufficiency, and the decomposition of proper scores · Section 2, equation (13), pp. 5–6 of the preprint
Some forecasting literature also calls the resolution component a sharpness term, so the terminology is not uniform.
Footnote 4 explicitly records this alternate label. The sidecar clarifies the property being measured rather than claiming that the words can never be used as synonyms.
Reliability, sufficiency, and the decomposition of proper scores · Section 2, footnote 4; Section 3 discussion
Forecast-score decomposition distinguishes miscalibration, discrimination or resolution, and uncertainty.
Rechecked selected Section 4 text in author arXiv v4; this agrees with the original card’s outcome-based claim but does not validate its wording about concentration-based sharpness.
Proper Scoring Rules for Estimation and Forecast Evaluation · Section 4, forecast-evaluation decomposition
Editorial sources (3)
- Probabilistic forecasts, calibration and sharpness — Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery
- Reliability, sufficiency, and the decomposition of proper scores — Jochen Bröcker
- Proper Scoring Rules for Estimation and Forecast Evaluation — Kartik Waghmare, Johanna Ziegel
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingSeparate confident probabilities from useful distinctions
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaScore your probability forecasts instead of remembering the wins
Original edition evidence and sources
These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.
Forecast evaluation distinguishes calibration, which concerns statistical compatibility between forecast probabilities and outcomes, from discrimination or resolution, which concerns separating cases with different outcomes.
Reliable calibration assessment needs enough comparable forecasts; small samples can look well or poorly calibrated by chance.
Proper Scoring Rules for Estimation and Forecast Evaluation · Scoring-rule decompositions
A 2025 interview study found that decision-makers want uncertainty information but differ in the level and form of detail they can use, and complex probabilistic communication can be hard to interpret.
The study is qualitative and context-dependent; it does not imply that probabilities should be avoided.
From scientific models to decisions: exploring uncertainty communication gaps between scientists and decision-makers · Abstract and uncertainty-communication findings
Original edition, downloads and provenance
The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.
Version 0.1.0
No review details are supplied for this edition.
Read the original edition and overlay as JSON · Release details