Principle

Separate confident probabilities from useful distinctions

The card groups sharpness and resolution together. That can hide an important difference: a probability can be confident without distinguishing cases well. State whether you are examining the forecasts alone or comparing them with what happened.

About 3 min read

Before you use it

Use when
  • A forecaster looks impressive because predictions are confident or because most favored outcomes occur.

Keep this limit in mind

  • Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.

Current editorial reading

Name the property before judging it

Calibration asks whether events assigned a probability occur at about that rate over enough comparable cases. In the concentration-based definition used by Gneiting and colleagues, sharpness describes the forecast distribution alone: a binary 80% forecast is more concentrated than 50%. Resolution asks whether groups receiving different forecasts have different event rates. That needs outcomes. Terminology varies: Bröcker notes that the resolution term is also called a sharpness term. For a practical review, keep the definitions explicit rather than treating confidence and observed separation as the same test.

An 80% forecast can tell every case the same story

Imagine 100 comparable milestones, all forecast at 80%, with 80 eventually completed by the agreed deadline. The observed rate matches the forecast for this sample, and each forecast is more concentrated than 50%. But there is only one forecast group, so these predictions provide no between-group resolution. Now imagine 50 cases forecast at 60% with 30 completions, and 50 at 100% with 50 completions. The overall completion rate is still 80%, but the two groups have different observed rates. These invented counts illustrate the distinction; they do not establish future calibration or justify assigning certainty to real milestones.

Check the evidence before changing confidence

Review probabilities, group sizes and observed rates together. With few cases, or a changed mix of tasks, an apparent pattern may be unstable. Do not move probabilities away from 50% merely to make them look more informative. Ask whether relevant evidence distinguishes the cases, then evaluate later forecasts on comparable events. Keep the Brier score as an overall performance check; neither a confident forecast nor a pleasing calibration summary is sufficient on its own.

Take it with you

Use calibration for agreement with event rates, concentration-based sharpness for how definite the forecasts are, and resolution for differences in outcome rates between forecast groups. Name the convention, retain sample-size limits and test any change on later cases.

Sources and evidence (3)

What we checked

Supports

In the concentration-based definition, sharpness is a property of the forecasts alone, while calibration compares forecasts with outcomes.

Selected introduction, diagnostic and discussion sections inspected. The paper primarily treats continuous distributions; the binary concentration example is a direct mathematical illustration, not an observed learner result.

Probabilistic forecasts, calibration and sharpness · Introduction; Sections 3.3–3.4 and 5

Supports

For binary Brier-score decomposition, resolution depends on variation in outcome probabilities conditional on forecasts, not merely on the extremity of issued probabilities.

Section 2 was inspected in the author preprint, including the interpretation of equation (13). The hypothetical group counts are independent illustrations, not the paper’s empirical data.

Reliability, sufficiency, and the decomposition of proper scores · Section 2, equation (13), pp. 5–6 of the preprint

Limits

Some forecasting literature also calls the resolution component a sharpness term, so the terminology is not uniform.

Footnote 4 explicitly records this alternate label. The sidecar clarifies the property being measured rather than claiming that the words can never be used as synonyms.

Reliability, sufficiency, and the decomposition of proper scores · Section 2, footnote 4; Section 3 discussion

Provides context

Forecast-score decomposition distinguishes miscalibration, discrimination or resolution, and uncertainty.

Rechecked selected Section 4 text in author arXiv v4; this agrees with the original card’s outcome-based claim but does not validate its wording about concentration-based sharpness.

Proper Scoring Rules for Estimation and Forecast Evaluation · Section 4, forecast-evaluation decomposition

Editorial sources (3)

Editorial version 1.0.0 · Evidence checked 2026-10-03

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingSeparate confident probabilities from useful distinctions

Original edition evidence and sources

These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.

Supports

Forecast evaluation distinguishes calibration, which concerns statistical compatibility between forecast probabilities and outcomes, from discrimination or resolution, which concerns separating cases with different outcomes.

Reliable calibration assessment needs enough comparable forecasts; small samples can look well or poorly calibrated by chance.

Proper Scoring Rules for Estimation and Forecast Evaluation · Scoring-rule decompositions

Supports

A 2025 interview study found that decision-makers want uncertainty information but differ in the level and form of detail they can use, and complex probabilistic communication can be hard to interpret.

The study is qualitative and context-dependent; it does not imply that probabilities should be avoided.

From scientific models to decisions: exploring uncertainty communication gaps between scientists and decision-makers · Abstract and uncertainty-communication findings

All sources (2)
Original edition, downloads and provenance

The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.

Version 0.1.0

No review details are supplied for this edition.

Read the original edition and overlay as JSON · Release details