# Separate calibration from sharpness

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

---

## Separate calibration from sharpness

ID: MHC-D-RESEARCH-0406 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/separate-calibration-from-sharpness

A forecast can be bold without being calibrated, and calibrated by being timid.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A forecaster looks impressive because predictions are confident or because most favored outcomes occur.

### Avoid when

- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.

### Explanation

Ask two different questions. Calibration: when you say 70%, do events in that class occur about 70% of the time? Sharpness or resolution: do your forecasts meaningfully distinguish higher-risk from lower-risk cases instead of clustering near the base rate? Improve confidence only while preserving calibration.

### Example

Always saying 50% may be well calibrated in a balanced environment but tells the decision-maker almost nothing about which cases differ.

### Check

Forecast review reports calibration and discriminatory sharpness as separate qualities rather than calling one number 'accuracy.'

### Limits

- Both diagnostics need adequate sample size and comparable cases; small bins can create unstable impressions.

### Evidence and sources

- supports: Forecast evaluation distinguishes calibration, which concerns statistical compatibility between forecast probabilities and outcomes, from discrimination or resolution, which concerns separating cases with different outcomes. — RS-58E31CFFFB569015. Reliable calibration assessment needs enough comparable forecasts; small samples can look well or poorly calibrated by chance. (Scoring-rule decompositions)
- supports: A 2025 interview study found that decision-makers want uncertainty information but differ in the level and form of detail they can use, and complex probabilistic communication can be hard to interpret. — RS-38A3FC4F5DD1B370. The study is qualitative and context-dependent; it does not imply that probabilities should be avoided. (Abstract and uncertainty-communication findings)
- RS-58E31CFFFB569015: Proper Scoring Rules for Estimation and Forecast Evaluation — https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-042424-050626
- RS-38A3FC4F5DD1B370: From scientific models to decisions: exploring uncertainty communication gaps between scientists and decision-makers — https://link.springer.com/article/10.1007/s10669-025-10039-w

### Vedokrok editorial review

Separate confident probabilities from useful distinctions

The card groups sharpness and resolution together. That can hide an important difference: a probability can be confident without distinguishing cases well. State whether you are examining the forecasts alone or comparing them with what happened.

#### Name the property before judging it

Calibration asks whether events assigned a probability occur at about that rate over enough comparable cases. In the concentration-based definition used by Gneiting and colleagues, sharpness describes the forecast distribution alone: a binary 80% forecast is more concentrated than 50%. Resolution asks whether groups receiving different forecasts have different event rates. That needs outcomes. Terminology varies: Bröcker notes that the resolution term is also called a sharpness term. For a practical review, keep the definitions explicit rather than treating confidence and observed separation as the same test.

#### An 80% forecast can tell every case the same story

Imagine 100 comparable milestones, all forecast at 80%, with 80 eventually completed by the agreed deadline. The observed rate matches the forecast for this sample, and each forecast is more concentrated than 50%. But there is only one forecast group, so these predictions provide no between-group resolution. Now imagine 50 cases forecast at 60% with 30 completions, and 50 at 100% with 50 completions. The overall completion rate is still 80%, but the two groups have different observed rates. These invented counts illustrate the distinction; they do not establish future calibration or justify assigning certainty to real milestones.

#### Check the evidence before changing confidence

Review probabilities, group sizes and observed rates together. With few cases, or a changed mix of tasks, an apparent pattern may be unstable. Do not move probabilities away from 50% merely to make them look more informative. Ask whether relevant evidence distinguishes the cases, then evaluate later forecasts on comparable events. Keep the Brier score as an overall performance check; neither a confident forecast nor a pleasing calibration summary is sufficient on its own.

**Editorial takeaway:** Use calibration for agreement with event rates, concentration-based sharpness for how definite the forecasts are, and resolution for differences in outcome rates between forecast groups. Name the convention, retain sample-size limits and test any change on later cases.

#### Editorial evidence

- supports: In the concentration-based definition, sharpness is a property of the forecasts alone, while calibration compares forecasts with outcomes. — gneiting-calibration-sharpness-2007. Selected introduction, diagnostic and discussion sections inspected. The paper primarily treats continuous distributions; the binary concentration example is a direct mathematical illustration, not an observed learner result. (Introduction; Sections 3.3–3.4 and 5)
- supports: For binary Brier-score decomposition, resolution depends on variation in outcome probabilities conditional on forecasts, not merely on the extremity of issued probabilities. — broecker-score-decomposition-2009. Section 2 was inspected in the author preprint, including the interpretation of equation (13). The hypothetical group counts are independent illustrations, not the paper’s empirical data. (Section 2, equation (13), pp. 5–6 of the preprint)
- limits: Some forecasting literature also calls the resolution component a sharpness term, so the terminology is not uniform. — broecker-score-decomposition-2009. Footnote 4 explicitly records this alternate label. The sidecar clarifies the property being measured rather than claiming that the words can never be used as synonyms. (Section 2, footnote 4; Section 3 discussion)
- contextualizes: Forecast-score decomposition distinguishes miscalibration, discrimination or resolution, and uncertainty. — waghmare-ziegel-proper-scores-2026. Rechecked selected Section 4 text in author arXiv v4; this agrees with the original card’s outcome-based claim but does not validate its wording about concentration-based sharpness. (Section 4, forecast-evaluation decomposition)
- gneiting-calibration-sharpness-2007: Probabilistic forecasts, calibration and sharpness — https://doi.org/10.1111/j.1467-9868.2007.00587.x — Tilmann Gneiting, Fadoua Balabdaoui, Adrian E. Raftery
- broecker-score-decomposition-2009: Reliability, sufficiency, and the decomposition of proper scores — https://doi.org/10.1002/qj.456 — Jochen Bröcker
- waghmare-ziegel-proper-scores-2026: Proper Scoring Rules for Estimation and Forecast Evaluation — https://doi.org/10.1146/annurev-statistics-042424-050626 — Kartik Waghmare, Johanna Ziegel

Editorial version: 1.0.0 · evidence checked 2026-10-03 · Agent editorial review; not human or expert certification.

No review details supplied.
