# Score your probability forecasts instead of remembering the wins

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

---

## Score your probability forecasts instead of remembering the wins

ID: MHC-D-RESEARCH-0405 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/score-your-probability-forecasts-instead-of-remembering-the-wins

A forecaster who remembers only being right has invented a very generous scoring rule.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You make recurring probabilistic judgments and want to improve calibration rather than collect anecdotes.

### Avoid when

- A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.

### Explanation

Record each binary forecast as a probability before the event resolves, then calculate a proper score such as the Brier score after resolution. Review a batch of forecasts rather than one dramatic miss. Keep the event definitions and timestamps so hindsight cannot edit the prediction.

### Steps

1. The probability is recorded before resolution.
2. The event has a fixed binary resolution rule.
3. The same scoring rule is used across comparable forecasts.
4. Resolved and unresolved forecasts remain distinguishable.
5. Review focuses on a batch, not one lucky or unlucky outcome.

### Example

Record 60%, 80% and 30% probabilities for project milestones, then score them after outcomes instead of labeling each prediction simply right or wrong.

### Check

Forecast performance is inspectable from the recorded probabilities and outcomes rather than memory.

### Limits

- A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.

### Evidence and sources

- supports: The Brier score is a strictly proper scoring rule for binary probabilistic forecasts based on squared distance between the forecast probability and the observed outcome. — RS-58E31CFFFB569015. One score summarizes performance and should be supplemented with diagnostics such as calibration when enough forecasts accumulate. (Brier score example)
- RS-58E31CFFFB569015: Proper Scoring Rules for Estimation and Forecast Evaluation — https://www.annualreviews.org/content/journals/10.1146/annurev-statistics-042424-050626

### Vedokrok editorial review

Make the forecast score reproducible

The card says to calculate a Brier score but stops before showing how. Use a fixed event, a recorded probability and a resolved outcome so another person can reproduce the result. A score describes this set of forecasts; it does not by itself show that the forecaster is calibrated.

#### Calculate one row, then the mean

For the binary convention used here, record probability p between 0 and 1 and outcome y as 1 if the defined event occurred, otherwise 0. Each score is (p − y)²; average these scores across the resolved batch. Lower is better, with values from 0 to 1. In a hypothetical example, probabilities 0.60, 0.80 and 0.30 followed by outcomes 1, 0 and 0 produce scores 0.16, 0.64 and 0.09. Their mean is about 0.297. This checks the calculation, not forecasting skill. Some systems sum across both binary categories and report twice this value; label the convention before comparing numbers.

#### Keep the comparison fair

Fix the event rule and forecast time before observing the outcome. Keep unresolved or cancelled cases visible with their status; do not code them as failures or quietly remove an embarrassing resolved forecast. To compare methods, score predictions for the same events made with the same lead time. Include a simple reference forecast specified before outcomes are known. A lower mean on a different or easier batch does not establish improvement. These record-keeping steps are an editorial application of forecast evaluation, not a tested training programme.

#### An honest probability can still lose

Strict propriety is an expected-score property: reporting the probability you actually believe minimizes expected loss under that belief. It does not make the belief correct, nor guarantee a good score on one event or a short batch. Keep the original forecast when a low-probability event occurs. Use the batch score alongside calibration and resolution checks, and investigate the available evidence rather than rewriting the forecast after the result.

**Editorial takeaway:** Save the forecast first, resolve it by the stated rule, calculate the squared error and average a comparable batch. Report the scoring convention and unresolved cases. Then ask what the score can explain and what needs another diagnostic.

#### Editorial evidence

- supports: The single-probability binary Brier score is (p − y)², where y is 0 or 1; lower values represent smaller loss. — waghmare-ziegel-proper-scores-2026. Example 1 and forecast-comparison passages inspected in the author-hosted arXiv v4 HTML. The worked numbers are an independently calculated hypothetical example. (Introduction, Example 1; Section 1.2)
- limits: Strict propriety concerns the expected score under a probability distribution, not guaranteed success for a realized event. — broecker-score-decomposition-2009. Primary mathematical treatment inspected, including definitions and the binary Brier row. No experiment on this logging routine or general improvement in human forecasting is claimed. (Section 2, equations (2)–(3), Table 1)
- contextualizes: An aggregate proper score can be separated into reliability, resolution and uncertainty components. — broecker-score-decomposition-2009. The decomposition motivates further diagnosis; a single realized mean is not itself a calibration measure. The sidecar does not estimate these components from the three-example batch. (Section 2, equation (13) and its interpretation)
- waghmare-ziegel-proper-scores-2026: Proper Scoring Rules for Estimation and Forecast Evaluation — https://doi.org/10.1146/annurev-statistics-042424-050626 — Kartik Waghmare, Johanna Ziegel
- broecker-score-decomposition-2009: Reliability, sufficiency, and the decomposition of proper scores — https://doi.org/10.1002/qj.456 — Jochen Bröcker

Editorial version: 1.0.0 · evidence checked 2026-10-03 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/separate-calibration-from-sharpness
