Protocol

Make the forecast score reproducible

The card says to calculate a Brier score but stops before showing how. Use a fixed event, a recorded probability and a resolved outcome so another person can reproduce the result. A score describes this set of forecasts; it does not by itself show that the forecaster is calibrated.

About 3 min read

Before you use it

Use when
  • You make recurring probabilistic judgments and want to improve calibration rather than collect anecdotes.

Keep this limit in mind

  • A proper score is not enough to diagnose why forecasts are weak; use calibration and case review as additional diagnostics.

Current editorial reading

Calculate one row, then the mean

For the binary convention used here, record probability p between 0 and 1 and outcome y as 1 if the defined event occurred, otherwise 0. Each score is (p − y)²; average these scores across the resolved batch. Lower is better, with values from 0 to 1. In a hypothetical example, probabilities 0.60, 0.80 and 0.30 followed by outcomes 1, 0 and 0 produce scores 0.16, 0.64 and 0.09. Their mean is about 0.297. This checks the calculation, not forecasting skill. Some systems sum across both binary categories and report twice this value; label the convention before comparing numbers.

Keep the comparison fair

Fix the event rule and forecast time before observing the outcome. Keep unresolved or cancelled cases visible with their status; do not code them as failures or quietly remove an embarrassing resolved forecast. To compare methods, score predictions for the same events made with the same lead time. Include a simple reference forecast specified before outcomes are known. A lower mean on a different or easier batch does not establish improvement. These record-keeping steps are an editorial application of forecast evaluation, not a tested training programme.

An honest probability can still lose

Strict propriety is an expected-score property: reporting the probability you actually believe minimizes expected loss under that belief. It does not make the belief correct, nor guarantee a good score on one event or a short batch. Keep the original forecast when a low-probability event occurs. Use the batch score alongside calibration and resolution checks, and investigate the available evidence rather than rewriting the forecast after the result.

Take it with you

Save the forecast first, resolve it by the stated rule, calculate the squared error and average a comparable batch. Report the scoring convention and unresolved cases. Then ask what the score can explain and what needs another diagnostic.

Sources and evidence (2)

What we checked

Supports

The single-probability binary Brier score is (p − y)², where y is 0 or 1; lower values represent smaller loss.

Example 1 and forecast-comparison passages inspected in the author-hosted arXiv v4 HTML. The worked numbers are an independently calculated hypothetical example.

Proper Scoring Rules for Estimation and Forecast Evaluation · Introduction, Example 1; Section 1.2

Limits

Strict propriety concerns the expected score under a probability distribution, not guaranteed success for a realized event.

Primary mathematical treatment inspected, including definitions and the binary Brier row. No experiment on this logging routine or general improvement in human forecasting is claimed.

Reliability, sufficiency, and the decomposition of proper scores · Section 2, equations (2)–(3), Table 1

Provides context

An aggregate proper score can be separated into reliability, resolution and uncertainty components.

The decomposition motivates further diagnosis; a single realized mean is not itself a calibration measure. The sidecar does not estimate these components from the three-example batch.

Reliability, sufficiency, and the decomposition of proper scores · Section 2, equation (13) and its interpretation

Editorial sources (2)

Editorial version 1.0.0 · Evidence checked 2026-10-03

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingMake the forecast score reproducible

Original edition evidence and sources

These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.

Supports

The Brier score is a strictly proper scoring rule for binary probabilistic forecasts based on squared distance between the forecast probability and the observed outcome.

One score summarizes performance and should be supplemented with diagnostics such as calibration when enough forecasts accumulate.

Proper Scoring Rules for Estimation and Forecast Evaluation · Brier score example

All sources (1)
Original edition, downloads and provenance

The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.

Version 0.1.0

No review details are supplied for this edition.

Read the original edition and overlay as JSON · Release details