Protocol

Run a duplicate-case noise audit

You cannot see noise from one judgment of one case.

About 1 min read

Before you use it

Use when
  • A judgment process is important but nobody knows how consistently the same evidence is treated.

Keep this limit in mind

  • Repeated cases can be remembered, and small samples are noisy themselves; use the audit as a diagnostic rather than a precision score.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Insert a small set of repeated or equivalently constructed cases without telling reviewers which ones are duplicates. Compare the same reviewer's judgments across time and different reviewers' judgments on the same evidence. Use the spread to identify where the process needs clearer evidence, scale anchors or training.

Steps

  1. Duplicate or equivalent cases preserve the decision-relevant evidence.
  2. Reviewers do not know which items are repeats during scoring.
  3. Within-reviewer and between-reviewer variation are calculated separately.
  4. Large disagreements are inspected at the component or criterion level.
  5. The audit is used to improve the process, not rank people from tiny samples.

An example

Reinsert several anonymized past support-priority cases into a quarterly triage review and compare ratings.

Check your result

The team has an empirical estimate of unwanted variability on nominally equivalent evidence.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingRun a duplicate-case noise audit

Evidence and sources
Supports

Decision-noise research distinguishes level noise, stable pattern noise and occasion noise, which arise from different sources and require different diagnostics.

The taxonomy is a conceptual model; real systems can contain several forms of noise at once.

Human judgment error in the intensive care unit: a perspective on bias and noise · Noise taxonomy

Supports

A 2026 mixed-methods experiment found both systematic bias and level noise among 2,763 professional prescribing judgments presented with the same decision context.

The specific amount and direction of variation are domain-dependent; this does not quantify noise in other professions.

Managing noise through nudges: a mixed methods study on decision-hygiene in public management · Abstract

All sources (2)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details