Protocol
Run a duplicate-case noise audit
You cannot see noise from one judgment of one case.
When it fits
- A judgment process is important but nobody knows how consistently the same evidence is treated.
When to avoid it
- Repeated cases can be remembered, and small samples are noisy themselves; use the audit as a diagnostic rather than a precision score.
Why it matters
Insert a small set of repeated or equivalently constructed cases without telling reviewers which ones are duplicates. Compare the same reviewer's judgments across time and different reviewers' judgments on the same evidence. Use the spread to identify where the process needs clearer evidence, scale anchors or training.
Steps
- Duplicate or equivalent cases preserve the decision-relevant evidence.
- Reviewers do not know which items are repeats during scoring.
- Within-reviewer and between-reviewer variation are calculated separately.
- Large disagreements are inspected at the component or criterion level.
- The audit is used to improve the process, not rank people from tiny samples.
An example
Reinsert several anonymized past support-priority cases into a quarterly triage review and compare ratings.
Check your result
The team has an empirical estimate of unwanted variability on nominally equivalent evidence.
Keep this limit in mind
- Repeated cases can be remembered, and small samples are noisy themselves; use the audit as a diagnostic rather than a precision score.
Connected ideas
Useful withName the noise before choosing the fix
Evidence and sources
Decision-noise research distinguishes level noise, stable pattern noise and occasion noise, which arise from different sources and require different diagnostics.
The taxonomy is a conceptual model; real systems can contain several forms of noise at once.
Human judgment error in the intensive care unit: a perspective on bias and noise · Noise taxonomy
A 2026 mixed-methods experiment found both systematic bias and level noise among 2,763 professional prescribing judgments presented with the same decision context.
The specific amount and direction of variation are domain-dependent; this does not quantify noise in other professions.
Managing noise through nudges: a mixed methods study on decision-hygiene in public management · Abstract