Protocol
Run a duplicate-case noise audit
You cannot see noise from one judgment of one case.
Before you use it
- Use when
- A judgment process is important but nobody knows how consistently the same evidence is treated.
Keep this limit in mind
- Repeated cases can be remembered, and small samples are noisy themselves; use the audit as a diagnostic rather than a precision score.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Insert a small set of repeated or equivalently constructed cases without telling reviewers which ones are duplicates. Compare the same reviewer's judgments across time and different reviewers' judgments on the same evidence. Use the spread to identify where the process needs clearer evidence, scale anchors or training.
Steps
- Duplicate or equivalent cases preserve the decision-relevant evidence.
- Reviewers do not know which items are repeats during scoring.
- Within-reviewer and between-reviewer variation are calculated separately.
- Large disagreements are inspected at the component or criterion level.
- The audit is used to improve the process, not rank people from tiny samples.
An example
Reinsert several anonymized past support-priority cases into a quarterly triage review and compare ratings.
Check your result
The team has an empirical estimate of unwanted variability on nominally equivalent evidence.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingRun a duplicate-case noise audit
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaStandardize irrelevant context across cases
Evidence and sources
Decision-noise research distinguishes level noise, stable pattern noise and occasion noise, which arise from different sources and require different diagnostics.
The taxonomy is a conceptual model; real systems can contain several forms of noise at once.
Human judgment error in the intensive care unit: a perspective on bias and noise · Noise taxonomy
A 2026 mixed-methods experiment found both systematic bias and level noise among 2,763 professional prescribing judgments presented with the same decision context.
The specific amount and direction of variation are domain-dependent; this does not quantify noise in other professions.
Managing noise through nudges: a mixed methods study on decision-hygiene in public management · Abstract
Edition and review
Version 0.1.0
No review details are supplied for this edition.