Collection
Reduce judgment noise without erasing useful disagreement
Make repeated expert judgments more consistent by separating independent assessment, structured criteria, discussion, aggregation and legitimate perspective differences.
12 elements · Explore in any order.
1 checklist · 1 concept · 3 principles · 7 protocols
12 elements
Make the first judgment before the group talks
The first voice in the room should not silently become part of everybody else's evidence.
Read the idea PrincipleHide first scores until everyone has one
A score can become an anchor even when nobody says a word.
Read the idea ProtocolDiscuss the reasons, then re-rate privately
Discussion is useful for finding what you missed; consensus is not its only valid output.
Read the idea ProtocolDecompose the judgment before scoring the whole
A single number can hide five different arguments.
Read the idea ChecklistUse the same rubric for comparable cases
A standard that changes with the case is not doing much standardizing.
Read the idea ProtocolStandardize irrelevant context across cases
If the paper color should not matter, do not let it become a hidden experimental variable.
Read the idea ProtocolRun a duplicate-case noise audit
You cannot see noise from one judgment of one case.
Read the idea ConceptName the noise before choosing the fix
Not all inconsistency comes from the same place.
Read the idea ProtocolAverage independent estimates before debating one number
A group average built before the argument preserves more independent signal than one built after the room converges.
Read the idea PrinciplePreserve disagreement that survives error correction
Consistency is not the goal when the differences themselves contain information.
Read the idea PrincipleDo not assume a rubric reduced noise—measure it
A rubric can look structured while reviewers still interpret every line differently.
Read the idea ProtocolMeasure agreement on the criteria, not only the verdict
Two people can disagree on the verdict because they never agreed on what the evidence meant one step earlier.
Read the idea