Principle
Do not assume a rubric reduced noise—measure it
A rubric can look structured while reviewers still interpret every line differently.
Before you use it
- Use when
- A team introduces a checklist or criteria set and declares the judgment process standardized.
Keep this limit in mind
- The IFCN null result is from a clinical expert task; it motivates measurement, not a claim that rubrics generally fail.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Compare reliability before and after the structured criteria on repeated or shared cases. If agreement does not improve, inspect criterion definitions, training examples and how criteria are integrated into the final judgment. Keep the null result: structure that does not change reliability may still aid explanation, but it has not earned a noise-reduction claim.
An example
After adding a six-item architecture-risk checklist, compare whether reviewers actually converge more on the same proposals.
Check your result
The team can distinguish 'we added structure' from 'the structure measurably improved consistency.'
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingDo not assume a rubric reduced noise—measure it
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaUse the same rubric for comparable casesUseful with this idea
Read eval transcripts before trusting the score
Evidence and sources
The 2025 IFCN study found that explicitly adding six expert criteria did not materially improve inter-rater reliability, performance or overall calibration in the studied expert judgments.
This null result applies to one clinical judgment task and does not imply that structured criteria are generally useless.
Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability · Abstract results
Edition and review
Version 0.1.0
No review details are supplied for this edition.