Protocol
Measure agreement on the criteria, not only the verdict
Two people can disagree on the verdict because they never agreed on what the evidence meant one step earlier.
When it fits
- Reviewers disagree on final decisions and the team cannot tell whether the issue is evidence, criteria or synthesis.
When to avoid it
- Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.
Why it matters
Capture reviewer judgments on the important component criteria as well as the final outcome. Calculate or inspect agreement at each layer. Low criterion agreement suggests definitions, examples or evidence interpretation need work; high criterion agreement with different final verdicts points instead to weighting or integration rules.
Steps
- The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.
An example
Reviewers may agree that a change has high blast radius and weak rollback but disagree on the overall approval because their weighting differs.
Check your result
The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.
Keep this limit in mind
- Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.
Evidence and sources
In the IFCN study, agreement on individual criteria ranged only from fair to moderate even though the criteria were explicitly defined.
Criterion-level agreement may depend on training, examples and the intrinsic ambiguity of the observed signal.
Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability · Inter-rater agreement on individual criteria