Protocol

Measure agreement on the criteria, not only the verdict

Two people can disagree on the verdict because they never agreed on what the evidence meant one step earlier.

About 1 min read

Before you use it

Use when
  • Reviewers disagree on final decisions and the team cannot tell whether the issue is evidence, criteria or synthesis.

Keep this limit in mind

  • Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Capture reviewer judgments on the important component criteria as well as the final outcome. Calculate or inspect agreement at each layer. Low criterion agreement suggests definitions, examples or evidence interpretation need work; high criterion agreement with different final verdicts points instead to weighting or integration rules.

Steps

  1. The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.

An example

Reviewers may agree that a change has high blast radius and weak rollback but disagree on the overall approval because their weighting differs.

Check your result

The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingMeasure agreement on the criteria, not only the verdict

Evidence and sources
Supports

In the IFCN study, agreement on individual criteria ranged only from fair to moderate even though the criteria were explicitly defined.

Criterion-level agreement may depend on training, examples and the intrinsic ambiguity of the observed signal.

Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability · Inter-rater agreement on individual criteria

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details