Principle

Do not assume a rubric reduced noise—measure it

A rubric can look structured while reviewers still interpret every line differently.

About 1 min read

Before you use it

Use when
  • A team introduces a checklist or criteria set and declares the judgment process standardized.

Keep this limit in mind

  • The IFCN null result is from a clinical expert task; it motivates measurement, not a claim that rubrics generally fail.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Compare reliability before and after the structured criteria on repeated or shared cases. If agreement does not improve, inspect criterion definitions, training examples and how criteria are integrated into the final judgment. Keep the null result: structure that does not change reliability may still aid explanation, but it has not earned a noise-reduction claim.

An example

After adding a six-item architecture-risk checklist, compare whether reviewers actually converge more on the same proposals.

Check your result

The team can distinguish 'we added structure' from 'the structure measurably improved consistency.'

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingDo not assume a rubric reduced noise—measure it

Evidence and sources
Limits

The 2025 IFCN study found that explicitly adding six expert criteria did not materially improve inter-rater reliability, performance or overall calibration in the studied expert judgments.

This null result applies to one clinical judgment task and does not imply that structured criteria are generally useless.

Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability · Abstract results

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details