# Do not assume a rubric reduced noise—measure it

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

---

## Do not assume a rubric reduced noise—measure it

ID: MHC-D-RESEARCH-0417 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/do-not-assume-a-rubric-reduced-noise-measure-it

A rubric can look structured while reviewers still interpret every line differently.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A team introduces a checklist or criteria set and declares the judgment process standardized.

### Avoid when

- The IFCN null result is from a clinical expert task; it motivates measurement, not a claim that rubrics generally fail.

### Explanation

Compare reliability before and after the structured criteria on repeated or shared cases. If agreement does not improve, inspect criterion definitions, training examples and how criteria are integrated into the final judgment. Keep the null result: structure that does not change reliability may still aid explanation, but it has not earned a noise-reduction claim.

### Example

After adding a six-item architecture-risk checklist, compare whether reviewers actually converge more on the same proposals.

### Check

The team can distinguish 'we added structure' from 'the structure measurably improved consistency.'

### Limits

- The IFCN null result is from a clinical expert task; it motivates measurement, not a claim that rubrics generally fail.

### Evidence and sources

- limits: The 2025 IFCN study found that explicitly adding six expert criteria did not materially improve inter-rater reliability, performance or overall calibration in the studied expert judgments. — RS-4DC9763DF7D5A389. This null result applies to one clinical judgment task and does not imply that structured criteria are generally useless. (Abstract results)
- RS-4DC9763DF7D5A389: Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability — https://www.sciencedirect.com/science/article/pii/S1388245725003268

No review details supplied.
