# Measure agreement on the criteria, not only the verdict

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

---

## Measure agreement on the criteria, not only the verdict

ID: MHC-D-RESEARCH-0418 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/measure-agreement-on-the-criteria-not-only-the-verdict

Two people can disagree on the verdict because they never agreed on what the evidence meant one step earlier.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Reviewers disagree on final decisions and the team cannot tell whether the issue is evidence, criteria or synthesis.

### Avoid when

- Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.

### Explanation

Capture reviewer judgments on the important component criteria as well as the final outcome. Calculate or inspect agreement at each layer. Low criterion agreement suggests definitions, examples or evidence interpretation need work; high criterion agreement with different final verdicts points instead to weighting or integration rules.

### Steps

1. The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.

### Example

Reviewers may agree that a change has high blast radius and weak rollback but disagree on the overall approval because their weighting differs.

### Check

The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.

### Limits

- Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.

### Evidence and sources

- supports: In the IFCN study, agreement on individual criteria ranged only from fair to moderate even though the criteria were explicitly defined. — RS-4DC9763DF7D5A389. Criterion-level agreement may depend on training, examples and the intrinsic ambiguity of the observed signal. (Inter-rater agreement on individual criteria)
- RS-4DC9763DF7D5A389: Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability — https://www.sciencedirect.com/science/article/pii/S1388245725003268

No review details supplied.
