# Reduce judgment noise without erasing useful disagreement

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Make repeated expert judgments more consistent by separating independent assessment, structured criteria, discussion, aggregation and legitimate perspective differences.
Collection kind: collection
Source: https://vedokrok.com/collections/reduce-judgment-noise-without-erasing-useful-disagreement
Version: 0.1.0

---

## Make the first judgment before the group talks

ID: MHC-D-RESEARCH-0407 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/make-the-first-judgment-before-the-group-talks

The first voice in the room should not silently become part of everybody else's evidence.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Several people will judge the same case and early discussion could anchor everyone on the first confident opinion.

### Avoid when

- Do not force a judgment before reviewers have information that is genuinely necessary and only available collaboratively.

### Explanation

Give each reviewer the same available case information and ask for an initial judgment before discussion. Capture the score or recommendation plus one short reason. Only then reveal the spread and discuss. This preserves information about independent perspectives that disappears once everyone has heard the dominant view.

### Steps

1. All reviewers receive the same intended information before rating.
2. Each initial judgment is recorded before group discussion.
3. Reviewers give a brief reason or key evidence, not only a number.
4. Initial ratings remain available after the final decision for audit.

### Example

Before an architecture review call, each reviewer independently rates the proposed change's operational risk and notes the main reason.

### Check

The team can see the pre-discussion distribution rather than reconstructing who supposedly thought what afterward.

### Limits

- Do not force a judgment before reviewers have information that is genuinely necessary and only available collaboratively.

### Evidence and sources

- supports: Workplace judgment research recommends collecting independent judgments before social discussion when the goal is to reduce unwanted variability and influence effects. — RS-755DDB8F19ABC634. Independence is less important when the task genuinely requires shared information before any individual judgment is possible. (Independent judgments and aggregation)
- supports: The 2026 AMP procedure asked analysts to make initial ratings independently before group discussion to reduce groupthink while preserving distinct perspectives. — RS-B528AD8A51A47CB0. The study involved five advanced music analysts and is a proof of concept rather than a general effect-size estimate. (Decision-hygiene procedure)
- RS-755DDB8F19ABC634: Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research — https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-120920-050708
- RS-B528AD8A51A47CB0: Analysis from multiple perspectives (AMP): Applying decision hygiene to analysis of musical structure — https://journals.sagepub.com/doi/10.1177/10298649251385727

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/discuss-the-reasons-then-re-rate-privately

---

## Hide first scores until everyone has one

ID: MHC-D-RESEARCH-0408 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/hide-first-scores-until-everyone-has-one

A score can become an anchor even when nobody says a word.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Reviewers can see earlier ratings while entering their own.

### Avoid when

- For tasks where later reviewers must build on earlier specialized analysis, strict score independence may be less appropriate than staged information sharing.

### Explanation

Collect initial ratings privately and reveal the distribution only after the submission window closes. Anonymity can reduce status pressure, but timing matters too: later reviewers should not inherit earlier numbers before forming their own view. Preserve identities separately when accountability requires them.

### Example

A code-review risk poll collects all ratings before displaying the histogram or senior engineer's score.

### Check

No participant can name an earlier rating they saw before submitting their own initial judgment.

### Limits

- For tasks where later reviewers must build on earlier specialized analysis, strict score independence may be less appropriate than staged information sharing.

### Evidence and sources

- supports: Workplace judgment research recommends collecting independent judgments before social discussion when the goal is to reduce unwanted variability and influence effects. — RS-755DDB8F19ABC634. Independence is less important when the task genuinely requires shared information before any individual judgment is possible. (Independent judgments and aggregation)
- RS-755DDB8F19ABC634: Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research — https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-120920-050708

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/make-the-first-judgment-before-the-group-talks

---

## Discuss the reasons, then re-rate privately

ID: MHC-D-RESEARCH-0409 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/discuss-the-reasons-then-re-rate-privately

Discussion is useful for finding what you missed; consensus is not its only valid output.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Independent reviewers disagree and discussion may expose missed evidence but can also create conformity.

### Avoid when

- Discussion can still influence private rerating; the method reduces but does not eliminate social effects.

### Explanation

After independent ratings are collected, compare reasons and evidence rather than arguing immediately for one number. Let reviewers ask questions and inspect overlooked information. Then end discussion and have each person submit a final judgment privately. Compare what changed and which disagreements remain.

### Steps

1. Reveal the distribution and key reasons after independent scoring.
2. Discuss evidence, definitions and overlooked considerations rather than negotiating an average.
3. End the discussion before final ratings are entered.
4. Collect final ratings privately.
5. Distinguish corrected oversights from remaining principled disagreement.

### Example

Security reviewers discuss why two people scored a threat differently, then each privately revises or keeps their rating.

### Check

The process can show which judgments changed because of new information and which disagreements survived informed discussion.

### Limits

- Discussion can still influence private rerating; the method reduces but does not eliminate social effects.

### Evidence and sources

- supports: In AMP, participants discussed perspectives and then completed a final review independently so discussion could reveal oversights without requiring consensus. — RS-B528AD8A51A47CB0. Private re-rating does not remove prestige, persuasion or shared-information effects that already occurred during discussion. (Discussion and final independent review)
- supports: The AMP proof of concept found that structured independent analysis, information sharing and private re-evaluation reduced self-identified errors or oversights while preserving meaningful differences in perspective. — RS-B528AD8A51A47CB0. The result is context-specific and does not prove the exact procedure improves accuracy in unrelated domains. (Abstract results)
- RS-B528AD8A51A47CB0: Analysis from multiple perspectives (AMP): Applying decision hygiene to analysis of musical structure — https://journals.sagepub.com/doi/10.1177/10298649251385727

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/preserve-disagreement-that-survives-error-correction

---

## Decompose the judgment before scoring the whole

ID: MHC-D-RESEARCH-0410 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/decompose-the-judgment-before-scoring-the-whole

A single number can hide five different arguments.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A complex case receives one holistic score and reviewers disagree without knowing which part drove the difference.

### Avoid when

- Do not decompose away an emergent property that genuinely depends on the whole pattern; leave room for a documented holistic check.

### Explanation

Break the judgment into a small set of components that jointly matter, score or assess those components before the overall verdict, then integrate them using the agreed process. Decomposition makes inconsistency inspectable and prevents one vivid feature from silently dominating everything else.

### Steps

1. When two reviewers disagree overall, the team can locate the component where their judgments diverge.

### Example

Assess deployment risk separately for reversibility, blast radius, observability and data integrity before assigning the overall risk level.

### Check

When two reviewers disagree overall, the team can locate the component where their judgments diverge.

### Limits

- Do not decompose away an emergent property that genuinely depends on the whole pattern; leave room for a documented holistic check.

### Evidence and sources

- supports: A century of selection research reviewed by Annual Reviews supports decomposing complex judgments into components and applying standards consistently to reduce noise. — RS-755DDB8F19ABC634. Decomposition can omit emergent qualities when components do not capture the real judgment target. (Structuring judgment process)
- RS-755DDB8F19ABC634: Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research — https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-120920-050708

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/measure-agreement-on-the-criteria-not-only-the-verdict

---

## Use the same rubric for comparable cases

ID: MHC-D-RESEARCH-0411 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/use-the-same-rubric-for-comparable-cases

A standard that changes with the case is not doing much standardizing.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Repeated cases should be judged consistently but reviewers rely on a shifting intuitive standard.

### Avoid when

- Shared rubrics can be inappropriate where multiple legitimate perspectives are the object of inquiry rather than noise to eliminate.

### Explanation

Define the criteria and scale anchors before reviewing the batch, then apply them to each comparable case. Include examples near category boundaries when possible. If the rubric must change, version it and decide whether earlier cases need re-rating under the new rule.

### Checklist

- Criteria are written before the current case is scored.
- Scale anchors describe observable differences rather than vague adjectives.
- Comparable cases use the same current rubric version.
- Rubric changes are recorded with a reason.
- Material changes trigger a decision about re-rating earlier cases.

### Example

All enhancement requests use the same impact, urgency, reversibility and evidence anchors instead of a new interpretation for each requester.

### Check

A reviewer can explain why two similar cases received different scores using the rubric rather than personal preference.

### Limits

- Shared rubrics can be inappropriate where multiple legitimate perspectives are the object of inquiry rather than noise to eliminate.

### Evidence and sources

- supports: The same workplace judgment review supports agreeing on standards and applying them consistently across comparable cases as a noise-reduction strategy. — RS-755DDB8F19ABC634. A common standard is useful only when the relevant perspective is intended to be shared; some creative or exploratory judgments legitimately differ. (Agreed standards and consistent application)
- RS-755DDB8F19ABC634: Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research — https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-120920-050708

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/do-not-assume-a-rubric-reduced-noise-measure-it

---

## Standardize irrelevant context across cases

ID: MHC-D-RESEARCH-0412 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/standardize-irrelevant-context-across-cases

If the paper color should not matter, do not let it become a hidden experimental variable.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Judgments should depend on case content but may also be influenced by order, formatting, identity or presentation differences.

### Avoid when

- Never blind information that is actually required for safety, authorization, legal context or legitimate interpretation.

### Explanation

Identify context features that should not influence the judgment and hold them constant or randomize them across cases. Use the same information order and presentation where comparison matters. When identity or source status is not decision-relevant, blind it during the initial judgment if operationally and ethically appropriate.

### Steps

1. Which presentation details should have zero influence on this judgment?
2. Can they be standardized or randomized?
3. Is reviewer order exposing later cases to fatigue or learning effects?
4. Does identity need to be visible for the decision itself?

### Example

Present candidate incident reports in the same structure and randomize order instead of placing the most politically visible case first.

### Check

A case would receive the same intended evidence even if its non-relevant presentation details changed.

### Limits

- Never blind information that is actually required for safety, authorization, legal context or legitimate interpretation.

### Evidence and sources

- supports: Decision-noise research distinguishes level noise, stable pattern noise and occasion noise, which arise from different sources and require different diagnostics. — RS-687C74439B4BD8F2. The taxonomy is a conceptual model; real systems can contain several forms of noise at once. (Noise taxonomy)
- RS-687C74439B4BD8F2: Human judgment error in the intensive care unit: a perspective on bias and noise — https://link.springer.com/article/10.1186/s13054-025-05315-9

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/run-a-duplicate-case-noise-audit

---

## Run a duplicate-case noise audit

ID: MHC-D-RESEARCH-0413 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/run-a-duplicate-case-noise-audit

You cannot see noise from one judgment of one case.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A judgment process is important but nobody knows how consistently the same evidence is treated.

### Avoid when

- Repeated cases can be remembered, and small samples are noisy themselves; use the audit as a diagnostic rather than a precision score.

### Explanation

Insert a small set of repeated or equivalently constructed cases without telling reviewers which ones are duplicates. Compare the same reviewer's judgments across time and different reviewers' judgments on the same evidence. Use the spread to identify where the process needs clearer evidence, scale anchors or training.

### Steps

1. Duplicate or equivalent cases preserve the decision-relevant evidence.
2. Reviewers do not know which items are repeats during scoring.
3. Within-reviewer and between-reviewer variation are calculated separately.
4. Large disagreements are inspected at the component or criterion level.
5. The audit is used to improve the process, not rank people from tiny samples.

### Example

Reinsert several anonymized past support-priority cases into a quarterly triage review and compare ratings.

### Check

The team has an empirical estimate of unwanted variability on nominally equivalent evidence.

### Limits

- Repeated cases can be remembered, and small samples are noisy themselves; use the audit as a diagnostic rather than a precision score.

### Evidence and sources

- supports: Decision-noise research distinguishes level noise, stable pattern noise and occasion noise, which arise from different sources and require different diagnostics. — RS-687C74439B4BD8F2. The taxonomy is a conceptual model; real systems can contain several forms of noise at once. (Noise taxonomy)
- supports: A 2026 mixed-methods experiment found both systematic bias and level noise among 2,763 professional prescribing judgments presented with the same decision context. — RS-F7DE75C7534D4A9F. The specific amount and direction of variation are domain-dependent; this does not quantify noise in other professions. (Abstract)
- RS-687C74439B4BD8F2: Human judgment error in the intensive care unit: a perspective on bias and noise — https://link.springer.com/article/10.1186/s13054-025-05315-9
- RS-F7DE75C7534D4A9F: Managing noise through nudges: a mixed methods study on decision-hygiene in public management — https://www.tandfonline.com/doi/full/10.1080/14719037.2026.2654197

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/name-the-noise-before-choosing-the-fix

---

## Name the noise before choosing the fix

ID: MHC-D-RESEARCH-0414 · Version: 0.1.0 · Kind: concept
Source: https://vedokrok.com/knowledge/name-the-noise-before-choosing-the-fix

Not all inconsistency comes from the same place.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A review process is inconsistent and the team proposes one generic remedy.

### Avoid when

- Real systems often mix all three noise forms, and some variation may represent legitimate perspective differences.

### Explanation

Separate three possibilities. Level noise: reviewers are systematically harsher or more lenient overall. Pattern noise: reviewers react differently to particular case features. Occasion noise: the same reviewer changes with time, context or random variation. Diagnose which pattern dominates before choosing calibration, rubric, aggregation or workflow changes.

### Example

Two consultants may agree on average severity but disagree sharply whenever data loss is involved; that is different from one consultant rating everything higher.

### Check

The proposed process change targets an observed form of variability rather than the word 'noise' in general.

### Limits

- Real systems often mix all three noise forms, and some variation may represent legitimate perspective differences.

### Evidence and sources

- supports: Decision-noise research distinguishes level noise, stable pattern noise and occasion noise, which arise from different sources and require different diagnostics. — RS-687C74439B4BD8F2. The taxonomy is a conceptual model; real systems can contain several forms of noise at once. (Noise taxonomy)
- RS-687C74439B4BD8F2: Human judgment error in the intensive care unit: a perspective on bias and noise — https://link.springer.com/article/10.1186/s13054-025-05315-9

No review details supplied.

---

## Average independent estimates before debating one number

ID: MHC-D-RESEARCH-0415 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/average-independent-estimates-before-debating-one-number

A group average built before the argument preserves more independent signal than one built after the room converges.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Several people estimate the same underlying quantity and no one has unique authoritative information.

### Avoid when

- Do not average judgments that answer different questions or encode genuinely different stakeholder values.

### Explanation

Collect estimates independently, calculate a simple or justified weighted aggregate, then use discussion to investigate large outliers and missing information. Keep the independent distribution visible even if the group later chooses a different final number. Aggregation is most useful when the judgments target the same measurable quantity.

### Steps

1. Collect estimates independently.
2. Check that all estimates refer to the same quantity and horizon.
3. Aggregate before group discussion.
4. Inspect the spread and reasons for extreme estimates.
5. Revise only when new information or a justified weighting rule warrants it.

### Example

Collect independent migration-duration estimates from four experienced leads before discussing the schedule together.

### Check

The baseline forecast contains the information from multiple independent judgments rather than only the negotiated consensus.

### Limits

- Do not average judgments that answer different questions or encode genuinely different stakeholder values.

### Evidence and sources

- supports: Averaging independent judgments is a repeatedly supported method for reducing random judgment error when the judgments target a common quantity. — RS-755DDB8F19ABC634. Averaging is not appropriate when disagreement reflects different legitimate objectives or when one evaluator has unique decisive information. (Aggregation of independent judgments)
- RS-755DDB8F19ABC634: Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research — https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-120920-050708

### Vedokrok editorial review

Average estimates, then check what everyone missed

Separate answers can still share the same blind spot. An average does not add a task that every estimate left out.

#### Keep the first answers independent

Ask people to estimate the same quantity, scope and time horizon before seeing each other’s numbers. Keep the initial answers and assumptions, calculate a simple or justified weighted average, then discuss missing work and large differences. Change the estimate when new information warrants it, rather than hiding the spread behind a negotiated number.

#### Check common errors as well as disagreement

In a hypothetical project, three people estimate 8, 10 and 12 hours. The average is 10 hours. If each omitted the same 4-hour transfer step, averaging cannot recover that missing work. Independently collected answers need not have independent errors: people may share a source, assumption or omission. Do not average different objectives, or overrule decisive information merely because it came from one person.

**Editorial takeaway:** Keep both the range of estimates and the shared assumptions visible. The arithmetic example illustrates a limitation; it is not a measured result from a study.

#### Editorial evidence

- supports: The review abstract identifies structured judgment and aggregation of independent judgments as approaches to reducing judgment noise. — RS-755DDB8F19ABC634. Publisher abstract and metadata inspected on 2026-10-05; detailed methods and effect estimates were not accessible in this appraisal. (Abstract)
- limits: The accessible abstract supports a noise-reduction rationale, but does not establish that averaging removes shared errors. — RS-755DDB8F19ABC634. Only the publisher abstract was inspected. The 8/10/12-hour example and common-omission check are original editorial reasoning, not results attributed to this review; no conclusion about the inaccessible full text is asserted. (Abstract)
- RS-755DDB8F19ABC634: Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research — https://www.annualreviews.org/content/journals/10.1146/annurev-orgpsych-120920-050708

Editorial version: 1.0.0 · evidence checked 2026-10-05 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/discuss-the-reasons-then-re-rate-privately

---

## Preserve disagreement that survives error correction

ID: MHC-D-RESEARCH-0416 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/preserve-disagreement-that-survives-error-correction

Consistency is not the goal when the differences themselves contain information.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A decision-hygiene process is treating all variation as something to eliminate.

### Avoid when

- Do not label avoidable inconsistency 'diversity' to escape process improvement; first test shared evidence and criteria.

### Explanation

After reviewers have seen the same evidence, corrected clear oversights and clarified criteria, preserve remaining disagreement when it reflects defensible perspectives or objectives. Record the reason for the split instead of forcing a false consensus. Noise reduction should remove accidental variability, not diversity of legitimate models.

### Example

Two experts can agree on all facts yet differ on how much reversibility versus speed should matter; show that tradeoff instead of averaging their preference into an unexplained midpoint.

### Check

Residual disagreement has an articulated substantive reason rather than an unexamined mistake or scale mismatch.

### Limits

- Do not label avoidable inconsistency 'diversity' to escape process improvement; first test shared evidence and criteria.

### Evidence and sources

- supports: The AMP proof of concept found that structured independent analysis, information sharing and private re-evaluation reduced self-identified errors or oversights while preserving meaningful differences in perspective. — RS-B528AD8A51A47CB0. The result is context-specific and does not prove the exact procedure improves accuracy in unrelated domains. (Abstract results)
- RS-B528AD8A51A47CB0: Analysis from multiple perspectives (AMP): Applying decision hygiene to analysis of musical structure — https://journals.sagepub.com/doi/10.1177/10298649251385727

No review details supplied.

---

## Do not assume a rubric reduced noise—measure it

ID: MHC-D-RESEARCH-0417 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/do-not-assume-a-rubric-reduced-noise-measure-it

A rubric can look structured while reviewers still interpret every line differently.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A team introduces a checklist or criteria set and declares the judgment process standardized.

### Avoid when

- The IFCN null result is from a clinical expert task; it motivates measurement, not a claim that rubrics generally fail.

### Explanation

Compare reliability before and after the structured criteria on repeated or shared cases. If agreement does not improve, inspect criterion definitions, training examples and how criteria are integrated into the final judgment. Keep the null result: structure that does not change reliability may still aid explanation, but it has not earned a noise-reduction claim.

### Example

After adding a six-item architecture-risk checklist, compare whether reviewers actually converge more on the same proposals.

### Check

The team can distinguish 'we added structure' from 'the structure measurably improved consistency.'

### Limits

- The IFCN null result is from a clinical expert task; it motivates measurement, not a claim that rubrics generally fail.

### Evidence and sources

- limits: The 2025 IFCN study found that explicitly adding six expert criteria did not materially improve inter-rater reliability, performance or overall calibration in the studied expert judgments. — RS-4DC9763DF7D5A389. This null result applies to one clinical judgment task and does not imply that structured criteria are generally useless. (Abstract results)
- RS-4DC9763DF7D5A389: Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability — https://www.sciencedirect.com/science/article/pii/S1388245725003268

No review details supplied.

---

## Measure agreement on the criteria, not only the verdict

ID: MHC-D-RESEARCH-0418 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/measure-agreement-on-the-criteria-not-only-the-verdict

Two people can disagree on the verdict because they never agreed on what the evidence meant one step earlier.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Reviewers disagree on final decisions and the team cannot tell whether the issue is evidence, criteria or synthesis.

### Avoid when

- Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.

### Explanation

Capture reviewer judgments on the important component criteria as well as the final outcome. Calculate or inspect agreement at each layer. Low criterion agreement suggests definitions, examples or evidence interpretation need work; high criterion agreement with different final verdicts points instead to weighting or integration rules.

### Steps

1. The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.

### Example

Reviewers may agree that a change has high blast radius and weak rollback but disagree on the overall approval because their weighting differs.

### Check

The team can locate whether disagreement enters at evidence interpretation, criterion rating or final synthesis.

### Limits

- Agreement is not accuracy; reviewers can consistently agree on the wrong interpretation.

### Evidence and sources

- supports: In the IFCN study, agreement on individual criteria ranged only from fair to moderate even though the criteria were explicitly defined. — RS-4DC9763DF7D5A389. Criterion-level agreement may depend on training, examples and the intrinsic ambiguity of the observed signal. (Inter-rater agreement on individual criteria)
- RS-4DC9763DF7D5A389: Utility of the IFCN criteria for identifying interictal epileptiform discharges by experts: A decision hygiene approach to improve inter-rater reliability — https://www.sciencedirect.com/science/article/pii/S1388245725003268

No review details supplied.
