Checklist
Use the same rubric for comparable cases
A standard that changes with the case is not doing much standardizing.
When it fits
- Repeated cases should be judged consistently but reviewers rely on a shifting intuitive standard.
When to avoid it
- Shared rubrics can be inappropriate where multiple legitimate perspectives are the object of inquiry rather than noise to eliminate.
Checklist
- Criteria are written before the current case is scored.
- Scale anchors describe observable differences rather than vague adjectives.
- Comparable cases use the same current rubric version.
- Rubric changes are recorded with a reason.
- Material changes trigger a decision about re-rating earlier cases.
Why it matters
Define the criteria and scale anchors before reviewing the batch, then apply them to each comparable case. Include examples near category boundaries when possible. If the rubric must change, version it and decide whether earlier cases need re-rating under the new rule.
An example
All enhancement requests use the same impact, urgency, reversibility and evidence anchors instead of a new interpretation for each requester.
Check your result
A reviewer can explain why two similar cases received different scores using the rubric rather than personal preference.
Keep this limit in mind
- Shared rubrics can be inappropriate where multiple legitimate perspectives are the object of inquiry rather than noise to eliminate.
Connected ideas
Useful withDo not assume a rubric reduced noise—measure it
Evidence and sources
The same workplace judgment review supports agreeing on standards and applying them consistently across comparable cases as a noise-reduction strategy.
A common standard is useful only when the relevant perspective is intended to be shared; some creative or exploratory judgments legitimately differ.
Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research · Agreed standards and consistent application