Principle
Validate confidence before using it to assign decisions
A score can rank answers usefully and still be a poor probability. Check calibration, error discrimination and the decision rule separately.
Before you use it
- Use when
- A team wants a rule such as 'AI decides when AI is more confident; human decides otherwise.'
Keep this limit in mind
- The theoretical complementarity model assumes confidence signals more disciplined than most ad hoc workplace ratings.
Current editorial reading
Ask two different questions
Calibration asks whether stated probabilities match observed correctness. Discrimination asks whether confidence tends to be higher on correct answers than on errors. Test both human and AI signals on relevant labeled cases. Do not compare their raw numbers just because both use percentages. If the scales or task coverage are unreliable, use evidence, task-specific routing or independent review instead.
Test the rule, not just its ingredients
Suppose an AI ranks difficult cases sensibly but consistently reports probabilities above its observed accuracy. Its score may contain information without being directly comparable with a reviewer’s percentage. Develop the routing rule on one set of cases, then compare its accuracy and consequential errors with simpler alternatives on held-out cases. The cited model studies optimal combination; it does not guarantee that choosing the larger displayed number improves a workplace decision.
Take it with you
Adopt confidence-based routing only after the actual rule earns its place on the task. Keep a review path for important unresolved cases and recheck after material changes.
Sources and evidence (1)
What we checked
Calibration and metacognitive sensitivity answer distinct questions about confidence and correctness.
Sensitivity is also called discrimination or resolution. Neither high average confidence nor calibration alone establishes useful case ranking.
Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Author-hosted published PDF, sections 1.1–1.2, pages 1–2
Theoretical complementarity depends on the combination rule and assumptions, including the relationship between human and AI errors.
The inspected model distinguishes independent from correlated decisions. It is not a universal endorsement of raw maximum-confidence routing.
Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Author-hosted published PDF, sections 5.1–5.2, pages 5–6
Editorial sources (1)
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingValidate confidence before using it to assign decisions
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaMake disagreement between you and AI a review trigger
Original edition evidence and sources
These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.
Metacognitive sensitivity concerns how well confidence distinguishes correct from incorrect decisions, which is different from average confidence or simple calibration.
Formal metacognitive metrics require enough labeled decisions; a single confidence value cannot establish sensitivity.
Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Abstract and theoretical model
The 2026 mathematical model shows that human and AI metacognitive sensitivity jointly affect achievable combined accuracy when confidence is used to combine decisions.
The Bayes-optimal assumptions are stronger than ordinary workplace decision support.
Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Analytic results
Original edition, downloads and provenance
The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.
Version 0.1.0
No review details are supplied for this edition.
Read the original edition and overlay as JSON · Release details