Principle

Validate confidence before using it to assign decisions

A score can rank answers usefully and still be a poor probability. Check calibration, error discrimination and the decision rule separately.

About 2 min read

Before you use it

Use when
  • A team wants a rule such as 'AI decides when AI is more confident; human decides otherwise.'

Keep this limit in mind

  • The theoretical complementarity model assumes confidence signals more disciplined than most ad hoc workplace ratings.

Current editorial reading

Ask two different questions

Calibration asks whether stated probabilities match observed correctness. Discrimination asks whether confidence tends to be higher on correct answers than on errors. Test both human and AI signals on relevant labeled cases. Do not compare their raw numbers just because both use percentages. If the scales or task coverage are unreliable, use evidence, task-specific routing or independent review instead.

Test the rule, not just its ingredients

Suppose an AI ranks difficult cases sensibly but consistently reports probabilities above its observed accuracy. Its score may contain information without being directly comparable with a reviewer’s percentage. Develop the routing rule on one set of cases, then compare its accuracy and consequential errors with simpler alternatives on held-out cases. The cited model studies optimal combination; it does not guarantee that choosing the larger displayed number improves a workplace decision.

Take it with you

Adopt confidence-based routing only after the actual rule earns its place on the task. Keep a review path for important unresolved cases and recheck after material changes.

Sources and evidence (1)

What we checked

Supports

Calibration and metacognitive sensitivity answer distinct questions about confidence and correctness.

Sensitivity is also called discrimination or resolution. Neither high average confidence nor calibration alone establishes useful case ranking.

Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Author-hosted published PDF, sections 1.1–1.2, pages 1–2

Limits

Theoretical complementarity depends on the combination rule and assumptions, including the relationship between human and AI errors.

The inspected model distinguishes independent from correlated decisions. It is not a universal endorsement of raw maximum-confidence routing.

Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Author-hosted published PDF, sections 5.1–5.2, pages 5–6

Editorial sources (1)

Editorial version 1.0.0 · Evidence checked 2026-10-04

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingValidate confidence before using it to assign decisions

Original edition evidence and sources

These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.

Supports

Metacognitive sensitivity concerns how well confidence distinguishes correct from incorrect decisions, which is different from average confidence or simple calibration.

Formal metacognitive metrics require enough labeled decisions; a single confidence value cannot establish sensitivity.

Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Abstract and theoretical model

Supports

The 2026 mathematical model shows that human and AI metacognitive sensitivity jointly affect achievable combined accuracy when confidence is used to combine decisions.

The Bayes-optimal assumptions are stronger than ordinary workplace decision support.

Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Analytic results

All sources (1)
Original edition, downloads and provenance

The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.

Version 0.1.0

No review details are supplied for this edition.

Read the original edition and overlay as JSON · Release details