# Route cases using both human and AI confidence only after both are calibrated

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

---

## Route cases using both human and AI confidence only after both are calibrated

ID: MHC-D-RESEARCH-0505 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/route-cases-using-both-human-and-ai-confidence-only-after-both-are-calibrated

Comparing two uncalibrated confidence numbers creates a precise-looking coin toss.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A team wants a rule such as 'AI decides when AI is more confident; human decides otherwise.'

### Avoid when

- The theoretical complementarity model assumes confidence signals more disciplined than most ad hoc workplace ratings.

### Explanation

Before using relative confidence to assign authority, test whether human confidence and AI confidence each discriminate correctness on the relevant task. If one signal is weak, use task class, evidence or independent review instead. Relative-confidence routing is valuable only when the inputs carry reliable metacognitive information.

### Example

Do not let an LLM override an experienced operator merely because it outputs 0.94 and the operator says 80% until those scales are validated.

### Check

Confidence-based routing outperforms or meaningfully complements simpler task-specific routing on held-out cases.

### Limits

- The theoretical complementarity model assumes confidence signals more disciplined than most ad hoc workplace ratings.

### Evidence and sources

- supports: Metacognitive sensitivity concerns how well confidence distinguishes correct from incorrect decisions, which is different from average confidence or simple calibration. — RS-45C4F121413B315D. Formal metacognitive metrics require enough labeled decisions; a single confidence value cannot establish sensitivity. (Abstract and theoretical model)
- supports: The 2026 mathematical model shows that human and AI metacognitive sensitivity jointly affect achievable combined accuracy when confidence is used to combine decisions. — RS-45C4F121413B315D. The Bayes-optimal assumptions are stronger than ordinary workplace decision support. (Analytic results)
- RS-45C4F121413B315D: Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration — https://www.sciencedirect.com/science/article/pii/S0022249626000192

### Vedokrok editorial review

Validate confidence before using it to assign decisions

A score can rank answers usefully and still be a poor probability. Check calibration, error discrimination and the decision rule separately.

#### Ask two different questions

Calibration asks whether stated probabilities match observed correctness. Discrimination asks whether confidence tends to be higher on correct answers than on errors. Test both human and AI signals on relevant labeled cases. Do not compare their raw numbers just because both use percentages. If the scales or task coverage are unreliable, use evidence, task-specific routing or independent review instead.

#### Test the rule, not just its ingredients

Suppose an AI ranks difficult cases sensibly but consistently reports probabilities above its observed accuracy. Its score may contain information without being directly comparable with a reviewer’s percentage. Develop the routing rule on one set of cases, then compare its accuracy and consequential errors with simpler alternatives on held-out cases. The cited model studies optimal combination; it does not guarantee that choosing the larger displayed number improves a workplace decision.

**Editorial takeaway:** Adopt confidence-based routing only after the actual rule earns its place on the task. Keep a review path for important unresolved cases and recheck after material changes.

#### Editorial evidence

- supports: Calibration and metacognitive sensitivity answer distinct questions about confidence and correctness. — ai-metasensitivity-2026. Sensitivity is also called discrimination or resolution. Neither high average confidence nor calibration alone establishes useful case ranking. (Author-hosted published PDF, sections 1.1–1.2, pages 1–2)
- limits: Theoretical complementarity depends on the combination rule and assumptions, including the relationship between human and AI errors. — ai-metasensitivity-2026. The inspected model distinguishes independent from correlated decisions. It is not a universal endorsement of raw maximum-confidence routing. (Author-hosted published PDF, sections 5.1–5.2, pages 5–6)
- ai-metasensitivity-2026: Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration — https://www.sciencedirect.com/science/article/pii/S0022249626000192

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
