Principle
Judge AI confidence by whether it separates right from wrong
A confidence score is useful when it knows which answers deserve confidence.
When it fits
- A system reports confidence scores and the team treats high average confidence as evidence of useful uncertainty.
When to avoid it
- Metacognitive sensitivity can drift when data, model or prompts change; revalidate it over time.
Why it matters
Evaluate whether higher AI confidence actually corresponds to more correct outputs and lower confidence to more errors. This discrimination—metacognitive sensitivity—is different from the model simply sounding or scoring confidently. Use a labeled task set before letting confidence drive routing or human reliance.
An example
If an extraction model gives 95% confidence to both correct and wrong tax IDs, the confidence field is not useful for selective review.
Check your result
Higher confidence demonstrably separates more reliable outputs from less reliable ones on the relevant task class.
Keep this limit in mind
- Metacognitive sensitivity can drift when data, model or prompts change; revalidate it over time.
Connected ideas
Use beforeDo not expose AI confidence until you have tested its calibration
Evidence and sources
Metacognitive sensitivity concerns how well confidence distinguishes correct from incorrect decisions, which is different from average confidence or simple calibration.
Formal metacognitive metrics require enough labeled decisions; a single confidence value cannot establish sensitivity.
Modeling the joint impact of human and AI metacognitive sensitivity on human-AI collaboration · Abstract and theoretical model