Protocol
Do not expose AI confidence until you have tested its calibration
A number beside an answer can create trust even when the number is wrong.
When it fits
- A product plans to show users a confidence percentage because it seems transparent.
When to avoid it
- Even calibrated confidence cannot replace source evidence or authorization for high-stakes decisions.
Why it matters
Before displaying confidence, compare reported probabilities with observed accuracy and inspect high-confidence mistakes. Test whether the display improves appropriate reliance rather than merely increasing agreement. If calibration is unstable, prefer bounded uncertainty language, failure-mode information or no confidence score until the signal is better validated.
Steps
- Confidence is evaluated against labeled outcomes.
- Miscalibration is measured by task segment, not only overall.
- High-confidence errors receive explicit review.
- User behavior with and without the display is tested where stakes justify it.
- The display is versioned with the model/configuration it was validated on.
An example
Do not show '97% confident' on a compliance recommendation until that confidence has been evaluated on comparable compliance cases.
Check your result
The confidence display has evidence of both statistical quality and useful behavioral effect.
Keep this limit in mind
- Even calibrated confidence cannot replace source evidence or authorization for high-stakes decisions.
Connected ideas
Useful withPair confidence with known failure modes
Evidence and sources
An AAAI 2026 experiment found well-calibrated AI confidence improved participant decision accuracy more than miscalibrated confidence, while miscalibrated cues increased reliance-related errors.
The task involved logic puzzles and controlled confidence manipulations.
Too Sure for Our Own Good: A User Study on AI Confidence and Human Reliance · Results
A 2026 experiment found visual uncertainty cues could increase users' subjective confidence-accuracy discrimination while simultaneously increasing behavioral overreliance on incorrect LLM outputs.
Interface effects depend on cue design and task; more uncertainty display is not automatically harmful.