Checklist
Report how much work an abstaining system actually covers
Being right on the easiest fifth is not the same as handling the whole queue.
When it fits
- An AI workflow sends uncertain cases to a human and reports high accuracy on what remains.
When to avoid it
- Raw model confidence is not a validated probability. Evidence from selective classifiers does not provide an automatic safety guarantee for a language-model workflow.
Checklist
- Define what counts as acceptance, abstention and a correct result.
- Report accepted-case errors alongside accepted share and review workload.
- Check whether difficult groups are disproportionately left for people.
Why it matters
Measure both the quality of accepted answers and coverage: the share of incoming cases the system handles. Track the rejected cases and the human work they create. Choose the operating point from observed trade-offs, not from the model's confident tone.
An example
An extractor handles 70 of 100 documents and refers 30. Its accuracy on the 70 must not be reported as accuracy on all 100.
Check your result
The report makes accepted coverage, residual errors and referral load visible together.
Keep this limit in mind
- Raw model confidence is not a validated probability. Evidence from selective classifiers does not provide an automatic safety guarantee for a language-model workflow.
Connected ideas
Useful withJudge the expensive error separately from the common one
Evidence and sources
Supports
Selective classification can trade coverage for lower risk on the cases it accepts, as demonstrated in the cited image-classification setting.
No guarantee transfers to an uncalibrated language-model confidence statement or a new population.
Selective Classification for Deep Neural Networks · Abstract