Template
Judge the expensive error separately from the common one
A thousand correct commas do not cancel one wrong recipient.
Before you use it
- Use when
- An AI tool has an attractive average score but some mistakes would be much more costly than others.
Keep this limit in mind
- Rare failures can be absent from a small test by chance. Consequence weights are judgments and should not be disguised as measured facts.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
A template to use
Failure type: [error]. Consequence: [harm]. Observed cases and denominator: [evidence]. Required handling: [gate]. Owner of acceptance: [owner].
Why it matters
Separate error types before deciding whether the system is acceptable. For a classifier, begin with false positives and false negatives; for an assistant, define similarly concrete failure categories. Assign review or blocking rules to consequential failures rather than letting them disappear inside an average.
An example
A routing assistant's harmless category mistakes and messages sent to an unauthorized team are reported separately.
Check your result
The acceptance decision names the critical error counts and rules, not just the overall pass rate.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingJudge the expensive error separately from the common one
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaReport how much work an abstaining system actually covers
Evidence and sources
A confusion matrix separates actual classes from predicted classes and exposes different error directions hidden by an aggregate accuracy figure.
The matrix does not determine the harm or acceptability of each error; those require task-specific judgment.
Metrics and scoring: quantifying the quality of predictions · Confusion matrix
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.