Template
Judge the expensive error separately from the common one
A thousand correct commas do not cancel one wrong recipient.
When it fits
- An AI tool has an attractive average score but some mistakes would be much more costly than others.
When to avoid it
- Rare failures can be absent from a small test by chance. Consequence weights are judgments and should not be disguised as measured facts.
A template to use
Failure type: [error]. Consequence: [harm]. Observed cases and denominator: [evidence]. Required handling: [gate]. Owner of acceptance: [owner].
Why it matters
Separate error types before deciding whether the system is acceptable. For a classifier, begin with false positives and false negatives; for an assistant, define similarly concrete failure categories. Assign review or blocking rules to consequential failures rather than letting them disappear inside an average.
An example
A routing assistant's harmless category mistakes and messages sent to an unauthorized team are reported separately.
Check your result
The acceptance decision names the critical error counts and rules, not just the overall pass rate.
Keep this limit in mind
- Rare failures can be absent from a small test by chance. Consequence weights are judgments and should not be disguised as measured facts.
Connected ideas
Useful withKeep acceptance examples out of the prompt workshop
Evidence and sources
A confusion matrix separates actual classes from predicted classes and exposes different error directions hidden by an aggregate accuracy figure.
The matrix does not determine the harm or acceptability of each error; those require task-specific judgment.
Metrics and scoring: quantifying the quality of predictions · Confusion matrix