Protocol
Keep an eval set that can embarrass the agent
A demo asks whether the system can succeed once. An eval asks where it reliably fails.
When it fits
- An AI workflow will be reused and quality cannot be judged from one polished demo.
When to avoid it
- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.
Why it matters
Maintain a versioned set of representative tasks, edge cases and known failure examples with explicit pass criteria. Record the model or system version, relevant configuration, metrics and evaluation tooling. Add important real failures after review so the set becomes harder in useful ways instead of merely larger.
Steps
- Cases represent the real task distribution, not only easy examples.
- Known failure modes and boundary cases are included.
- Each case has an observable pass criterion.
- Model, prompt/tool configuration and evaluation version are recorded.
- New cases are added deliberately when they expose a distinct risk or capability gap.
An example
A document-extraction eval includes clean forms, missing fields, conflicting fields and a case where the correct answer is 'unknown.'
Check your result
A new system version can be compared against the same defined cases without reconstructing the test from memory.
Keep this limit in mind
- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.
Connected ideas
Use beforeTest the agent under conditions that resemble the real job
Evidence and sources
NIST AI RMF calls for documenting AI test sets, metrics and evaluation tools.
A documented evaluation set can still be unrepresentative or weak; provenance and coverage must be judged separately.
AI RMF Core · Measure 2.1