Protocol
Evaluate the same agent task across multiple trials
A nondeterministic system needs more than one audition.
When it fits
- One successful run is being treated as proof that an agent workflow is reliable.
When to avoid it
- Trial count should match cost and risk; small samples still leave uncertainty.
Why it matters
Run representative tasks multiple times and report distribution, not only pass@1 from a favorable run. Capture success, failure mode, steps, cost and variance. This distinguishes occasional capability from repeatable reliability, especially on longer tasks.
Steps
- Reliability claims include repeated trials and failure distribution rather than one anecdote.
An example
Run the same repository change ten times in isolated copies to see whether the agent consistently edits the right files and completes tests.
Check your result
Reliability claims include repeated trials and failure distribution rather than one anecdote.
Keep this limit in mind
- Trial count should match cost and risk; small samples still leave uncertainty.
Connected ideas
Useful withKeep an eval set that can embarrass the agent
Evidence and sources
Supports
Anthropic's agent-eval guidance defines separate trials because model outputs vary and recommends multiple trials for more consistent evaluation.
Trial count should match cost and risk; small samples still leave uncertainty.
Demystifying evals for AI agents · See source record