Protocol
Repeat agent evaluations without selecting the best run
One successful run shows that success is possible. To estimate repeatability, preserve the failures too, and state exactly what your success metric counts.
Before you use it
- Use when
- One successful run is being treated as proof that an agent workflow is reliable.
Keep this limit in mind
- Trial count should match cost and risk; small samples still leave uncertainty.
Current editorial reading
Make attempts comparable
Choose representative tasks and a trial count before inspecting results. Keep model, tools, task and grading versions fixed; reset each trial to a clean starting state. Save every outcome, failure type, duration and cost. For example, report seven successes and three failures on a task across ten attempts, rather than showing only the best output. Ten is an illustration, not a recommended sample size.
Name the event being measured
Pass@1 concerns single-attempt success; it is not a name for a cherry-picked favorable run. Pass@k concerns at least one success in k attempts, while pass^k concerns success on every one of k attempts. State the chosen metric, sample size and per-task results. Shared files or resource shortages can connect trials and distort results. Repetition also cannot repair unrepresentative tasks or establish reliability outside the tested conditions.
Take it with you
Keep a complete trial ledger and explain which success event the score estimates. A better best-case result and more dependable single attempts answer different questions.
Sources and evidence (1)
What we checked
The guidance distinguishes individual trials and the different success events represented by pass@k and pass^k.
Official engineering guidance published 2026-01-09. Selected full-text passages read. This corrects the original treatment of pass@1; no estimator formula or confidence interval is prescribed.
Demystifying evals for AI agents · The structure of an evaluation; How to think about non-determinism in evaluations for agents
Shared trial state can inflate results, while shared environment constraints can produce correlated failures.
Selected full-text passage read. Starting clean is necessary for a useful comparison but does not prove statistical independence or coverage of real-world tasks.
Demystifying evals for AI agents · Step 4: Build a robust eval harness with a stable environment
Editorial sources (1)
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingRepeat agent evaluations without selecting the best run
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaTest an intervention across a replay cohort
Original edition evidence and sources
These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.
Anthropic's agent-eval guidance defines separate trials because model outputs vary and recommends multiple trials for more consistent evaluation.
Trial count should match cost and risk; small samples still leave uncertainty.
Demystifying evals for AI agents · See source record
All sources (1)
Original edition, downloads and provenance
The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.
Version 0.1.0
No review details are supplied for this edition.
Read the original edition and overlay as JSON · Release details