Protocol

Repeat agent evaluations without selecting the best run

One successful run shows that success is possible. To estimate repeatability, preserve the failures too, and state exactly what your success metric counts.

About 2 min read

Before you use it

Use when
  • One successful run is being treated as proof that an agent workflow is reliable.

Keep this limit in mind

  • Trial count should match cost and risk; small samples still leave uncertainty.

Current editorial reading

Make attempts comparable

Choose representative tasks and a trial count before inspecting results. Keep model, tools, task and grading versions fixed; reset each trial to a clean starting state. Save every outcome, failure type, duration and cost. For example, report seven successes and three failures on a task across ten attempts, rather than showing only the best output. Ten is an illustration, not a recommended sample size.

Name the event being measured

Pass@1 concerns single-attempt success; it is not a name for a cherry-picked favorable run. Pass@k concerns at least one success in k attempts, while pass^k concerns success on every one of k attempts. State the chosen metric, sample size and per-task results. Shared files or resource shortages can connect trials and distort results. Repetition also cannot repair unrepresentative tasks or establish reliability outside the tested conditions.

Take it with you

Keep a complete trial ledger and explain which success event the score estimates. A better best-case result and more dependable single attempts answer different questions.

Sources and evidence (1)

What we checked

Supports

The guidance distinguishes individual trials and the different success events represented by pass@k and pass^k.

Official engineering guidance published 2026-01-09. Selected full-text passages read. This corrects the original treatment of pass@1; no estimator formula or confidence interval is prescribed.

Demystifying evals for AI agents · The structure of an evaluation; How to think about non-determinism in evaluations for agents

Limits

Shared trial state can inflate results, while shared environment constraints can produce correlated failures.

Selected full-text passage read. Starting clean is necessary for a useful comparison but does not prove statistical independence or coverage of real-world tasks.

Demystifying evals for AI agents · Step 4: Build a robust eval harness with a stable environment

Editorial sources (1)

Editorial version 1.0.0 · Evidence checked 2026-10-04

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingRepeat agent evaluations without selecting the best run

Original edition evidence and sources

These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.

Supports

Anthropic's agent-eval guidance defines separate trials because model outputs vary and recommends multiple trials for more consistent evaluation.

Trial count should match cost and risk; small samples still leave uncertainty.

Demystifying evals for AI agents · See source record

All sources (1)
Original edition, downloads and provenance

The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.

Version 0.1.0

No review details are supplied for this edition.

Read the original edition and overlay as JSON · Release details