# Evaluate the same agent task across multiple trials

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

---

## Evaluate the same agent task across multiple trials

ID: MHC-D-RESEARCH-0955 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/evaluate-the-same-agent-task-across-multiple-trials

A nondeterministic system needs more than one audition.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- One successful run is being treated as proof that an agent workflow is reliable.

### Avoid when

- Trial count should match cost and risk; small samples still leave uncertainty.

### Explanation

Run representative tasks multiple times and report distribution, not only pass@1 from a favorable run. Capture success, failure mode, steps, cost and variance. This distinguishes occasional capability from repeatable reliability, especially on longer tasks.

### Steps

1. Reliability claims include repeated trials and failure distribution rather than one anecdote.

### Example

Run the same repository change ten times in isolated copies to see whether the agent consistently edits the right files and completes tests.

### Check

Reliability claims include repeated trials and failure distribution rather than one anecdote.

### Limits

- Trial count should match cost and risk; small samples still leave uncertainty.

### Evidence and sources

- supports: Anthropic's agent-eval guidance defines separate trials because model outputs vary and recommends multiple trials for more consistent evaluation. — RS-EE9379CE3D2ABBDD. Trial count should match cost and risk; small samples still leave uncertainty. (See source record)
- RS-EE9379CE3D2ABBDD: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

### Vedokrok editorial review

Repeat agent evaluations without selecting the best run

One successful run shows that success is possible. To estimate repeatability, preserve the failures too, and state exactly what your success metric counts.

#### Make attempts comparable

Choose representative tasks and a trial count before inspecting results. Keep model, tools, task and grading versions fixed; reset each trial to a clean starting state. Save every outcome, failure type, duration and cost. For example, report seven successes and three failures on a task across ten attempts, rather than showing only the best output. Ten is an illustration, not a recommended sample size.

#### Name the event being measured

Pass@1 concerns single-attempt success; it is not a name for a cherry-picked favorable run. Pass@k concerns at least one success in k attempts, while pass^k concerns success on every one of k attempts. State the chosen metric, sample size and per-task results. Shared files or resource shortages can connect trials and distort results. Repetition also cannot repair unrepresentative tasks or establish reliability outside the tested conditions.

**Editorial takeaway:** Keep a complete trial ledger and explain which success event the score estimates. A better best-case result and more dependable single attempts answer different questions.

#### Editorial evidence

- supports: The guidance distinguishes individual trials and the different success events represented by pass@k and pass^k. — RS-EE9379CE3D2ABBDD. Official engineering guidance published 2026-01-09. Selected full-text passages read. This corrects the original treatment of pass@1; no estimator formula or confidence interval is prescribed. (The structure of an evaluation; How to think about non-determinism in evaluations for agents)
- limits: Shared trial state can inflate results, while shared environment constraints can produce correlated failures. — RS-EE9379CE3D2ABBDD. Selected full-text passage read. Starting clean is necessary for a useful comparison but does not prove statistical independence or coverage of real-world tasks. (Step 4: Build a robust eval harness with a stable environment)
- RS-EE9379CE3D2ABBDD: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/keep-an-eval-set-that-can-embarrass-the-agent
