# Keep an eval set that can embarrass the agent

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

---

## Keep an eval set that can embarrass the agent

ID: MHC-D-RESEARCH-0319 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/keep-an-eval-set-that-can-embarrass-the-agent

A demo asks whether the system can succeed once. An eval asks where it reliably fails.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI workflow will be reused and quality cannot be judged from one polished demo.

### Avoid when

- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.

### Explanation

Maintain a versioned set of representative tasks, edge cases and known failure examples with explicit pass criteria. Record the model or system version, relevant configuration, metrics and evaluation tooling. Add important real failures after review so the set becomes harder in useful ways instead of merely larger.

### Steps

1. Cases represent the real task distribution, not only easy examples.
2. Known failure modes and boundary cases are included.
3. Each case has an observable pass criterion.
4. Model, prompt/tool configuration and evaluation version are recorded.
5. New cases are added deliberately when they expose a distinct risk or capability gap.

### Example

A document-extraction eval includes clean forms, missing fields, conflicting fields and a case where the correct answer is 'unknown.'

### Check

A new system version can be compared against the same defined cases without reconstructing the test from memory.

### Limits

- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.

### Evidence and sources

- supports: NIST AI RMF calls for documenting AI test sets, metrics and evaluation tools. — RS-89634FBC4E1347EC. A documented evaluation set can still be unrepresentative or weak; provenance and coverage must be judged separately. (Measure 2.1)
- RS-89634FBC4E1347EC: AI RMF Core — https://airc.nist.gov/airmf-resources/airmf/5-sec-core/

### Vedokrok editorial review

A regression set should remember failures without becoming the whole exam

A stable eval set is valuable because it makes regressions visible. The trap is using the same familiar cases as the only proof that a changing agent still works. Once those cases guide development, the team also needs evidence from work the system has not been tuned around.

#### Separate regression from generalization

Keep a stable core of representative tasks, edge cases and known failures. Its job is to answer a practical question: did this change break something we already know matters? NIST’s AI RMF supports documenting test sets, metrics and evaluation tools so results can be repeated and interpreted. But a regression suite and a fresh test answer different questions. The first protects known behavior; the second gives a better check on whether improvements extend beyond the cases the team has repeatedly seen.

#### Repeated reuse can turn the test into part of the development loop

This is not only a language-model problem. Work on adaptive data analysis shows that repeatedly using a holdout to choose what to try next can overfit the holdout itself. LLM evaluation adds another route: benchmark items may already appear in training data. A 2025 study of 83 software-engineering benchmarks found leakage was uneven rather than universal, but heavily leaked benchmarks existed and the authors reported that leakage affected measured model performance. A rising score therefore needs a provenance question as well as a scoring question.

#### Fresh cases help, but freshness is not a magic control

LiveBench responds to contamination risk by adding questions from recent sources and updating the benchmark over time. That is useful evidence for a rolling or held-out slice, not a reason to discard the stable regression set. Fresh data can also differ in difficulty or distribution. In the ImageNet replication study, accuracy fell on newly collected test sets, yet the authors concluded the drop was better explained by the new images being slightly harder than by adaptive overfitting. If a fresh slice scores worse, investigate both possibilities before declaring the model overfit.

#### Three questions before you trust the score

Should every production failure enter the eval? Add a failure when it represents a distinct, important risk or capability gap, but put it in the regression set; do not let all future measurement collapse into known cases. How large must the eval be? There is no universal number: coverage of real task types and uncertainty around the score matter more than a round target. Does 100% mean the agent is ready? Only for the defined cases, configuration and scoring rule. Keep a fresh or time-separated check when generalization matters, and record which set influenced development.

**Editorial takeaway:** Use two complementary views: a stable regression set that remembers important failures, and a fresh or protected slice that the development loop has not repeatedly optimized against. Version both, keep scoring rules explicit, and treat a score as evidence about those cases—not a certificate for unseen work.

#### Editorial evidence

- supports: NIST AI RMF Measure 2.1 calls for documenting test sets, metrics and details of tools used during test, evaluation, validation and verification. — nist-ai-rmf-core-measure. This supports versioned, repeatable evaluation records. NIST does not say that one finite set proves performance outside the tested conditions. (Measure 2.1)
- limits: Adaptive reuse of a holdout can overfit the holdout when observed results influence later hypotheses or choices. — dwork-holdout-2015. The paper is about adaptive data analysis rather than agent-eval operations, so the regression-versus-fresh-slice workflow here is an editorial application of the general reuse problem. (Abstract)
- challenges: Fresh CIFAR-10 and ImageNet test sets produced lower accuracy than the original test sets, but the authors attributed the drop mainly to the new data being slightly harder rather than to adaptive test-set overfitting. — recht-imagenet-2019. A worse fresh-set score is therefore not, by itself, proof that the development process overfit the old eval. Distribution and difficulty need checking too. (Abstract)
- contextualizes: LiveBench uses frequently updated questions from recent sources and objective scoring to reduce test-set contamination and keep the benchmark useful as models change. — livebench-iclr-2025. This is a benchmark-design example, not evidence that monthly refresh is the correct cadence for every internal agent eval. (Abstract)
- challenges: A 2025 analysis of 83 software-engineering benchmarks found benchmark leakage into StarCoder training data was uneven, with some benchmarks heavily affected, and reported that leakage materially affected evaluation results. — lessleak-2025. The study concerns one open-model training corpus and software-engineering benchmarks. Its leakage rates should not be generalized to unrelated models or private eval sets. (Abstract)
- nist-ai-rmf-core-measure: AI RMF Core — https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ — National Institute of Standards and Technology
- dwork-holdout-2015: Generalization in Adaptive Data Analysis and Holdout Reuse — https://proceedings.neurips.cc/paper/2015/hash/bad5f33780c42f2588878a9d07405083-Abstract.html — Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth
- recht-imagenet-2019: Do ImageNet Classifiers Generalize to ImageNet? — https://proceedings.mlr.press/v97/recht19a.html — Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar
- livebench-iclr-2025: LiveBench: A Challenging, Contamination-Limited LLM Benchmark — https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html
- lessleak-2025: LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks — https://arxiv.org/abs/2502.06215 — Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, David Lo

Editorial version: 1.0.0 · evidence checked 2026-09-25 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/test-the-agent-under-conditions-that-resemble-the-real-job
