Protocol
A regression set should remember failures without becoming the whole exam
A stable eval set is valuable because it makes regressions visible. The trap is using the same familiar cases as the only proof that a changing agent still works. Once those cases guide development, the team also needs evidence from work the system has not been tuned around.
Before you use it
- Use when
- An AI workflow will be reused and quality cannot be judged from one polished demo.
Keep this limit in mind
- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.
Current editorial reading
Separate regression from generalization
Keep a stable core of representative tasks, edge cases and known failures. Its job is to answer a practical question: did this change break something we already know matters? NIST’s AI RMF supports documenting test sets, metrics and evaluation tools so results can be repeated and interpreted. But a regression suite and a fresh test answer different questions. The first protects known behavior; the second gives a better check on whether improvements extend beyond the cases the team has repeatedly seen.
Repeated reuse can turn the test into part of the development loop
This is not only a language-model problem. Work on adaptive data analysis shows that repeatedly using a holdout to choose what to try next can overfit the holdout itself. LLM evaluation adds another route: benchmark items may already appear in training data. A 2025 study of 83 software-engineering benchmarks found leakage was uneven rather than universal, but heavily leaked benchmarks existed and the authors reported that leakage affected measured model performance. A rising score therefore needs a provenance question as well as a scoring question.
Fresh cases help, but freshness is not a magic control
LiveBench responds to contamination risk by adding questions from recent sources and updating the benchmark over time. That is useful evidence for a rolling or held-out slice, not a reason to discard the stable regression set. Fresh data can also differ in difficulty or distribution. In the ImageNet replication study, accuracy fell on newly collected test sets, yet the authors concluded the drop was better explained by the new images being slightly harder than by adaptive overfitting. If a fresh slice scores worse, investigate both possibilities before declaring the model overfit.
Three questions before you trust the score
Should every production failure enter the eval? Add a failure when it represents a distinct, important risk or capability gap, but put it in the regression set; do not let all future measurement collapse into known cases. How large must the eval be? There is no universal number: coverage of real task types and uncertainty around the score matter more than a round target. Does 100% mean the agent is ready? Only for the defined cases, configuration and scoring rule. Keep a fresh or time-separated check when generalization matters, and record which set influenced development.
Take it with you
Use two complementary views: a stable regression set that remembers important failures, and a fresh or protected slice that the development loop has not repeatedly optimized against. Version both, keep scoring rules explicit, and treat a score as evidence about those cases—not a certificate for unseen work.
Sources and evidence (5)
What we checked
NIST AI RMF Measure 2.1 calls for documenting test sets, metrics and details of tools used during test, evaluation, validation and verification.
This supports versioned, repeatable evaluation records. NIST does not say that one finite set proves performance outside the tested conditions.
AI RMF Core · Measure 2.1
Adaptive reuse of a holdout can overfit the holdout when observed results influence later hypotheses or choices.
The paper is about adaptive data analysis rather than agent-eval operations, so the regression-versus-fresh-slice workflow here is an editorial application of the general reuse problem.
Generalization in Adaptive Data Analysis and Holdout Reuse · Abstract
Fresh CIFAR-10 and ImageNet test sets produced lower accuracy than the original test sets, but the authors attributed the drop mainly to the new data being slightly harder rather than to adaptive test-set overfitting.
A worse fresh-set score is therefore not, by itself, proof that the development process overfit the old eval. Distribution and difficulty need checking too.
LiveBench uses frequently updated questions from recent sources and objective scoring to reduce test-set contamination and keep the benchmark useful as models change.
This is a benchmark-design example, not evidence that monthly refresh is the correct cadence for every internal agent eval.
LiveBench: A Challenging, Contamination-Limited LLM Benchmark · Abstract
A 2025 analysis of 83 software-engineering benchmarks found benchmark leakage into StarCoder training data was uneven, with some benchmarks heavily affected, and reported that leakage materially affected evaluation results.
The study concerns one open-model training corpus and software-engineering benchmarks. Its leakage rates should not be generalized to unrelated models or private eval sets.
Editorial sources (5)
- AI RMF Core — National Institute of Standards and Technology
- Generalization in Adaptive Data Analysis and Holdout Reuse — Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth
- Do ImageNet Classifiers Generalize to ImageNet? — Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks — Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, David Lo
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingA regression set should remember failures without becoming the whole exam
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaMonitor the agent after deploymentUseful with this idea
Re-benchmark AI when the model or task changesUseful with this idea
Turn a verified AI failure into a regression testUse before this idea
Write the completion condition before the agent starts iteratingUseful with this idea
Evaluate the same agent task across multiple trials
Original edition evidence and sources
These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.
NIST AI RMF calls for documenting AI test sets, metrics and evaluation tools.
A documented evaluation set can still be unrepresentative or weak; provenance and coverage must be judged separately.
AI RMF Core · Measure 2.1
All sources (1)
Original edition, downloads and provenance
The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.
Version 0.1.0
No review details are supplied for this edition.
Read the original edition and overlay as JSON · Release details