Protocol

A regression set should remember failures without becoming the whole exam

A stable eval set is valuable because it makes regressions visible. The trap is using the same familiar cases as the only proof that a changing agent still works. Once those cases guide development, the team also needs evidence from work the system has not been tuned around.

About 4 min read

Before you use it

Use when
  • An AI workflow will be reused and quality cannot be judged from one polished demo.

Keep this limit in mind

  • Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.

Current editorial reading

Separate regression from generalization

Keep a stable core of representative tasks, edge cases and known failures. Its job is to answer a practical question: did this change break something we already know matters? NIST’s AI RMF supports documenting test sets, metrics and evaluation tools so results can be repeated and interpreted. But a regression suite and a fresh test answer different questions. The first protects known behavior; the second gives a better check on whether improvements extend beyond the cases the team has repeatedly seen.

Repeated reuse can turn the test into part of the development loop

This is not only a language-model problem. Work on adaptive data analysis shows that repeatedly using a holdout to choose what to try next can overfit the holdout itself. LLM evaluation adds another route: benchmark items may already appear in training data. A 2025 study of 83 software-engineering benchmarks found leakage was uneven rather than universal, but heavily leaked benchmarks existed and the authors reported that leakage affected measured model performance. A rising score therefore needs a provenance question as well as a scoring question.

Fresh cases help, but freshness is not a magic control

LiveBench responds to contamination risk by adding questions from recent sources and updating the benchmark over time. That is useful evidence for a rolling or held-out slice, not a reason to discard the stable regression set. Fresh data can also differ in difficulty or distribution. In the ImageNet replication study, accuracy fell on newly collected test sets, yet the authors concluded the drop was better explained by the new images being slightly harder than by adaptive overfitting. If a fresh slice scores worse, investigate both possibilities before declaring the model overfit.

Three questions before you trust the score

Should every production failure enter the eval? Add a failure when it represents a distinct, important risk or capability gap, but put it in the regression set; do not let all future measurement collapse into known cases. How large must the eval be? There is no universal number: coverage of real task types and uncertainty around the score matter more than a round target. Does 100% mean the agent is ready? Only for the defined cases, configuration and scoring rule. Keep a fresh or time-separated check when generalization matters, and record which set influenced development.

Take it with you

Use two complementary views: a stable regression set that remembers important failures, and a fresh or protected slice that the development loop has not repeatedly optimized against. Version both, keep scoring rules explicit, and treat a score as evidence about those cases—not a certificate for unseen work.

Sources and evidence (5)

What we checked

Supports

NIST AI RMF Measure 2.1 calls for documenting test sets, metrics and details of tools used during test, evaluation, validation and verification.

This supports versioned, repeatable evaluation records. NIST does not say that one finite set proves performance outside the tested conditions.

AI RMF Core · Measure 2.1

Limits

Adaptive reuse of a holdout can overfit the holdout when observed results influence later hypotheses or choices.

The paper is about adaptive data analysis rather than agent-eval operations, so the regression-versus-fresh-slice workflow here is an editorial application of the general reuse problem.

Generalization in Adaptive Data Analysis and Holdout Reuse · Abstract

Challenges

Fresh CIFAR-10 and ImageNet test sets produced lower accuracy than the original test sets, but the authors attributed the drop mainly to the new data being slightly harder rather than to adaptive test-set overfitting.

A worse fresh-set score is therefore not, by itself, proof that the development process overfit the old eval. Distribution and difficulty need checking too.

Do ImageNet Classifiers Generalize to ImageNet? · Abstract

Provides context

LiveBench uses frequently updated questions from recent sources and objective scoring to reduce test-set contamination and keep the benchmark useful as models change.

This is a benchmark-design example, not evidence that monthly refresh is the correct cadence for every internal agent eval.

LiveBench: A Challenging, Contamination-Limited LLM Benchmark · Abstract

Challenges

A 2025 analysis of 83 software-engineering benchmarks found benchmark leakage into StarCoder training data was uneven, with some benchmarks heavily affected, and reported that leakage materially affected evaluation results.

The study concerns one open-model training corpus and software-engineering benchmarks. Its leakage rates should not be generalized to unrelated models or private eval sets.

LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks · Abstract

Editorial sources (5)

Editorial version 1.0.0 · Evidence checked 2026-09-25

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingA regression set should remember failures without becoming the whole exam

Original edition evidence and sources

These records belong to the original released edition. The current editorial reading has its own evidence above; neither set silently replaces the other.

Supports

NIST AI RMF calls for documenting AI test sets, metrics and evaluation tools.

A documented evaluation set can still be unrepresentative or weak; provenance and coverage must be judged separately.

AI RMF Core · Measure 2.1

All sources (1)
Original edition, downloads and provenance

The reading above is editorial version 1.0.0, bound to this exact source edition. The original data and saved downloads retain that edition and the separate editorial layer. They may contain earlier wording; use the current reading above when interpreting it.

Version 0.1.0

No review details are supplied for this edition.

Read the original edition and overlay as JSON · Release details