Protocol

Run AI evals from a clean starting state

A hidden file from yesterday can make today's agent look brilliant.

About 1 min read

Before you use it

Use when
  • You compare prompts, skills, agents or models across repeated tasks.

Keep this limit in mind

  • If production intentionally carries memory across sessions, evaluate that stateful workflow separately rather than pretending every run is cold.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Reset the workspace, conversation and generated artifacts to the intended baseline before each comparable trial. Seed only the state the real workflow is supposed to have. Record versions of model, tools, prompt or skill and important dependencies. This prevents leftover outputs, caches or prior messages from leaking answers into the next run.

Steps

  1. Workspace matches the intended starting state.
  2. Prior generated artifacts are removed unless production would keep them.
  3. Conversation history is controlled.
  4. Model, harness and skill versions are recorded.
  5. Random or external dependencies are noted where relevant.

An example

When testing a research skill, delete the previous report and memory files unless persistent memory is explicitly part of the product being evaluated.

Check your result

A trial can be reproduced without depending on residue from a previous run.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingRun AI evals from a clean starting state

Evidence and sources
Supports

AI Engineer skill-eval guidance recommends clean workspaces and repeated trials so hidden artifacts or prior messages do not contaminate comparisons.

Perfect isolation may be unrealistic for production workflows; the eval should represent intended starting state.

Don't Ship Skills Without Evals · Test cleanly and repeat

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details