Protocol
Run AI evals from a clean starting state
A hidden file from yesterday can make today's agent look brilliant.
When it fits
- You compare prompts, skills, agents or models across repeated tasks.
When to avoid it
- If production intentionally carries memory across sessions, evaluate that stateful workflow separately rather than pretending every run is cold.
Why it matters
Reset the workspace, conversation and generated artifacts to the intended baseline before each comparable trial. Seed only the state the real workflow is supposed to have. Record versions of model, tools, prompt or skill and important dependencies. This prevents leftover outputs, caches or prior messages from leaking answers into the next run.
Steps
- Workspace matches the intended starting state.
- Prior generated artifacts are removed unless production would keep them.
- Conversation history is controlled.
- Model, harness and skill versions are recorded.
- Random or external dependencies are noted where relevant.
An example
When testing a research skill, delete the previous report and memory files unless persistent memory is explicitly part of the product being evaluated.
Check your result
A trial can be reproduced without depending on residue from a previous run.
Keep this limit in mind
- If production intentionally carries memory across sessions, evaluate that stateful workflow separately rather than pretending every run is cold.
Connected ideas
Use beforeAblate a skill before you trust it
Evidence and sources
AI Engineer skill-eval guidance recommends clean workspaces and repeated trials so hidden artifacts or prior messages do not contaminate comparisons.
Perfect isolation may be unrealistic for production workflows; the eval should represent intended starting state.
Don't Ship Skills Without Evals · Test cleanly and repeat