Protocol
Run AI evals from a clean starting state
A hidden file from yesterday can make today's agent look brilliant.
Before you use it
- Use when
- You compare prompts, skills, agents or models across repeated tasks.
Keep this limit in mind
- If production intentionally carries memory across sessions, evaluate that stateful workflow separately rather than pretending every run is cold.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Reset the workspace, conversation and generated artifacts to the intended baseline before each comparable trial. Seed only the state the real workflow is supposed to have. Record versions of model, tools, prompt or skill and important dependencies. This prevents leftover outputs, caches or prior messages from leaking answers into the next run.
Steps
- Workspace matches the intended starting state.
- Prior generated artifacts are removed unless production would keep them.
- Conversation history is controlled.
- Model, harness and skill versions are recorded.
- Random or external dependencies are noted where relevant.
An example
When testing a research skill, delete the previous report and memory files unless persistent memory is explicitly part of the product being evaluated.
Check your result
A trial can be reproduced without depending on residue from a previous run.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingRun AI evals from a clean starting state
From this idea
Use this idea beforeAblate a skill before you trust it
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaGive computer-use tasks a start state and success check
Evidence and sources
AI Engineer skill-eval guidance recommends clean workspaces and repeated trials so hidden artifacts or prior messages do not contaminate comparisons.
Perfect isolation may be unrealistic for production workflows; the eval should represent intended starting state.
Don't Ship Skills Without Evals · Test cleanly and repeat
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.