Protocol
Give computer-use tasks a start state and success check
A screenshot is not a test case until you know where the task starts and what state counts as success.
When it fits
- A GUI or computer-use agent is evaluated on tasks whose initial desktop state or completion condition is ambiguous.
When to avoid it
- Clean start states can underrepresent messy production environments. Add controlled variants after the base task is reliable.
Why it matters
Specify the initial environment, relevant files or accounts, and a success predicate before the run. Reset to that state for comparable trials. Prefer checking the resulting system state over inferring success from the agent's final message.
Steps
- Define the starting desktop or application state.
- Seed the files, account data or document needed for the task.
- Define a success predicate on resulting state.
- Reset the environment before the next comparable trial.
An example
For a spreadsheet task, seed the same workbook and mark success by the requested values and formulas in the saved file, not by the agent saying 'done.'
Check your result
Another evaluator can reset the task and determine pass or fail from the resulting state without reading the agent's self-report.
Keep this limit in mind
- Clean start states can underrepresent messy production environments. Add controlled variants after the base task is reliable.
Connected ideas
Useful withRun AI evals from a clean starting state
Evidence and sources
Computer-use evaluations are easier to interpret when each task defines a known initial environment and a verifiable success condition.
A clean initial state improves reproducibility but can be easier than messy production desktops; representative variants should be added after the basic task is stable.
Computer-Use 2.0: Agents Just Got Multi-Cursor · 6:34-9:34, give each GUI task an initial state and success check, then test how the computer tool changes the result