Protocol
Give computer-use tasks a start state and success check
A screenshot is not a test case until you know where the task starts and what state counts as success.
Before you use it
- Use when
- A GUI or computer-use agent is evaluated on tasks whose initial desktop state or completion condition is ambiguous.
Keep this limit in mind
- Clean start states can underrepresent messy production environments. Add controlled variants after the base task is reliable.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Specify the initial environment, relevant files or accounts, and a success predicate before the run. Reset to that state for comparable trials. Prefer checking the resulting system state over inferring success from the agent's final message.
Steps
- Define the starting desktop or application state.
- Seed the files, account data or document needed for the task.
- Define a success predicate on resulting state.
- Reset the environment before the next comparable trial.
An example
For a spreadsheet task, seed the same workbook and mark success by the requested values and formulas in the saved file, not by the agent saying 'done.'
Check your result
Another evaluator can reset the task and determine pass or fail from the resulting state without reading the agent's self-report.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingGive computer-use tasks a start state and success check
Evidence and sources
Computer-use evaluations are easier to interpret when each task defines a known initial environment and a verifiable success condition.
A clean initial state improves reproducibility but can be easier than messy production desktops; representative variants should be added after the basic task is stable.
Computer-Use 2.0: Agents Just Got Multi-Cursor · 6:34-9:34, give each GUI task an initial state and success check, then test how the computer tool changes the result
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.