Protocol

Give computer-use tasks a start state and success check

A screenshot is not a test case until you know where the task starts and what state counts as success.

About 1 min read

Before you use it

Use when
  • A GUI or computer-use agent is evaluated on tasks whose initial desktop state or completion condition is ambiguous.

Keep this limit in mind

  • Clean start states can underrepresent messy production environments. Add controlled variants after the base task is reliable.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Specify the initial environment, relevant files or accounts, and a success predicate before the run. Reset to that state for comparable trials. Prefer checking the resulting system state over inferring success from the agent's final message.

Steps

  1. Define the starting desktop or application state.
  2. Seed the files, account data or document needed for the task.
  3. Define a success predicate on resulting state.
  4. Reset the environment before the next comparable trial.

An example

For a spreadsheet task, seed the same workbook and mark success by the requested values and formulas in the saved file, not by the agent saying 'done.'

Check your result

Another evaluator can reset the task and determine pass or fail from the resulting state without reading the agent's self-report.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingGive computer-use tasks a start state and success check

Evidence and sources
Supports

Computer-use evaluations are easier to interpret when each task defines a known initial environment and a verifiable success condition.

A clean initial state improves reproducibility but can be easier than messy production desktops; representative variants should be added after the basic task is stable.

Computer-Use 2.0: Agents Just Got Multi-Cursor · 6:34-9:34, give each GUI task an initial state and success check, then test how the computer tool changes the result

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details