Protocol

Pin the browser environment when comparing agent behavior

If the browser moves under your feet, you do not know what the agent change caused.

About 1 min read

Before you use it

Use when
  • Browser-agent benchmarks change between runs because the environment changes along with the model or harness.

Keep this limit in mind

  • A perfectly fixed browser is not production. Use it for causal comparison, then test realistic variability.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

For controlled comparison, pin relevant browser version, locale, viewport, extensions, login state and seed data. Reset the environment between trials. After the comparison is stable, add representative environment variants to test robustness.

Steps

  1. Pin browser and relevant runtime versions.
  2. Control locale, viewport and login state.
  3. Reset task data between trials.
  4. Run environment variants after the controlled baseline.

An example

When comparing two browser agents on expense submission, keep the same browser version, account state and seeded expense before testing different models.

Check your result

The benchmark can distinguish agent changes from environment changes, and later robustness runs intentionally vary the environment.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

More from this collection

Operate AI agents as systems you can replay, verify and constrain

Exchange broad credentials for task-scoped accessSeparate user-visible UI state from model-visible context
Evidence and sources
Supports

Keeping the browser environment consistent improves the interpretability of comparisons between agent strategies by reducing unrelated execution variation.

A pinned environment can hide real-world variability. Use it for controlled comparison, then test against representative live conditions.

Bringing agents onto the world wide web · Sections on measuring the harness against a baseline, combining browser interaction with code and keeping the browser environment consistent

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details