Protocol
Pin the browser environment when comparing agent behavior
If the browser moves under your feet, you do not know what the agent change caused.
When it fits
- Browser-agent benchmarks change between runs because the environment changes along with the model or harness.
When to avoid it
- A perfectly fixed browser is not production. Use it for causal comparison, then test realistic variability.
Why it matters
For controlled comparison, pin relevant browser version, locale, viewport, extensions, login state and seed data. Reset the environment between trials. After the comparison is stable, add representative environment variants to test robustness.
Steps
- Pin browser and relevant runtime versions.
- Control locale, viewport and login state.
- Reset task data between trials.
- Run environment variants after the controlled baseline.
An example
When comparing two browser agents on expense submission, keep the same browser version, account state and seeded expense before testing different models.
Check your result
The benchmark can distinguish agent changes from environment changes, and later robustness runs intentionally vary the environment.
Keep this limit in mind
- A perfectly fixed browser is not production. Use it for causal comparison, then test realistic variability.
Evidence and sources
Keeping the browser environment consistent improves the interpretability of comparisons between agent strategies by reducing unrelated execution variation.
A pinned environment can hide real-world variability. Use it for controlled comparison, then test against representative live conditions.
Bringing agents onto the world wide web · Sections on measuring the harness against a baseline, combining browser interaction with code and keeping the browser environment consistent