Protocol
Pin the browser environment when comparing agent behavior
If the browser moves under your feet, you do not know what the agent change caused.
Before you use it
- Use when
- Browser-agent benchmarks change between runs because the environment changes along with the model or harness.
Keep this limit in mind
- A perfectly fixed browser is not production. Use it for causal comparison, then test realistic variability.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
For controlled comparison, pin relevant browser version, locale, viewport, extensions, login state and seed data. Reset the environment between trials. After the comparison is stable, add representative environment variants to test robustness.
Steps
- Pin browser and relevant runtime versions.
- Control locale, viewport and login state.
- Reset task data between trials.
- Run environment variants after the controlled baseline.
An example
When comparing two browser agents on expense submission, keep the same browser version, account state and seeded expense before testing different models.
Check your result
The benchmark can distinguish agent changes from environment changes, and later robustness runs intentionally vary the environment.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
More from this collection
Operate AI agents as systems you can replay, verify and constrain
Exchange broad credentials for task-scoped accessSeparate user-visible UI state from model-visible contextEvidence and sources
Keeping the browser environment consistent improves the interpretability of comparisons between agent strategies by reducing unrelated execution variation.
A pinned environment can hide real-world variability. Use it for controlled comparison, then test against representative live conditions.
Bringing agents onto the world wide web · Sections on measuring the harness against a baseline, combining browser interaction with code and keeping the browser environment consistent
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.