Protocol
Try to break the evaluator before trusting it
If the agent can game the judge, a higher score can mean a worse system.
When it fits
- An automated evaluator will decide whether agent behavior passes, fails or improves.
When to avoid it
- Passing adversarial checks does not prove the evaluator is complete. New shortcuts can appear as the agent changes.
Why it matters
Design counterexamples that should clearly fail and see whether the evaluator catches them. Test shortcuts, superficial completion signals and plausible-but-wrong outputs. Repair the evaluator before optimizing the agent against its score.
Steps
- Create a clear true-pass case and a clear true-fail case.
- Add a shortcut that looks successful but violates the task.
- Add a plausible output with the wrong underlying state.
- Inspect false passes before using the evaluator for model or prompt selection.
An example
A desktop agent's evaluator checks that a file exists. Add a case where the file exists with the wrong contents; if it passes, the evaluator is too shallow.
Check your result
The evaluator rejects the intentionally broken cases that matter to the task before its score is used as an optimization target.
Keep this limit in mind
- Passing adversarial checks does not prove the evaluator is complete. New shortcuts can appear as the agent changes.
Evidence and sources
A task evaluator should itself be challenged with cases designed to expose false passes before its score is used as a trusted optimization target.
Adversarial evaluator tests improve confidence only for the tested failure modes; evaluator drift and unanticipated shortcuts still require monitoring.
Computer-Use 2.0: Agents Just Got Multi-Cursor · 9:04-10:50, tool changes the result and try to break the evaluator before trusting its score