Protocol
Try to break the evaluator before trusting it
If the agent can game the judge, a higher score can mean a worse system.
Before you use it
- Use when
- An automated evaluator will decide whether agent behavior passes, fails or improves.
Keep this limit in mind
- Passing adversarial checks does not prove the evaluator is complete. New shortcuts can appear as the agent changes.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Design counterexamples that should clearly fail and see whether the evaluator catches them. Test shortcuts, superficial completion signals and plausible-but-wrong outputs. Repair the evaluator before optimizing the agent against its score.
Steps
- Create a clear true-pass case and a clear true-fail case.
- Add a shortcut that looks successful but violates the task.
- Add a plausible output with the wrong underlying state.
- Inspect false passes before using the evaluator for model or prompt selection.
An example
A desktop agent's evaluator checks that a file exists. Add a case where the file exists with the wrong contents; if it passes, the evaluator is too shallow.
Check your result
The evaluator rejects the intentionally broken cases that matter to the task before its score is used as an optimization target.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
More from this collection
Operate AI agents as systems you can replay, verify and constrain
Simulate users who behave like usersPut uncertainty around consequential eval differencesEvidence and sources
A task evaluator should itself be challenged with cases designed to expose false passes before its score is used as a trusted optimization target.
Adversarial evaluator tests improve confidence only for the tested failure modes; evaluator drift and unanticipated shortcuts still require monitoring.
Computer-Use 2.0: Agents Just Got Multi-Cursor · 9:04-10:50, tool changes the result and try to break the evaluator before trusting its score
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.