Protocol

Try to break the evaluator before trusting it

If the agent can game the judge, a higher score can mean a worse system.

About 1 min read

Before you use it

Use when
  • An automated evaluator will decide whether agent behavior passes, fails or improves.

Keep this limit in mind

  • Passing adversarial checks does not prove the evaluator is complete. New shortcuts can appear as the agent changes.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Design counterexamples that should clearly fail and see whether the evaluator catches them. Test shortcuts, superficial completion signals and plausible-but-wrong outputs. Repair the evaluator before optimizing the agent against its score.

Steps

  1. Create a clear true-pass case and a clear true-fail case.
  2. Add a shortcut that looks successful but violates the task.
  3. Add a plausible output with the wrong underlying state.
  4. Inspect false passes before using the evaluator for model or prompt selection.

An example

A desktop agent's evaluator checks that a file exists. Add a case where the file exists with the wrong contents; if it passes, the evaluator is too shallow.

Check your result

The evaluator rejects the intentionally broken cases that matter to the task before its score is used as an optimization target.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

More from this collection

Operate AI agents as systems you can replay, verify and constrain

Simulate users who behave like usersPut uncertainty around consequential eval differences
Evidence and sources
Supports

A task evaluator should itself be challenged with cases designed to expose false passes before its score is used as a trusted optimization target.

Adversarial evaluator tests improve confidence only for the tested failure modes; evaluator drift and unanticipated shortcuts still require monitoring.

Computer-Use 2.0: Agents Just Got Multi-Cursor · 9:04-10:50, tool changes the result and try to break the evaluator before trusting its score

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details