Protocol
Turn subjective quality into evaluator criteria before looping on it
An evaluator cannot enforce taste that nobody has made legible.
Before you use it
- Use when
- An agent is told to make a design, report or interface 'better' and keeps polishing without a stable target.
Keep this limit in mind
- Some quality remains irreducibly judgmental; a rubric makes it discussable but does not turn taste into objective truth.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Before adding an evaluator loop, write a small rubric for the dimensions that matter, with observable examples or failure anchors. Separate hard requirements from preference. Let the evaluator point to specific violations and evidence, not emit one opaque score. Periodically compare evaluator judgments with human review so the loop does not optimize a distorted proxy.
Steps
- A reviewer can explain why the evaluator passed or failed the artifact without appealing to its confidence.
An example
For a dashboard, grade information hierarchy, task completion and responsive behavior separately instead of asking a judge model whether it 'looks professional.'
Check your result
A reviewer can explain why the evaluator passed or failed the artifact without appealing to its confidence.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingTurn subjective quality into evaluator criteria before looping on it
Evidence and sources
Anthropic's evaluator work operationalized subjective quality by writing concrete grading criteria before using an evaluator agent to drive iteration.
Rubrics encode judgment imperfectly and can be gamed; periodic human calibration remains important.
Harness design for long-running application development · Generator-evaluator loop and evaluator criteria
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.