Protocol
Combine grader types instead of asking one LLM judge to decide everything
A flexible judge is useful; a monoculture of judgment is fragile.
Before you use it
- Use when
- An open-ended agent task is scored entirely by one model-based evaluator.
Keep this limit in mind
- More graders can add cost and conflicting signals; each must serve a clear decision.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Use deterministic checks where the outcome is objective, model-based rubrics for nuanced qualities, and human review to calibrate subjective grading. Design each grader for a specific dimension rather than one omnibus 'good/bad' score. Recalibrate model graders when tasks or models change.
Steps
- Use code/tests for objective conditions.
- Use model rubrics for open-ended qualities.
- Use human examples to calibrate subjective graders.
- Keep grader dimensions separate enough to debug disagreement.
An example
A research agent can be graded by source existence checks, citation-groundedness rubric, coverage checks and expert spot review.
Check your result
No single probabilistic grader has unchecked authority over every success dimension.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingCombine grader types instead of asking one LLM judge to decide everything
Ideas that link here
These ideas link to this one. Check their conditions and limits before use.
Useful with this ideaGive verification an independent evidence path
Evidence and sources
Anthropic's 2026 agent-eval guidance recommends combining code-based, model-based and human graders according to task needs.
More graders can add cost and conflicting signals; each must serve a clear decision.
Demystifying evals for AI agents · See source record
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.