Protocol
Combine grader types instead of asking one LLM judge to decide everything
A flexible judge is useful; a monoculture of judgment is fragile.
When it fits
- An open-ended agent task is scored entirely by one model-based evaluator.
When to avoid it
- More graders can add cost and conflicting signals; each must serve a clear decision.
Why it matters
Use deterministic checks where the outcome is objective, model-based rubrics for nuanced qualities, and human review to calibrate subjective grading. Design each grader for a specific dimension rather than one omnibus 'good/bad' score. Recalibrate model graders when tasks or models change.
Steps
- Use code/tests for objective conditions.
- Use model rubrics for open-ended qualities.
- Use human examples to calibrate subjective graders.
- Keep grader dimensions separate enough to debug disagreement.
An example
A research agent can be graded by source existence checks, citation-groundedness rubric, coverage checks and expert spot review.
Check your result
No single probabilistic grader has unchecked authority over every success dimension.
Keep this limit in mind
- More graders can add cost and conflicting signals; each must serve a clear decision.
Connected ideas
Useful withMeasure agreement on the criteria, not only the verdict
Evidence and sources
Supports
Anthropic's 2026 agent-eval guidance recommends combining code-based, model-based and human graders according to task needs.
More graders can add cost and conflicting signals; each must serve a clear decision.
Demystifying evals for AI agents · See source record