Protocol

Combine grader types instead of asking one LLM judge to decide everything

A flexible judge is useful; a monoculture of judgment is fragile.

About 1 min read

Before you use it

Use when
  • An open-ended agent task is scored entirely by one model-based evaluator.

Keep this limit in mind

  • More graders can add cost and conflicting signals; each must serve a clear decision.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Use deterministic checks where the outcome is objective, model-based rubrics for nuanced qualities, and human review to calibrate subjective grading. Design each grader for a specific dimension rather than one omnibus 'good/bad' score. Recalibrate model graders when tasks or models change.

Steps

  1. Use code/tests for objective conditions.
  2. Use model rubrics for open-ended qualities.
  3. Use human examples to calibrate subjective graders.
  4. Keep grader dimensions separate enough to debug disagreement.

An example

A research agent can be graded by source existence checks, citation-groundedness rubric, coverage checks and expert spot review.

Check your result

No single probabilistic grader has unchecked authority over every success dimension.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingCombine grader types instead of asking one LLM judge to decide everything

Evidence and sources
Supports

Anthropic's 2026 agent-eval guidance recommends combining code-based, model-based and human graders according to task needs.

More graders can add cost and conflicting signals; each must serve a clear decision.

Demystifying evals for AI agents · See source record

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details