Protocol

Read eval transcripts before trusting the score

A score can fail because the agent failed—or because the eval did.

About 1 min read

Before you use it

Use when
  • An agent's eval score moves and the team immediately attributes the change to model quality.

Keep this limit in mind

  • Transcript review is expensive; sample strategically while keeping automated coverage broad.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Sample transcripts from passes and failures. Check whether tasks were unambiguous, graders penalized valid solutions, tool constraints caused artificial failures, or the agent genuinely made mistakes. Update the eval when the failure is unfair; update the agent when the failure is real.

Steps

  1. Read multiple failed transcripts.
  2. Read some passes for hidden shortcuts.
  3. Compare grader decision with artifact reality.
  4. Classify agent failure versus eval/harness failure.
  5. Feed real failures back into the eval set.

An example

A coding agent 'fails' because the grader assumed a filepath the task never specified; that is an eval bug, not an agent regression.

Check your result

Score changes have a transcript-backed explanation before they drive model or workflow decisions.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingRead eval transcripts before trusting the score

Evidence and sources
Supports

Anthropic explicitly recommends reading eval transcripts to distinguish genuine agent mistakes from unfair grading or harness problems.

Transcript review is expensive; sample strategically while keeping automated coverage broad.

Demystifying evals for AI agents · See source record

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details