Protocol
Read eval transcripts before trusting the score
A score can fail because the agent failed—or because the eval did.
Before you use it
- Use when
- An agent's eval score moves and the team immediately attributes the change to model quality.
Keep this limit in mind
- Transcript review is expensive; sample strategically while keeping automated coverage broad.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Sample transcripts from passes and failures. Check whether tasks were unambiguous, graders penalized valid solutions, tool constraints caused artificial failures, or the agent genuinely made mistakes. Update the eval when the failure is unfair; update the agent when the failure is real.
Steps
- Read multiple failed transcripts.
- Read some passes for hidden shortcuts.
- Compare grader decision with artifact reality.
- Classify agent failure versus eval/harness failure.
- Feed real failures back into the eval set.
An example
A coding agent 'fails' because the grader assumed a filepath the task never specified; that is an eval bug, not an agent regression.
Check your result
Score changes have a transcript-backed explanation before they drive model or workflow decisions.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingRead eval transcripts before trusting the score
Evidence and sources
Anthropic explicitly recommends reading eval transcripts to distinguish genuine agent mistakes from unfair grading or harness problems.
Transcript review is expensive; sample strategically while keeping automated coverage broad.
Demystifying evals for AI agents · See source record
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.