Protocol
Read eval transcripts before trusting the score
A score can fail because the agent failed—or because the eval did.
When it fits
- An agent's eval score moves and the team immediately attributes the change to model quality.
When to avoid it
- Transcript review is expensive; sample strategically while keeping automated coverage broad.
Why it matters
Sample transcripts from passes and failures. Check whether tasks were unambiguous, graders penalized valid solutions, tool constraints caused artificial failures, or the agent genuinely made mistakes. Update the eval when the failure is unfair; update the agent when the failure is real.
Steps
- Read multiple failed transcripts.
- Read some passes for hidden shortcuts.
- Compare grader decision with artifact reality.
- Classify agent failure versus eval/harness failure.
- Feed real failures back into the eval set.
An example
A coding agent 'fails' because the grader assumed a filepath the task never specified; that is an eval bug, not an agent regression.
Check your result
Score changes have a transcript-backed explanation before they drive model or workflow decisions.
Keep this limit in mind
- Transcript review is expensive; sample strategically while keeping automated coverage broad.
Connected ideas
Useful withDo not assume a rubric reduced noise—measure it
Evidence and sources
Supports
Anthropic explicitly recommends reading eval transcripts to distinguish genuine agent mistakes from unfair grading or harness problems.
Transcript review is expensive; sample strategically while keeping automated coverage broad.
Demystifying evals for AI agents · See source record