# Read eval transcripts before trusting the score

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

---

## Read eval transcripts before trusting the score

ID: MHC-D-RESEARCH-0956 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/read-eval-transcripts-before-trusting-the-score

A score can fail because the agent failed—or because the eval did.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent's eval score moves and the team immediately attributes the change to model quality.

### Avoid when

- Transcript review is expensive; sample strategically while keeping automated coverage broad.

### Explanation

Sample transcripts from passes and failures. Check whether tasks were unambiguous, graders penalized valid solutions, tool constraints caused artificial failures, or the agent genuinely made mistakes. Update the eval when the failure is unfair; update the agent when the failure is real.

### Steps

1. Read multiple failed transcripts.
2. Read some passes for hidden shortcuts.
3. Compare grader decision with artifact reality.
4. Classify agent failure versus eval/harness failure.
5. Feed real failures back into the eval set.

### Example

A coding agent 'fails' because the grader assumed a filepath the task never specified; that is an eval bug, not an agent regression.

### Check

Score changes have a transcript-backed explanation before they drive model or workflow decisions.

### Limits

- Transcript review is expensive; sample strategically while keeping automated coverage broad.

### Evidence and sources

- supports: Anthropic explicitly recommends reading eval transcripts to distinguish genuine agent mistakes from unfair grading or harness problems. — RS-EE9379CE3D2ABBDD. Transcript review is expensive; sample strategically while keeping automated coverage broad. (See source record)
- RS-EE9379CE3D2ABBDD: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/do-not-assume-a-rubric-reduced-noise-measure-it
