Protocol
Mine traces for failure categories before changing prompts
A low score tells you that something is wrong. A failure taxonomy tells you where to work.
When it fits
- An agent is underperforming and the team is tempted to add more instructions without knowing what actually fails.
When to avoid it
- Trace categories are hypotheses about mechanism. Confirm them with targeted tests before treating them as causes.
Why it matters
Sample failed and borderline traces, ask concrete diagnostic questions and group recurring causes: retrieval miss, bad tool choice, state loss, weak planning, invalid output, policy failure or model limitation. Change the component that matches the failure instead of accumulating prompt rules for every symptom.
Steps
- Sample failures and a few successful near-neighbours.
- Label the first material point where the run diverges.
- Group recurring failures by mechanism, not by wording.
- Choose an intervention for the largest or most costly supported category.
- Rerun the affected eval slice after the change.
An example
If most failed coding runs selected the wrong file, improve repository retrieval before adding another paragraph telling the model to 'be careful.'
Check your result
Every proposed intervention names the failure category and trace evidence it is supposed to change.
Keep this limit in mind
- Trace categories are hypotheses about mechanism. Confirm them with targeted tests before treating them as causes.
Connected ideas
Useful withTurn a verified AI failure into a regression test
Evidence and sources
Production traces can be mined into recurring failure categories and reusable evidence before teams decide whether prompts, tools, state or the model should change.
Failure categories are observational; they help focus investigation but do not prove which intervention caused an improvement.
Improving Agents Is a Data Mining Problem · 0:00-17:06, execution evidence, trace mining, failure diagnosis and continual learning