Protocol
Mine traces for failure categories before changing prompts
A low score tells you that something is wrong. A failure taxonomy tells you where to work.
Before you use it
- Use when
- An agent is underperforming and the team is tempted to add more instructions without knowing what actually fails.
Keep this limit in mind
- Trace categories are hypotheses about mechanism. Confirm them with targeted tests before treating them as causes.
An idea to explore.
Try this idea.
Inspect the example, try the idea in your situation, then check your next move.
Why it matters
Sample failed and borderline traces, ask concrete diagnostic questions and group recurring causes: retrieval miss, bad tool choice, state loss, weak planning, invalid output, policy failure or model limitation. Change the component that matches the failure instead of accumulating prompt rules for every symptom.
Steps
- Sample failures and a few successful near-neighbours.
- Label the first material point where the run diverges.
- Group recurring failures by mechanism, not by wording.
- Choose an intervention for the largest or most costly supported category.
- Rerun the affected eval slice after the change.
An example
If most failed coding runs selected the wrong file, improve repository retrieval before adding another paragraph telling the model to 'be careful.'
Check your result
Every proposed intervention names the failure category and trace evidence it is supposed to change.
Save and practise this idea
My learning
Read. Recall. Repeat.
Save this idea and return to it on any device.
Keep exploring
Connected ideas
You are readingMine traces for failure categories before changing prompts
Evidence and sources
Production traces can be mined into recurring failure categories and reusable evidence before teams decide whether prompts, tools, state or the model should change.
Failure categories are observational; they help focus investigation but do not prove which intervention caused an improvement.
Improving Agents Is a Data Mining Problem · 0:00-17:06, execution evidence, trace mining, failure diagnosis and continual learning
All sources (1)
Edition and review
Version 0.1.0
No review details are supplied for this edition.