Protocol

Mine traces for failure categories before changing prompts

A low score tells you that something is wrong. A failure taxonomy tells you where to work.

About 1 min read

Before you use it

Use when
  • An agent is underperforming and the team is tempted to add more instructions without knowing what actually fails.

Keep this limit in mind

  • Trace categories are hypotheses about mechanism. Confirm them with targeted tests before treating them as causes.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Sample failed and borderline traces, ask concrete diagnostic questions and group recurring causes: retrieval miss, bad tool choice, state loss, weak planning, invalid output, policy failure or model limitation. Change the component that matches the failure instead of accumulating prompt rules for every symptom.

Steps

  1. Sample failures and a few successful near-neighbours.
  2. Label the first material point where the run diverges.
  3. Group recurring failures by mechanism, not by wording.
  4. Choose an intervention for the largest or most costly supported category.
  5. Rerun the affected eval slice after the change.

An example

If most failed coding runs selected the wrong file, improve repository retrieval before adding another paragraph telling the model to 'be careful.'

Check your result

Every proposed intervention names the failure category and trace evidence it is supposed to change.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingMine traces for failure categories before changing prompts

Evidence and sources
Supports

Production traces can be mined into recurring failure categories and reusable evidence before teams decide whether prompts, tools, state or the model should change.

Failure categories are observational; they help focus investigation but do not prove which intervention caused an improvement.

Improving Agents Is a Data Mining Problem · 0:00-17:06, execution evidence, trace mining, failure diagnosis and continual learning

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details