Protocol

Turn a production finding into a reusable eval artifact

A failure is expensive twice if the system can forget it.

About 2 min read

Before you use it

Use when
  • A production incident teaches the team something, but the lesson is likely to disappear after the immediate fix.

Keep this limit in mind

  • Do not convert every noisy observation into permanent rules. Validate the finding and avoid leaking production secrets or personal data into repositories or eval sets.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

Why it matters

Preserve a validated production finding in the form that best prevents recurrence: an issue with evidence, a regression example, an evaluator case or a dataset item. Keep the link to the original trace and the execution boundary where the failure mattered. Then rerun the relevant eval after the fix.

Steps

  1. Validate the finding against the original production evidence.
  2. Minimize sensitive data before storing a durable artifact.
  3. Choose the artifact that can catch the failure again.
  4. Link it to the affected component and original evidence.
  5. Rerun the relevant evaluation after the repair.

An example

A coding agent repeatedly edits generated files instead of their source templates. Preserve one verified incident as a regression case and repository instruction test before merging the fix.

Check your result

The next version is tested against an artifact derived from the real failure, and the source evidence remains traceable.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingTurn a production finding into a reusable eval artifact

Evidence and sources
Supports

A production investigation can preserve evidence by turning a validated finding into a reviewable issue, evaluator or dataset example before the repair is automated.

Runtime evidence can contain noise, secrets or personal data; promotion into a durable artifact requires review, minimization and a clear link to the original failure.

From Signal to PR: Anatomy of a Self-Improving Agent · 3:31-18:04, gather evidence, bring it into the repository, create an issue/evaluator/dataset example and preserve the execution boundary

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details