Turn a production finding into a reusable eval artifact

When it fits

• A production incident teaches the team something, but the lesson is likely to disappear after the immediate fix.

When to avoid it

• Do not convert every noisy observation into permanent rules. Validate the finding and avoid leaking production secrets or personal data into repositories or eval sets.

Why it matters

Preserve a validated production finding in the form that best prevents recurrence: an issue with evidence, a regression example, an evaluator case or a dataset item. Keep the link to the original trace and the execution boundary where the failure mattered. Then rerun the relevant eval after the fix.

Steps

1. Validate the finding against the original production evidence.
2. Minimize sensitive data before storing a durable artifact.
3. Choose the artifact that can catch the failure again.
4. Link it to the affected component and original evidence.
5. Rerun the relevant evaluation after the repair.

Check your result

The next version is tested against an artifact derived from the real failure, and the source evidence remains traceable.

Keep this limit in mind

• Do not convert every noisy observation into permanent rules. Validate the finding and avoid leaking production secrets or personal data into repositories or eval sets.

Connected ideas