# Turn a verified AI failure into a regression test

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

---

## Turn a verified AI failure into a regression test

ID: MHC-D-RESEARCH-0718 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/turn-a-verified-ai-failure-into-a-regression-test

A painful failure should pay rent the second time.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You discover an AI output or agent trajectory that failed in a way you care about.

### Avoid when

- Known-failure tests do not cover novel failures; keep adding fresh tasks and production monitoring.

### Explanation

After fixing the immediate problem, preserve a minimal representative task that reproduces the failure and a grader that detects it. Add it to the relevant eval suite before changing prompts, skills or models again. Tag the failure class so later regressions can be diagnosed rather than hidden inside one aggregate score.

### Steps

1. The old failure would be caught automatically if it returned tomorrow.

### Example

If an agent silently edits an unrelated configuration file, add a task whose grader verifies both requested behavior and untouched-file constraints.

### Check

The old failure would be caught automatically if it returned tomorrow.

### Limits

- Known-failure tests do not cover novel failures; keep adding fresh tasks and production monitoring.

### Evidence and sources

- supports: Anthropic recommends sourcing eval tasks from real failures and keeping regression suites so fixes can be tested against previously observed problems. — RS-ED3CA2F877BE88CA. A regression suite covers known failures and must be supplemented with new, harder and distribution-shifted tasks. (Collect tasks and maintain eval suites)
- RS-ED3CA2F877BE88CA: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/keep-an-eval-set-that-can-embarrass-the-agent
