# Make AI work improve from failures instead of accumulating rituals

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Use eval-driven development, durable guardrails, clean experiments and reusable research or automation artifacts so everyday AI workflows learn from repeated work rather than restarting from chat every time.
Collection kind: collection
Source: https://vedokrok.com/collections/make-ai-work-improve-from-failures-instead-of-accumulating-rituals
Version: 0.1.0

---

## Turn a verified AI failure into a regression test

ID: MHC-D-RESEARCH-0718 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/turn-a-verified-ai-failure-into-a-regression-test

A painful failure should pay rent the second time.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You discover an AI output or agent trajectory that failed in a way you care about.

### Avoid when

- Known-failure tests do not cover novel failures; keep adding fresh tasks and production monitoring.

### Explanation

After fixing the immediate problem, preserve a minimal representative task that reproduces the failure and a grader that detects it. Add it to the relevant eval suite before changing prompts, skills or models again. Tag the failure class so later regressions can be diagnosed rather than hidden inside one aggregate score.

### Steps

1. The old failure would be caught automatically if it returned tomorrow.

### Example

If an agent silently edits an unrelated configuration file, add a task whose grader verifies both requested behavior and untouched-file constraints.

### Check

The old failure would be caught automatically if it returned tomorrow.

### Limits

- Known-failure tests do not cover novel failures; keep adding fresh tasks and production monitoring.

### Evidence and sources

- supports: Anthropic recommends sourcing eval tasks from real failures and keeping regression suites so fixes can be tested against previously observed problems. — RS-ED3CA2F877BE88CA. A regression suite covers known failures and must be supplemented with new, harder and distribution-shifted tasks. (Collect tasks and maintain eval suites)
- RS-ED3CA2F877BE88CA: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/keep-an-eval-set-that-can-embarrass-the-agent

---

## Turn repeated human corrections into guardrails with wider reach

ID: MHC-D-RESEARCH-0719 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/turn-repeated-human-corrections-into-guardrails-with-wider-reach

If humans repeat the same comment every week, the feedback is trapped at the wrong layer.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Reviewers keep making the same correction to AI-generated work.

### Avoid when

- Do not institutionalize a reviewer preference until the team agrees it is a real requirement.

### Explanation

Classify the correction. If it is deterministic, encode it as a lint rule, test, schema or script. If it is a stable convention, put it in discoverable project documentation or a skill. If it needs judgment, create a focused review rubric or reviewer agent and keep human escalation. Aim to move recurring feedback from one artifact to the system that produces many artifacts.

### Example

Instead of repeatedly telling an agent not to modify generated files, enforce the rule in repository instructions and CI.

### Check

The next similar task receives the correction before or during generation, not only after a human spots it again.

### Limits

- Do not institutionalize a reviewer preference until the team agrees it is a real requirement.

### Evidence and sources

- supports: AI Engineer harness guidance recommends turning repeated human feedback into durable documentation, lint rules, structural tests or specialized review agents. — RS-3DB1894302E64FC3. Automated guardrails can encode a bad preference; keep ownership and review for consequential rules. (Make quality legible and enforceable)
- RS-3DB1894302E64FC3: Harness Engineering: How to Build Software When Humans Steer, Agents Execute — https://ai.engineer/talks/am_oeAoUhew-harness-engineering

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/turn-a-verified-ai-failure-into-a-regression-test

---

## Own the task eval before shopping for a better model

ID: MHC-D-RESEARCH-0720 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/own-the-task-eval-before-shopping-for-a-better-model

Without your own test, a model leaderboard is somebody else's job description.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A team is comparing models because an AI workflow feels unreliable or slow.

### Avoid when

- Small suites can overfit current work; refresh them as the task distribution changes.

### Explanation

Build a small representative task suite with the quality, latency and cost measures that matter to your workflow. Run candidate models through the same harness and starting state. Use public benchmarks as context, not as a substitute for local evidence. Keep the suite so future model releases can be tested quickly instead of restarting the comparison from opinion.

### Example

For SAP incident analysis, compare models on anonymized diagnostic cases with required evidence and false-cause penalties rather than generic coding scores.

### Check

A model choice can be explained from task-level evidence relevant to your workflow.

### Limits

- Small suites can overfit current work; refresh them as the task distribution changes.

### Evidence and sources

- supports: Anthropic argues that task evals let teams compare model or system changes against stable requirements instead of relying on impressions. — RS-ED3CA2F877BE88CA. A stable eval can become stale or saturated and requires maintenance. (Evals for model upgrades and baselines)
- RS-ED3CA2F877BE88CA: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/re-benchmark-ai-when-the-model-or-task-changes

---

## Grade the final state and the agent trajectory separately

ID: MHC-D-RESEARCH-0721 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/grade-the-final-state-and-the-agent-trajectory-separately

The destination can be correct while the route is unacceptable.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent can reach the right answer through risky, wasteful or policy-breaking intermediate actions.

### Avoid when

- Overly prescriptive trajectory graders can punish creative valid solutions; inspect the path only where the path itself matters.

### Explanation

Use outcome graders for the resulting files, data or decision, and separate trajectory checks for properties that matter during execution: unauthorized tool calls, destructive retries, excessive loops, fabricated sources or other process constraints. Do not grade every stylistic path when only the end state matters; trajectory rules should protect real risks or costs.

### Steps

1. A run can pass the result while still failing a meaningful process constraint, and the reports distinguish those failures.

### Example

A deployment agent must produce the correct configuration and must not bypass the approval gate on the way there.

### Check

A run can pass the result while still failing a meaningful process constraint, and the reports distinguish those failures.

### Limits

- Overly prescriptive trajectory graders can punish creative valid solutions; inspect the path only where the path itself matters.

### Evidence and sources

- supports: Agent evals may grade both end-state outcomes and transcript or trajectory properties because a correct-looking final artifact can hide undesirable process behavior. — RS-ED3CA2F877BE88CA. Do not overconstrain legitimate alternative paths when only the end state matters. (Graders can evaluate outcome or transcript)
- RS-ED3CA2F877BE88CA: Demystifying evals for AI agents — https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/monitor-the-whole-agent-trajectory-not-only-individual-allowed-actions

---

## Run AI evals from a clean starting state

ID: MHC-D-RESEARCH-0722 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/run-ai-evals-from-a-clean-starting-state

A hidden file from yesterday can make today's agent look brilliant.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You compare prompts, skills, agents or models across repeated tasks.

### Avoid when

- If production intentionally carries memory across sessions, evaluate that stateful workflow separately rather than pretending every run is cold.

### Explanation

Reset the workspace, conversation and generated artifacts to the intended baseline before each comparable trial. Seed only the state the real workflow is supposed to have. Record versions of model, tools, prompt or skill and important dependencies. This prevents leftover outputs, caches or prior messages from leaking answers into the next run.

### Steps

1. Workspace matches the intended starting state.
2. Prior generated artifacts are removed unless production would keep them.
3. Conversation history is controlled.
4. Model, harness and skill versions are recorded.
5. Random or external dependencies are noted where relevant.

### Example

When testing a research skill, delete the previous report and memory files unless persistent memory is explicitly part of the product being evaluated.

### Check

A trial can be reproduced without depending on residue from a previous run.

### Limits

- If production intentionally carries memory across sessions, evaluate that stateful workflow separately rather than pretending every run is cold.

### Evidence and sources

- supports: AI Engineer skill-eval guidance recommends clean workspaces and repeated trials so hidden artifacts or prior messages do not contaminate comparisons. — RS-FD96DA85C7B10F5A. Perfect isolation may be unrealistic for production workflows; the eval should represent intended starting state. (Test cleanly and repeat)
- RS-FD96DA85C7B10F5A: Don't Ship Skills Without Evals — https://ai.engineer/talks/0vphxNt4wyk-dont-ship-skills-without-evals

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/ablate-a-skill-before-you-trust-it

---

## Use versioned artifacts as the meeting place for humans and agents

ID: MHC-D-RESEARCH-0723 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/use-versioned-artifacts-as-the-meeting-place-for-humans-and-agents

Shared chat is a poor substitute for shared state.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Humans and several agents need to coordinate work without staying in one synchronous conversation.

### Avoid when

- Not every task belongs in Git; choose an equivalent versioned or auditable artifact for documents, data or operations.

### Explanation

Coordinate through artifacts that can be inspected and versioned: pull requests, issues, decision records, test results or change sets. Let agents post work where humans and other agents can review it asynchronously. Keep the artifact self-contained enough that a reviewer does not need the originating chat to understand the change, rationale and checks.

### Example

An agent opens a pull request with the diff, test evidence and remaining risk; a second agent or human reviews that artifact rather than joining the first agent's context.

### Check

A new reviewer can take useful action from the shared artifact alone.

### Limits

- Not every task belongs in Git; choose an equivalent versioned or auditable artifact for documents, data or operations.

### Evidence and sources

- supports: AI Engineer harness guidance presents pull requests as an asynchronous shared interface where human and agent contributors can review and coordinate without sharing one live conversation. — RS-3DB1894302E64FC3. This pattern fits version-controlled work; other domains may need a different shared artifact. (GitHub pull requests as broadcast domain)
- RS-3DB1894302E64FC3: Harness Engineering: How to Build Software When Humans Steer, Agents Execute — https://ai.engineer/talks/am_oeAoUhew-harness-engineering

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/let-subagents-write-durable-artifacts-instead-of-relaying-everything-through-the-coordinator

---

## Turn stable repeated tool sequences into scripts

ID: MHC-D-RESEARCH-0724 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/turn-stable-repeated-tool-sequences-into-scripts

Do not pay a language model to rediscover a deterministic macro on every run.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI agent repeatedly performs the same browser, CLI or data-manipulation sequence with little judgment.

### Avoid when

- Do not automate an unstable process merely because it is repetitive; first understand authority, error handling and rollback.

### Explanation

Once a repeated sequence is understood and the inputs and failures are known, encode the deterministic portion as a script or task-shaped tool. Let the model decide when to use it and interpret exceptions, while the script handles the repeatable mechanics. Add logging and idempotence or dry-run behavior when repeated execution could create side effects.

### Example

Instead of having an agent manually transform the same export columns every week, generate a tested transformation script and let the agent handle only exceptions and interpretation.

### Check

The repeated sequence becomes faster, cheaper and more reproducible without hiding important decisions.

### Limits

- Do not automate an unstable process merely because it is repetitive; first understand authority, error handling and rollback.

### Evidence and sources

- supports: Current agent-engineering practice recommends moving repeatable deterministic operations into scripts or tools while reserving model judgment for ambiguous decisions. — RS-0B5DE7867BDB8E97. Do not automate a process before its inputs, errors and authority boundaries are understood. (Assign determinism, judgment and authority; reusable scripts)
- RS-0B5DE7867BDB8E97: Build Systems, Not Code — https://ai.engineer/talks/ZD9-4fW2HhM-build-systems-not-code

### Vedokrok editorial review

Script repeatable work without making retries blind

Move stable, mechanical steps into code once their inputs and failure cases are understood. Keep judgment with the model or a person, and design recovery before letting the script repeat real actions.

#### Extract the predictable part

Start with one recurring operation, such as normalizing an export. Define accepted inputs, expected output and errors that need attention. Test the script on representative cases, record failures and compare its results with the previous workflow. Automation has earned its place only if correctness holds and the reduced effort justifies maintaining it.

#### Separate preview from repeat safety

A dry run performs only its documented checks; it does not protect a later real call from duplicate effects. For writes, use the destination’s idempotency contract where available: retries of one operation retain the same identity and parameters. Without that protection, if a call times out and its result is unknown, reconcile what happened before repeating it. A new request identifier may mean a new operation.

**Editorial takeaway:** Test an interrupted run in a controlled environment, including a successful write whose response is lost. Check the final state for duplicates. Stop automatic retries when the result cannot be resolved safely; a preview or log alone is not that guarantee.

#### Editorial evidence

- supports: The talk assigns predictable calculations to ordinary code and describes system-enforced recovery that skips already completed actions. — RS-0B5DE7867BDB8E97. Practitioner design guidance, not controlled evidence that scripts always cost less or perform better. The export example and proposed checks are independently designed. (Transcript 9:26–10:44 and 13:01–15:18)
- supports: Idempotent API contracts can recognize retries through a stable caller request identifier, while changed intent or parameters require different handling. — aws-idempotent-retries-20261004. Specific API contracts determine scope and retention. A local log entry by itself does not establish atomic duplicate suppression at the destination. (Reducing client complexity; Same client request ID, different intent)
- limits: After a timeout, a write may already have succeeded; repeating it without a valid duplicate-handling contract can create another effect. — aws-idempotent-retries-20261004. Selected official passages read. Reconciliation can remain uncertain, so the stopping rule is conservative editorial guidance rather than an exactly-once guarantee. (Retrying and side effects; Late arriving requests)
- challenges: EC2 documents DryRun as a permission check without execution, separately from ClientToken for request idempotency. — aws-ec2-runinstances-20261004. This is a concrete counterexample to treating dry-run support as interchangeable with repeat safety. Other dry-run implementations may check different things; their own contracts must be read. (Request Parameters: DryRun; ClientToken)
- RS-0B5DE7867BDB8E97: Build Systems, Not Code — Angie Jones, timestamped talk transcript — https://ai.engineer/talks/ZD9-4fW2HhM-build-systems-not-code
- aws-idempotent-retries-20261004: Making retries safe with idempotent APIs — https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
- aws-ec2-runinstances-20261004: RunInstances - Amazon Elastic Compute Cloud — https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/prefer-a-task-shaped-tool-to-an-open-ended-one

---

## Build a reusable research wiki when the topic keeps coming back

ID: MHC-D-RESEARCH-0725 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/build-a-reusable-research-wiki-when-the-topic-keeps-coming-back

Deep research is expensive if every question starts by forgetting the last one.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- You repeatedly research the same domain, product, client or technical area across projects.

### Avoid when

- A one-off topic may not justify maintenance overhead; persistent memory becomes useful only when reuse exceeds upkeep.

### Explanation

Create a maintained project knowledge layer with compact findings, source links, terminology, unresolved questions and dated decisions. Let new research update that layer rather than producing another isolated report. Start small with trusted seed sources, preserve raw references, and retire stale claims when the underlying source or product changes.

### Steps

1. Seed the project with a few trusted sources.
2. Write compact source-linked findings.
3. Record open questions and decisions.
4. Reuse the wiki as input to later research.
5. Review freshness and supersede stale entries.

### Example

For an SAP program, maintain a source-linked wiki of interface behavior, known notes, mappings and prior incident findings that the AI can search before starting another investigation.

### Check

A later question reuses relevant prior evidence and can still reach the underlying sources.

### Limits

- A one-off topic may not justify maintenance overhead; persistent memory becomes useful only when reuse exceeds upkeep.

### Evidence and sources

- supports: The AI Engineer research-memory workflow turns repeated research into a maintained project wiki so later work can reuse sources and findings rather than restart from zero. — RS-30040385BE4C7BEB. Persistent research only pays when the topic recurs and somebody maintains freshness and provenance. (Reusable research workflow and growing project wiki)
- RS-30040385BE4C7BEB: Turn 10,994 Notes Into Your Agents' Memory — https://ai.engineer/talks/ZRM_TfEZcIo-turn-10-994-notes-into-your-agents

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/keep-raw-evidence-behind-ai-memory-summaries
