# Operate AI agents as systems you can replay, verify and constrain

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Add practical 2026 agent-engineering techniques for replay, evaluation, tool routing, computer use, scoped authorization and deterministic execution without turning conference claims into universal guarantees.
Collection kind: collection
Source: https://vedokrok.com/collections/operate-ai-agents-as-systems-you-can-replay-verify-and-constrain
Version: 0.1.0

---

## Replay from the failure checkpoint, not from memory

ID: MHC-D-RESEARCH-1158 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/replay-from-the-failure-checkpoint-not-from-memory

A reproducible failure is more useful than a fresh attempt that happens to succeed.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent failed late in a long run and a full rerun is expensive, slow or unlikely to recreate the same decision.

### Avoid when

- A checkpoint is only as good as the state it captures. External systems, time and hidden side effects can still prevent an exact reproduction.

### Explanation

Checkpoint meaningful execution state before expensive or consequential boundaries. When a failure appears, restore the nearest useful checkpoint and change one factor: the model, a tool response, a policy or a piece of state. Compare the resulting decision with the original path instead of asking the system to recreate the whole past from scratch.

### Steps

1. Choose a checkpoint before the decision you want to inspect.
2. Restore the captured model, tool and task state.
3. Change one intervention while holding the rest as stable as practical.
4. Compare both the next decision and the final outcome.

### Example

A support agent issued the wrong refund after seven tool calls. Replay from the checkpoint before the refund decision with the repaired policy tool rather than rerunning the entire conversation.

### Check

You can name what was held constant, what changed and whether the decision or outcome changed after replay.

### Limits

- A checkpoint is only as good as the state it captures. External systems, time and hidden side effects can still prevent an exact reproduction.

### Evidence and sources

- supports: A checkpointed agent run can be replayed from an intermediate state while changing one model or tool decision, which can make interventions easier to compare than full reruns. — RS-EA671DF45F465038. Replay quality depends on what state the checkpoint actually captures; hidden external state can still make the reproduced execution differ from production. (2:03-10:54, production checkpoints, change one execution part and apply an intervention across a cohort)
- RS-EA671DF45F465038: Your Agents Need a Save Button — https://ai.engineer/talks/bZISsg7H7DA-your-agents-need-save-button

### Vedokrok editorial review

Replay the decision without repeating the damage

A saved checkpoint is a place to restart an experiment. It is not permission to repeat the original payment, email or database write.

#### Define what runs again

Choose a checkpoint before the failed decision. Record the one change you want to test and which earlier results will be reused. Before starting, identify every later tool that can change an external system. Replace those effects with recorded responses or an isolated test service unless a live action is separately authorized.

#### Compare decisions in a safe test

For an agent that closed the wrong support ticket, restore the relevant state and change the routing rule. Capture the proposed closure in a test system rather than closing another live ticket. Compare the decision and final test state with the original. Missing files, changed services or unrecorded state can still break the comparison.

**Editorial takeaway:** Before replay, list the calls that will run live and show where their effects will go. If you cannot account for a consequential write, stop and isolate it first.

#### Editorial evidence

- supports: The conference demonstration reuses saved checkpoints before a changed step, then runs the remaining execution under the intervention. — RS-EA671DF45F465038. Selected timestamped transcript read on 2026-10-04, especially 5:03–7:35. This supports the demonstrated comparison pattern, not a universal checkpoint contract or a verified deployment. (Transcript 5:03–7:35; chapter Change one part of the execution)
- supports: Kitaru documentation describes replaying tool calls from recorded responses rather than repeating their real effects. — kitaru-readme. Selected README passage read on 2026-10-04. This is the current product description, not a test of the older conference version or this repository. (README, Why, bullet Replay re-executes your code)
- limits: The documented protection depends on substituting recorded tool responses; it does not establish that an arbitrary replay harness isolates every external write. — kitaru-readme. The call inventory and stop rule are editorial safety checks derived from the documented mechanism. Neither runtime behavior nor external services were executed in this review. (README, Why; Self-hosted, by design)
- RS-EA671DF45F465038: Your Agents Need a Save Button — https://ai.engineer/talks/bZISsg7H7DA-your-agents-need-save-button
- kitaru-readme: Kitaru: replay-based evaluation — https://github.com/zenml-io/kitaru

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/checkpoint-meaningful-progress-before-a-long-agent-crosses-a-fragile-boundary

---

## Capture semantic boundaries, not only raw logs

ID: MHC-D-RESEARCH-1159 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/capture-semantic-boundaries-not-only-raw-logs

A log can be complete and still omit the boundary where meaning changed.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A production agent failure disappears when developers try to reproduce it locally.

### Avoid when

- Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads.

### Explanation

Record the inputs and outputs at the model and tool boundaries that matter to the task: model request and response identifiers, parsed arguments, tool results, policy decisions and the state carried into the next step. This creates an execution envelope that can replay the bad decision against repaired code.

### Checklist

- Capture the model decision that selected the action.
- Capture the exact parsed tool arguments and returned result.
- Record policy or validation decisions around the call.
- Keep enough state to replay the boundary without storing unrelated sensitive data.

### Example

A trading assistant interpreted dollars as shares. The useful trace preserves the model's action, parsed amount, tool contract and validation result at the order boundary.

### Check

A developer can replay the consequential boundary without inventing missing inputs or re-prompting the model for a similar decision.

### Limits

- Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads.

### Evidence and sources

- supports: Recording semantic model and tool boundaries can preserve the decision context needed to replay a production failure against repaired enforcement code. — RS-F0FB27D3E1B6C4A1. Capturing more data is not automatically useful or safe; retain only the boundaries needed for diagnosis while respecting privacy and secret-handling requirements. (4:51-13:14, recover the run, record semantic boundaries, replay the bad decision and retain the execution envelope)
- RS-F0FB27D3E1B6C4A1: Your Agent Failed in Prod. Good Luck Reproducing It. — https://ai.engineer/talks/Lc8zRh9muoY-your-agent-failed-in-prod-good-luck

### Vedokrok editorial review

Record the boundary you need to retest

Saving a tool result is different from saving the world the tool changed. Keep enough evidence to replay the decision, then control what the test executes.

#### Keep the consequential inputs

At a model or tool boundary, preserve the arguments, returned value, relevant validation result and code or model version. Use the recorded model output when testing repaired enforcement code; asking the model again creates a new trial. Minimize secrets and personal data, and record any missing state that limits the test.

#### Separate recorded responses from live code

A replay stub returns a saved result without running that boundary. A boundary selected to run live executes its real code, including side effects. For a report-export agent, replay the original export request against the repaired destination check, connected to a test-only storage service. Assert that an unauthorized destination is rejected and that no report copy is created.

**Editorial takeaway:** Name the boundary under test, the recorded responses around it and the isolated target for its effects. A passing guard test does not show that the model will choose correctly on a new task.

#### Editorial evidence

- supports: The speakers demonstrate recording boundary inputs and outputs, stubbing a model decision and checking repaired tool enforcement. — RS-F0FB27D3E1B6C4A1. Selected timestamped transcript read on 2026-10-04: 6:14–7:51 and 10:18–12:53. The demonstration is practitioner evidence, not an independently verified production incident or efficacy study. (Transcript 6:14–7:51; 10:18–12:53)
- supports: Chronicle records boundary input/output and metadata; a live cut-point runs real code, while a stub does not repeat the boundary side effects. — chronicle-readme. Selected current README sections read on 2026-10-04. This documents the product contract; no library was installed or run. (README, Key terms; How it works; FAQ on side effects during replay)
- limits: A boundary record does not capture its internal file writes, network calls, database writes or time, so it is not an external-system snapshot. — chronicle-readme. This limits the original reproduction promise. The report-export scenario is an independently chosen editorial test design, not a source example, Chronicle benchmark or observed result. (README, How it works, Envelope black-box description; FAQ on live cut-points)
- RS-F0FB27D3E1B6C4A1: Your Agent Failed in Prod. Good Luck Reproducing It. — https://ai.engineer/talks/Lc8zRh9muoY-your-agent-failed-in-prod-good-luck
- chronicle-readme: Chronicle: boundary recording and cut-point replay — https://github.com/theagentplane/chronicle

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/track-tool-errors-tool-calls-runtime-and-tokens-as-agent-diagnostics

---

## Test an intervention across a replay cohort

ID: MHC-D-RESEARCH-1160 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/test-an-intervention-across-a-replay-cohort

One rescued trace is a story. A cohort begins to look like evidence.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- One replay looks better after a model, prompt, tool or routing change and you need to know whether the improvement is stable enough to matter.

### Avoid when

- A cohort is not the future. Rare events and distribution shifts may be absent, so production monitoring still matters.

### Explanation

Collect a bounded set of representative checkpoints from real runs. Apply the same intervention to each, then compare task outcomes, decisions, cost and failure types. This reveals whether a local fix generalizes across the cases that motivated it.

### Steps

1. Define the production slice the cohort should represent.
2. Replay each checkpoint with the same intervention.
3. Measure outcomes and important failure categories, not only trace similarity.
4. Review cases that improve, regress or remain unchanged before release.

### Example

Before moving support triage to a cheaper model, replay a sample of real triage checkpoints and compare correct routing, escalation and cost across the cohort.

### Check

The release decision cites cohort-level outcomes and notable regressions rather than one favorable replay.

### Limits

- A cohort is not the future. Rare events and distribution shifts may be absent, so production monitoring still matters.

### Evidence and sources

- supports: Comparing an intervention across multiple checkpointed runs can reveal outcome changes that a single replay would not represent reliably. — RS-EA671DF45F465038. A replay cohort only represents the production slice it was sampled from; rare cases and future distribution shifts can remain uncovered. (10:54-15:10, apply the intervention across a cohort and use the cohort to change the release decision)
- RS-EA671DF45F465038: Your Agents Need a Save Button — https://ai.engineer/talks/bZISsg7H7DA-your-agents-need-save-button

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/evaluate-the-same-agent-task-across-multiple-trials

---

## Treat done as an evidence object

ID: MHC-D-RESEARCH-1161 · Version: 0.1.0 · Kind: template
Source: https://vedokrok.com/knowledge/treat-done-as-an-evidence-object

Done is not a green check mark. It is a set of claims someone must be able to inspect.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent can mark work complete faster than a reviewer can tell whether it is actually ready for the next stage.

### Avoid when

- Do not add fields nobody uses. The object is valuable only when its evidence and ownership change real workflow decisions.

### Explanation

Represent completion with the artifact, scope, acceptance rubric, evidence, verifier, approval authority, residual risk and next owner. Different levels of done can then mean produced, reviewed, approved, merged or deployed without collapsing them into one status.

### Template

Artifact: {artifact}
Scope: {scope}
Rubric: {rubric}
Evidence: {evidence}
Verified by: {verifier}
Approved by: {authority}
Residual risk: {risk}
Next action / owner: {next}

### Example

A coding agent finishes a migration. The object says tests passed, the schema diff was reviewed, rollback remains untested, and the release owner is still a human maintainer.

### Check

A person can distinguish what exists, what was verified, who may approve it and what still happens next.

### Limits

- Do not add fields nobody uses. The object is valuable only when its evidence and ownership change real workflow decisions.

### Evidence and sources

- supports: Agent completion can be represented as structured claims about artifact, scope, evidence, rubric, verification, approval, residual risk and next ownership rather than one Boolean status. — RS-3BF9DF61A375DF34. The exact fields should match the workflow; adding completion metadata without enforceable review or ownership can become administrative noise. (0:50-6:40, completion contents and levels, done as an object, independent verifier and explicit next owner)
- RS-3BF9DF61A375DF34: What Does Done Even Mean? Agents and Paperclip's Liveness Model — https://ai.engineer/talks/7P0elyLIxXo-what-does-done-even-mean-agents-paperclips

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/write-the-completion-condition-before-the-agent-starts-iterating

---

## Give verification an independent evidence path

ID: MHC-D-RESEARCH-1162 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/give-verification-an-independent-evidence-path

A second opinion is weak when it only rereads the first opinion.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- The same agent that produced an artifact is also asked whether its own work is correct.

### Avoid when

- A separate model is not automatically independent. Shared data, prompts, tools or blind spots can still produce correlated errors.

### Explanation

Let the verifier inspect the artifact and evidence directly: run tests, query the system, open the source, inspect screenshots or recompute a result. Separating author and verifier can reduce shared incentives, but the larger gain comes from giving verification an evidence path that does not depend on the author's summary.

### Example

A code agent says a page works. The verifier opens the deployed preview, runs the relevant tests and checks the requested behavior rather than asking the code agent to self-critique its explanation.

### Check

At least one important acceptance claim is checked against independent artifact or system evidence.

### Limits

- A separate model is not automatically independent. Shared data, prompts, tools or blind spots can still produce correlated errors.

### Evidence and sources

- supports: Verification is stronger when the verifier can inspect evidence with tools and is not limited to accepting the generator's own explanation of success. — RS-3BF9DF61A375DF34. A separate verifier can share the same blind spots or bad evidence; independence is a design aid, not a proof of correctness. (5:56-6:40, separate verifier from author, provide verification tools and make the next owner explicit)
- RS-3BF9DF61A375DF34: What Does Done Even Mean? Agents and Paperclip's Liveness Model — https://ai.engineer/talks/7P0elyLIxXo-what-does-done-even-mean-agents-paperclips

### Vedokrok editorial review

Give the verifier evidence it can check

A second agent agreeing with the first is still only agreement. Ask for a check against the artifact or system state that the task was meant to change.

#### Check the claim directly

Choose one important acceptance claim and the evidence that could disprove it. Give the verifier access to that evidence: a saved file, test result, source record or live preview, as appropriate. Keep the producer’s explanation as context, not as the only basis for a pass.

#### Keep independence in proportion

Suppose an agent says a report includes every overdue invoice. A verifier compares the report with the authorized source data and records any missing or wrongly included rows. Using another model may add a different perspective, but shared data, tools or assumptions can preserve the same error. The cited talk recommends separation and verification tools; it does not measure which contributes the larger gain.

**Editorial takeaway:** Record the claim, the evidence inspected and the result. If both agents only read the same success summary, the acceptance claim remains unchecked.

#### Editorial evidence

- supports: The speaker recommends separating the verifier from the author and giving it tools to inspect the produced work. — RS-3BF9DF61A375DF34. Selected timestamped transcript read on 2026-10-04. The relevant passage is a practitioner recommendation; no Paperclip implementation or measurable reliability improvement was verified. (Transcript 5:56–6:40)
- limits: The inspected recommendation does not compare the measured gains from using a different model and from providing direct evidence access. — RS-3BF9DF61A375DF34. The original phrase larger gain was unsupported by this cited passage and has been removed. This is a bounded source check, not a claim that no comparative research exists. (Transcript 5:56–6:40; completion standards at 1:14–1:34)
- RS-3BF9DF61A375DF34: What Does Done Even Mean? Agents and Paperclip's Liveness Model — https://ai.engineer/talks/7P0elyLIxXo-what-does-done-even-mean-agents-paperclips

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/combine-grader-types-instead-of-asking-one-llm-judge-to-decide-everything

---

## Try to break the evaluator before trusting it

ID: MHC-D-RESEARCH-1163 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/try-to-break-the-evaluator-before-trusting-it

If the agent can game the judge, a higher score can mean a worse system.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An automated evaluator will decide whether agent behavior passes, fails or improves.

### Avoid when

- Passing adversarial checks does not prove the evaluator is complete. New shortcuts can appear as the agent changes.

### Explanation

Design counterexamples that should clearly fail and see whether the evaluator catches them. Test shortcuts, superficial completion signals and plausible-but-wrong outputs. Repair the evaluator before optimizing the agent against its score.

### Steps

1. Create a clear true-pass case and a clear true-fail case.
2. Add a shortcut that looks successful but violates the task.
3. Add a plausible output with the wrong underlying state.
4. Inspect false passes before using the evaluator for model or prompt selection.

### Example

A desktop agent's evaluator checks that a file exists. Add a case where the file exists with the wrong contents; if it passes, the evaluator is too shallow.

### Check

The evaluator rejects the intentionally broken cases that matter to the task before its score is used as an optimization target.

### Limits

- Passing adversarial checks does not prove the evaluator is complete. New shortcuts can appear as the agent changes.

### Evidence and sources

- supports: A task evaluator should itself be challenged with cases designed to expose false passes before its score is used as a trusted optimization target. — RS-273F6FA5C48C6C0B. Adversarial evaluator tests improve confidence only for the tested failure modes; evaluator drift and unanticipated shortcuts still require monitoring. (9:04-10:50, tool changes the result and try to break the evaluator before trusting its score)
- RS-273F6FA5C48C6C0B: Computer-Use 2.0: Agents Just Got Multi-Cursor — https://ai.engineer/talks/ZSQb5fzRFPw-computer-use-2-0-agents-just-got

No review details supplied.

---

## Simulate users who behave like users

ID: MHC-D-RESEARCH-1164 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/simulate-users-who-behave-like-users

A simulator that helps the agent too much is grading a different job.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A multi-turn agent looks excellent in synthetic conversations but real users still produce surprising failures.

### Avoid when

- Synthetic users remain approximations. Do not replace production monitoring or real-user research with simulation alone.

### Explanation

Model the target user's information, patience, ambiguity, mistakes and willingness to cooperate. Do not let the simulator volunteer missing details merely because an assistant model knows they would help. Compare simulated failures with production traces and refine the user model when the two diverge.

### Example

A support simulator should sometimes say 'it still doesn't work' instead of naming the exact error code the agent needs to solve the case.

### Check

The simulator can reproduce at least some real user friction without turning into a hidden co-pilot for the agent.

### Limits

- Synthetic users remain approximations. Do not replace production monitoring or real-user research with simulation alone.

### Evidence and sources

- supports: Multi-turn agent evaluation can become unrealistically easy when the simulated user cooperates like a helpful assistant instead of behaving like the target user population. — RS-49EB9A55618CBC03. No simulator perfectly reproduces real users; production evidence remains necessary to check whether simulated behavior covers the failure modes that matter. (7:20-17:37, simulate the whole interaction and avoid an unrealistically helpful user simulator)
- RS-49EB9A55618CBC03: Build Evals That Actually Matter — https://ai.engineer/talks/3z2uT5aDx_Y-build-evals-that-actually-matter

No review details supplied.

---

## Put uncertainty around consequential eval differences

ID: MHC-D-RESEARCH-1165 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/put-uncertainty-around-consequential-eval-differences

A two-point score gap is not a verdict until you know how noisy the estimate is.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Two agent versions have close evaluation scores and the result may change a launch, model-routing or safety decision.

### Avoid when

- Statistical uncertainty does not repair biased sampling, weak labels or an irrelevant metric.

### Explanation

Report sample size and an appropriate uncertainty estimate around important metrics. Pair the aggregate with error counts and failure categories. A small apparent improvement should not drive a consequential release if the evaluation cannot distinguish it from sampling variation or grader noise.

### Example

Two support agents score 91% and 93% on a small eval. Before switching, inspect confidence around the difference and whether severe escalation failures changed.

### Check

The release note reports uncertainty and meaningful error changes, not only a single point estimate.

### Limits

- Statistical uncertainty does not repair biased sampling, weak labels or an irrelevant metric.

### Evidence and sources

- supports: Consequential evaluation metrics should be reported with uncertainty so small observed differences are not treated as certain product improvements. — RS-49EB9A55618CBC03. An uncertainty interval does not repair biased samples, weak labels or the wrong metric; it quantifies uncertainty conditional on the evaluation design. (21:09-26:46, treat the judge as a classifier and put uncertainty around consequential numbers)
- RS-49EB9A55618CBC03: Build Evals That Actually Matter — https://ai.engineer/talks/3z2uT5aDx_Y-build-evals-that-actually-matter

No review details supplied.

---

## Mine traces for failure categories before changing prompts

ID: MHC-D-RESEARCH-1166 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/mine-traces-for-failure-categories-before-changing-prompts

A low score tells you that something is wrong. A failure taxonomy tells you where to work.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent is underperforming and the team is tempted to add more instructions without knowing what actually fails.

### Avoid when

- Trace categories are hypotheses about mechanism. Confirm them with targeted tests before treating them as causes.

### Explanation

Sample failed and borderline traces, ask concrete diagnostic questions and group recurring causes: retrieval miss, bad tool choice, state loss, weak planning, invalid output, policy failure or model limitation. Change the component that matches the failure instead of accumulating prompt rules for every symptom.

### Steps

1. Sample failures and a few successful near-neighbours.
2. Label the first material point where the run diverges.
3. Group recurring failures by mechanism, not by wording.
4. Choose an intervention for the largest or most costly supported category.
5. Rerun the affected eval slice after the change.

### Example

If most failed coding runs selected the wrong file, improve repository retrieval before adding another paragraph telling the model to 'be careful.'

### Check

Every proposed intervention names the failure category and trace evidence it is supposed to change.

### Limits

- Trace categories are hypotheses about mechanism. Confirm them with targeted tests before treating them as causes.

### Evidence and sources

- supports: Production traces can be mined into recurring failure categories and reusable evidence before teams decide whether prompts, tools, state or the model should change. — RS-AE31C3BE0BBBA1F6. Failure categories are observational; they help focus investigation but do not prove which intervention caused an improvement. (0:00-17:06, execution evidence, trace mining, failure diagnosis and continual learning)
- RS-AE31C3BE0BBBA1F6: Improving Agents Is a Data Mining Problem — https://ai.engineer/talks/CvRngaQZQ3Y-improving-agents-is-data-mining-problem

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/turn-a-verified-ai-failure-into-a-regression-test

---

## Route to the cheapest model that still passes your task eval

ID: MHC-D-RESEARCH-1167 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/route-to-the-cheapest-model-that-still-passes-your-task-eval

Do not pay for intelligence the task does not use—but do not guess where the boundary is.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A workflow uses a frontier model for every request even though many requests may be easier.

### Avoid when

- A cheaper request can still create a more expensive service if retries, failures or human recovery increase.

### Explanation

Partition requests by observable task features, evaluate candidate models on each slice and route only where the cheaper model continues to meet the required outcome and safety thresholds. Re-test the boundary after model, prompt, tool or traffic changes.

### Steps

1. Define task slices that can be identified before inference.
2. Run the same acceptance eval for candidate models on each slice.
3. Choose the least expensive model that passes the slice's required thresholds.
4. Keep a fallback for uncertain or failing cases.
5. Re-evaluate routing when the system or traffic changes.

### Example

Use a smaller model for straightforward classification only after local cases show it preserves routing accuracy and escalation behavior; send ambiguous cases to the stronger model.

### Check

Every routing rule points to a current local eval slice and a fallback path rather than to model reputation.

### Limits

- A cheaper request can still create a more expensive service if retries, failures or human recovery increase.

### Evidence and sources

- supports: Model routing can use local task evaluations to identify the least expensive model that still meets the required behavior for a bounded class of requests. — RS-AE31C3BE0BBBA1F6. The passing boundary can change with prompts, tools, model versions and request distribution, so routing rules require re-evaluation. (7:36-12:46, find the least expensive capable model and fit model, harness and task together)
- RS-AE31C3BE0BBBA1F6: Improving Agents Is a Data Mining Problem — https://ai.engineer/talks/CvRngaQZQ3Y-improving-agents-is-data-mining-problem

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/own-the-task-eval-before-shopping-for-a-better-model

---

## Retrieve tool schemas just in time

ID: MHC-D-RESEARCH-1168 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/retrieve-tool-schemas-just-in-time

A tool can exist without occupying attention on every turn.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent has access to a large tool catalog and every request carries all tool descriptions into the model context.

### Avoid when

- A router can hide the right tool. Optimize for end-task quality and safe fallback, not minimum schema count.

### Explanation

Index tool descriptions and retrieve a small candidate set from the current intent. Present only those schemas as callable tools, while preserving a fallback when retrieval confidence is weak. Evaluate routing recall, final task success, latency and context cost together.

### Steps

1. Index tools by the jobs and entities they operate on.
2. Retrieve a bounded candidate set from the current request.
3. Expose only those schemas to the model for the turn.
4. Fall back to broader discovery when the router is uncertain.
5. Measure missed-tool failures as well as cost and latency.

### Example

A finance question retrieves account and transaction tools instead of loading schemas for HR, calendar, CRM and deployment tools into the same request.

### Check

The system can report tool-retrieval recall and task success, not only the tokens saved by showing fewer schemas.

### Limits

- A router can hide the right tool. Optimize for end-task quality and safe fallback, not minimum schema count.

### Evidence and sources

- supports: Large tool catalogs can be routed semantically so only a request-relevant subset of tool schemas is placed in the model's active tool context. — RS-ED013467FB90D730. Routing can hide the correct tool. Recall and fallback behavior must be evaluated alongside token cost, latency and tool-selection accuracy. (9:19-24:27, retrieve tools, load schemas just in time, evaluate top-K and recover from misses)
- RS-ED013467FB90D730: The 100-Tool Agent Is a Trap: Scaling with Semantic Routers and JIT Context — https://ai.engineer/talks/vh2VGuQ3zhY-100-tool-agent-is-trap-scaling-semantic

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/use-progressive-disclosure-for-agent-instructions

---

## Measure the active tool set, not the catalog size

ID: MHC-D-RESEARCH-1169 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/measure-the-active-tool-set-not-the-catalog-size

The catalog is inventory. The working set is cognitive load.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A team argues about how many tools an agent can support but does not know how much tool context each request actually sees.

### Avoid when

- Do not optimize the working set in isolation. Extra routing steps or low recall can erase the savings.

### Explanation

Track the count and token footprint of tool schemas active per request, alongside tool-selection accuracy and task success. A large catalog can be acceptable if the working set remains relevant; a small catalog can still be wasteful if every verbose schema is always loaded.

### Example

Two agents both expose 80 available tools. One routinely shows six relevant schemas; the other shows all 80. Catalog size alone hides the meaningful difference.

### Check

Tool-context changes are evaluated with working-set size and task outcomes rather than only total tools registered.

### Limits

- Do not optimize the working set in isolation. Extra routing steps or low recall can erase the savings.

### Evidence and sources

- supports: The active number and size of tool schemas presented to the model can be measured separately from the total catalog to diagnose context and selection overhead. — RS-ED013467FB90D730. A smaller working set is not automatically better; over-pruning can reduce task coverage or force extra routing turns. (12:47-20:22, measure the working set and connect retrieval to evaluation and maintenance)
- RS-ED013467FB90D730: The 100-Tool Agent Is a Trap: Scaling with Semantic Routers and JIT Context — https://ai.engineer/talks/vh2VGuQ3zhY-100-tool-agent-is-trap-scaling-semantic

No review details supplied.

---

## Retrieve code before sending code

ID: MHC-D-RESEARCH-1170 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/retrieve-code-before-sending-code

The cheapest token is the irrelevant file you never send.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A coding assistant repeatedly receives large repository dumps or oversized file context to answer narrow questions.

### Avoid when

- Repository structure matters. Retrieval that works on one codebase can miss generated code, dynamic links or cross-cutting configuration elsewhere.

### Explanation

Build a local or privacy-appropriate code retrieval layer that combines complementary search signals, rejects weak matches and supplies only likely relevant fragments. Measure recall on known tasks before relying on the savings; a tiny context that omits the dependency is not efficient.

### Steps

1. Create a benchmark of code questions with known relevant files or symbols.
2. Combine lexical and semantic search when they cover different misses.
3. Reject low-relevance candidates instead of filling the context quota.
4. Measure retrieval recall, task success and total context cost together.

### Example

For a failing API endpoint, retrieve the route, service and referenced model instead of uploading the whole repository tree to every coding turn.

### Check

Known-task benchmarks show that the reduced context still retrieves the dependencies needed for correct answers.

### Limits

- Repository structure matters. Retrieval that works on one codebase can miss generated code, dynamic links or cross-cutting configuration elsewhere.

### Evidence and sources

- supports: A local code-retrieval layer can select relevant repository fragments before model inference instead of repeatedly sending broad code context. — RS-E45EC996E3B1A3AA. Retrieval can miss dependencies or return plausible irrelevant code. The reported headline token reduction is case-specific and is not a universal expectation. (Architecture sections on local retrieval, complementary searches, relevance rejection and benchmark comparison)
- RS-E45EC996E3B1A3AA: We Cut 94% of Our AI Coding Tokens With a Local Code Index. Here's the Architecture. — https://ai.engineer/talks/dRmWYHuIJxM-we-cut-94-our-ai-coding-tokens

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/treat-context-as-a-budget-not-a-transcript

---

## Turn repeated browser work into persistent programs

ID: MHC-D-RESEARCH-1171 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/turn-repeated-browser-work-into-persistent-programs

Use the browser to learn the path; use code to stop relearning it every time.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A browser agent repeatedly rediscovers the same navigation and action sequence on a site.

### Avoid when

- Do not freeze brittle selectors or bypass site rules. Persistent automation still needs authorization, monitoring and recovery.

### Explanation

After a browser interaction pattern becomes stable, encode the repeatable mechanics in a script or reusable skill. Let the live browser remain the state and feedback surface for checking whether the program still matches reality. Fall back to exploratory interaction when the site changes.

### Example

An agent first learns how to export a monthly report through the UI; after repeated success, a reusable automation handles the known sequence while the browser verifies the resulting state.

### Check

Repeated work uses a versioned, testable action path and still detects when the live site no longer matches it.

### Limits

- Do not freeze brittle selectors or bypass site rules. Persistent automation still needs authorization, monitoring and recovery.

### Evidence and sources

- supports: Repeated browser interactions can be moved into persistent programs or reusable website skills while live browser state remains the feedback surface. — RS-05CEF906EF829016. Persistent automation becomes stale when websites or permissions change; it still needs error handling, observation and a fallback to fresh discovery. (8:12, turn browser actions into persistent programs, within a harness that continues until a verifiable goal is reached)
- RS-05CEF906EF829016: Codex, Behind the Harness — https://ai.engineer/talks/shRR1e2HXMk-codex-behind-harness

No review details supplied.

---

## Pin the browser environment when comparing agent behavior

ID: MHC-D-RESEARCH-1172 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/pin-the-browser-environment-when-comparing-agent-behavior

If the browser moves under your feet, you do not know what the agent change caused.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Browser-agent benchmarks change between runs because the environment changes along with the model or harness.

### Avoid when

- A perfectly fixed browser is not production. Use it for causal comparison, then test realistic variability.

### Explanation

For controlled comparison, pin relevant browser version, locale, viewport, extensions, login state and seed data. Reset the environment between trials. After the comparison is stable, add representative environment variants to test robustness.

### Steps

1. Pin browser and relevant runtime versions.
2. Control locale, viewport and login state.
3. Reset task data between trials.
4. Run environment variants after the controlled baseline.

### Example

When comparing two browser agents on expense submission, keep the same browser version, account state and seeded expense before testing different models.

### Check

The benchmark can distinguish agent changes from environment changes, and later robustness runs intentionally vary the environment.

### Limits

- A perfectly fixed browser is not production. Use it for causal comparison, then test realistic variability.

### Evidence and sources

- supports: Keeping the browser environment consistent improves the interpretability of comparisons between agent strategies by reducing unrelated execution variation. — RS-6442420642D13047. A pinned environment can hide real-world variability. Use it for controlled comparison, then test against representative live conditions. (Sections on measuring the harness against a baseline, combining browser interaction with code and keeping the browser environment consistent)
- RS-6442420642D13047: Bringing agents onto the world wide web — https://ai.engineer/talks/GqoNrUz8hEU-bringing-agents-onto-world-wide-web

No review details supplied.

---

## Exchange broad credentials for task-scoped access

ID: MHC-D-RESEARCH-1173 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/exchange-broad-credentials-for-task-scoped-access

Do not hand an agent the master key when it needs one door for one job.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent currently receives a long-lived API key with more authority than the current task requires.

### Avoid when

- Scoped tokens are not a substitute for authorization design, audit or human approval where the impact requires it.

### Explanation

At the tool boundary, exchange delegated identity for a short-lived credential scoped to the resource and action the task actually needs. Keep policy and approval outside the model, and log which authority was granted for each consequential call.

### Steps

1. Start scope design from the resource and action being requested.
2. Issue or exchange for the narrowest practical short-lived credential.
3. Enforce scope in the downstream service, not in prompt text.
4. Require approval for authority the current task should not receive automatically.
5. Log the granted scope with the action.

### Example

An incident agent may read the current ticket and restart one service, but it does not inherit a broad cloud administrator key for the entire account.

### Check

A mistaken model action outside the task's granted scope is rejected even if the prompt asks for it.

### Limits

- Scoped tokens are not a substitute for authorization design, audit or human approval where the impact requires it.

### Evidence and sources

- supports: Broad long-lived credentials can be exchanged at runtime for tool-specific, task-scoped credentials so an agent receives only the authority needed for the current action. — RS-3834F341D660F727. Token exchange does not replace authorization policy, approval or audit. The credential issuer and downstream service must actually enforce the requested scope. (4:52-21:32, enforcement point, token exchange, task-scoped credentials and resource-first scope design)
- RS-3834F341D660F727: It's 10pm. Do You Know Where Your Agents Are? — https://ai.engineer/talks/I3znWC3MEXM-its-10pm-do-you-know-where

### Vedokrok editorial review

Check the authority a scoped token actually grants

Requesting narrow access is only the first half of the job. The issuer must grant it, and the receiving service must enforce it.

#### Start with permissions the service understands

Where token exchange is supported, request access for the required resource and operation, with a short lifetime appropriate to the task. Check the issued scope and expiry instead of assuming the request was granted unchanged. Keep approval and policy outside the model. Do not invent a task-level permission that the receiving service cannot enforce.

#### Keep the broad credential out of reach

For a triage agent, test that reading the allowed ticket succeeds while an out-of-scope write is denied. Also check that the agent cannot fall back to a broader credential. OAuth token exchange normally leaves the input token valid; one-time or provider-specific rules can differ. Revocation of the old and new tokens is not automatically linked. Removing broad access needs its own credential-handling and revocation plan.

**Editorial takeaway:** Record what was granted and test one allowed and one forbidden operation in a safe environment. If the service cannot express the needed boundary, use an enforcing gateway or withhold that access.

#### Editorial evidence

- contextualizes: The talk recommends placing policy at the tool boundary and starting scope design with the resource server’s existing permissions. — RS-3834F341D660F727. Selected transcript passages and chapter text read on 2026-10-04. This is vendor architecture guidance, not proof that every API supports the proposed scopes. (Transcript 4:52–7:23 and 21:32–22:26; chapter Start scope design with the resource)
- supports: RFC 8693 distinguishes requested target resources and scope from the issued token response, whose scope must be reported if it differs. — oauth-token-exchange. Selected specification sections read on 2026-10-04. Provider support and enforcement must be checked separately; no token was created, exchanged or tested. (Sections 2.1.1 and 2.2.1, scope and expires_in)
- limits: Token exchange normally leaves the input token valid unless token-specific semantics say otherwise; linked revocation is not a general protocol property. — oauth-token-exchange. This limits the idea that exchange alone removes broad authority. Keeping the original credential inaccessible and testing denial are editorial deployment checks. (Section 2.1, final paragraphs before 2.1.1)
- RS-3834F341D660F727: It's 10pm. Do You Know Where Your Agents Are? — https://ai.engineer/talks/I3znWC3MEXM-its-10pm-do-you-know-where
- oauth-token-exchange: RFC 8693: OAuth 2.0 Token Exchange — https://www.rfc-editor.org/rfc/rfc8693.html

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/run-agent-actions-in-the-user-s-authorization-context

---

## Separate user-visible UI state from model-visible context

ID: MHC-D-RESEARCH-1174 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/separate-user-visible-ui-state-from-model-visible-context

What the user can see and what the model must receive are different design decisions.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An interactive AI tool shows rich application state but the model does not need every displayed field to continue the task.

### Avoid when

- Client implementations and protocol features differ. Verify the actual host's context-sharing behavior rather than assuming a privacy boundary exists.

### Explanation

Define which UI changes must return to the model, which can stay local to the interactive component and which require explicit user action before sharing. This can reduce context noise and unnecessary exposure while keeping the interface useful.

### Example

A planning widget can display all editable form fields to the user while sending only the changed date and selected option back to the model.

### Check

Every model-visible UI update has a task reason; rich display state is not copied into context by default.

### Limits

- Client implementations and protocol features differ. Verify the actual host's context-sharing behavior rather than assuming a privacy boundary exists.

### Evidence and sources

- supports: Interactive agent applications can separate information displayed to the user from information sent back into the model context. — RS-ECDEF17A94432233. Exact MCP App capabilities vary by client and protocol revision; privacy still depends on the host and tool implementation honoring the boundary. (10:01-16:36, application messages, streamed inputs and separation of user-visible state from model-visible context)
- RS-ECDEF17A94432233: MCP Apps: Primitives, Discovery, and the Future of Software — https://ai.engineer/talks/sAOBXCDiDOs-mcp-apps-primitives-discovery-future-software

No review details supplied.

---

## Give computer-use tasks a start state and success check

ID: MHC-D-RESEARCH-1175 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/give-computer-use-tasks-a-start-state-and-success-check

A screenshot is not a test case until you know where the task starts and what state counts as success.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A GUI or computer-use agent is evaluated on tasks whose initial desktop state or completion condition is ambiguous.

### Avoid when

- Clean start states can underrepresent messy production environments. Add controlled variants after the base task is reliable.

### Explanation

Specify the initial environment, relevant files or accounts, and a success predicate before the run. Reset to that state for comparable trials. Prefer checking the resulting system state over inferring success from the agent's final message.

### Steps

1. Define the starting desktop or application state.
2. Seed the files, account data or document needed for the task.
3. Define a success predicate on resulting state.
4. Reset the environment before the next comparable trial.

### Example

For a spreadsheet task, seed the same workbook and mark success by the requested values and formulas in the saved file, not by the agent saying 'done.'

### Check

Another evaluator can reset the task and determine pass or fail from the resulting state without reading the agent's self-report.

### Limits

- Clean start states can underrepresent messy production environments. Add controlled variants after the base task is reliable.

### Evidence and sources

- supports: Computer-use evaluations are easier to interpret when each task defines a known initial environment and a verifiable success condition. — RS-273F6FA5C48C6C0B. A clean initial state improves reproducibility but can be easier than messy production desktops; representative variants should be added after the basic task is stable. (6:34-9:34, give each GUI task an initial state and success check, then test how the computer tool changes the result)
- RS-273F6FA5C48C6C0B: Computer-Use 2.0: Agents Just Got Multi-Cursor — https://ai.engineer/talks/ZSQb5fzRFPw-computer-use-2-0-agents-just-got

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/run-ai-evals-from-a-clean-starting-state

---

## Let the model propose; let deterministic code commit

ID: MHC-D-RESEARCH-1176 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/let-the-model-propose-let-deterministic-code-commit

Use the model for judgment; use the control plane for authority.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent can trigger consequential mutations directly from model output.

### Avoid when

- Deterministic guards constrain effects but cannot guarantee the model chose the right business objective; consequential intent still needs appropriate review.

### Explanation

Treat model output as a proposed action. Deterministic infrastructure validates schema, permissions, limits, idempotency and current state before committing the mutation. Record the decision and result so failures can be audited, retried or reconciled without asking the model what probably happened.

### Example

An agent proposes updating 500 customer records. The control plane validates the allowed fields, batch size and authorization, assigns an operation ID and performs the write.

### Check

A malformed or unauthorized proposal fails before mutation, and a later operator can reconstruct why an accepted action ran.

### Limits

- Deterministic guards constrain effects but cannot guarantee the model chose the right business objective; consequential intent still needs appropriate review.

### Evidence and sources

- supports: A deterministic control plane can treat model outputs as proposals, validate them, commit permitted actions and keep a decision history for recovery and audit. — RS-CC5DC111315BEF2B. Deterministic infrastructure cannot make model judgment deterministic; it constrains effects and records decisions rather than eliminating model uncertainty. (3:08-6:27, model proposals, control-plane decision history, coordinated state and distributed-systems protections)
- RS-CC5DC111315BEF2B: Building Deterministic Infrastructure for Non-Deterministic AI Agents — https://ai.engineer/talks/APh1Vx0oLmQ-building-deterministic-infrastructure-non

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/enforce-permissions-outside-the-model

---

## Turn a production finding into a reusable eval artifact

ID: MHC-D-RESEARCH-1177 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/turn-a-production-finding-into-a-reusable-eval-artifact

A failure is expensive twice if the system can forget it.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A production incident teaches the team something, but the lesson is likely to disappear after the immediate fix.

### Avoid when

- Do not convert every noisy observation into permanent rules. Validate the finding and avoid leaking production secrets or personal data into repositories or eval sets.

### Explanation

Preserve a validated production finding in the form that best prevents recurrence: an issue with evidence, a regression example, an evaluator case or a dataset item. Keep the link to the original trace and the execution boundary where the failure mattered. Then rerun the relevant eval after the fix.

### Steps

1. Validate the finding against the original production evidence.
2. Minimize sensitive data before storing a durable artifact.
3. Choose the artifact that can catch the failure again.
4. Link it to the affected component and original evidence.
5. Rerun the relevant evaluation after the repair.

### Example

A coding agent repeatedly edits generated files instead of their source templates. Preserve one verified incident as a regression case and repository instruction test before merging the fix.

### Check

The next version is tested against an artifact derived from the real failure, and the source evidence remains traceable.

### Limits

- Do not convert every noisy observation into permanent rules. Validate the finding and avoid leaking production secrets or personal data into repositories or eval sets.

### Evidence and sources

- supports: A production investigation can preserve evidence by turning a validated finding into a reviewable issue, evaluator or dataset example before the repair is automated. — RS-A4532D2071A27E77. Runtime evidence can contain noise, secrets or personal data; promotion into a durable artifact requires review, minimization and a clear link to the original failure. (3:31-18:04, gather evidence, bring it into the repository, create an issue/evaluator/dataset example and preserve the execution boundary)
- RS-A4532D2071A27E77: From Signal to PR: Anatomy of a Self-Improving Agent — https://ai.engineer/talks/9HbzAWnKbo4-from-signal-pr-anatomy-self-improving-agent

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/turn-a-verified-ai-failure-into-a-regression-test
