# Let AI do useful work without giving it accidental authority

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Design AI-assisted workflows so tools, permissions, evidence, approvals, testing and monitoring match the real task and limit the blast radius of a bad output.
Collection kind: collection
Source: https://vedokrok.com/collections/let-ai-do-useful-work-without-giving-it-accidental-authority
Version: 0.1.0

---

## Give the agent only the tools this job needs

ID: MHC-D-RESEARCH-0308 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/give-the-agent-only-the-tools-this-job-needs

Every extra tool is another verb the system can accidentally conjugate.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI workflow can call multiple tools, plugins or functions on the user's behalf.

### Avoid when

- A minimal tool set does not replace input validation, authorization or review of the tool itself.

### Explanation

Define the job first, then expose only the tools required to finish that job. Remove abandoned experiments and convenient-but-unused integrations from the agent's reach. Tool selection is a capability boundary, not a menu-design problem.

### Checklist

- The task is stated without naming tools first.
- Every exposed tool maps to a required step in that task.
- Unused legacy or trial tools are unavailable to the agent.
- A missing tool produces a visible stop rather than an improvised substitute.

### Example

A research agent that only needs to read repository files should not also inherit issue deletion, release publishing and billing tools.

### Check

You can justify every available tool with a concrete task step.

### Limits

- A minimal tool set does not replace input validation, authorization or review of the tool itself.

### Evidence and sources

- supports: OWASP recommends limiting the extensions available to an LLM agent to the minimum needed for its intended operation. — RS-EEEBBCF69A614717. A smaller tool set reduces attack and error surface but does not establish that the remaining tools are safe. (Minimize extensions)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/start-an-agent-read-only-when-writing-is-not-required

---

## Start an agent read-only when writing is not required

ID: MHC-D-RESEARCH-0309 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/start-an-agent-read-only-when-writing-is-not-required

Reading first gives you evidence. Writing first gives you evidence plus cleanup.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI task begins with inspection, analysis or recommendation and may or may not need to change data later.

### Avoid when

- Read access can still expose sensitive information; scope what can be read as carefully as what can be changed.

### Explanation

Grant the minimum downstream permissions needed for the current phase. If the job is discovery, keep it read-only. Add write, delete, publish or administrative rights only when a concrete step requires them, and scope those rights to the smallest relevant resource.

### Example

Let an agent inspect a GitHub repository and draft a change before giving it permission to update the branch.

### Check

No granted write permission exists without a named operation that needs it.

### Limits

- Read access can still expose sensitive information; scope what can be read as carefully as what can be changed.

### Evidence and sources

- supports: OWASP recommends granting LLM extensions only the downstream permissions necessary for the intended task. — RS-EEEBBCF69A614717. Read access can still expose sensitive data; least privilege includes data scope as well as write capability. (Minimize extension permissions)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/prefer-a-task-shaped-tool-to-an-open-ended-one

---

## Prefer a task-shaped tool to an open-ended one

ID: MHC-D-RESEARCH-0310 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/prefer-a-task-shaped-tool-to-an-open-ended-one

A function named `create_invoice_draft` has fewer dangerous interpretations than `run_anything`.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent can accomplish a job either through a narrow function or through a general shell, browser or arbitrary-request tool.

### Avoid when

- Over-fragmented tools can make workflows brittle; use the narrowest interface that still represents the real job.

### Explanation

When practical, expose a function whose inputs and effects match the intended action instead of an open-ended executor. Narrow tools make authorization, validation, logging and testing more specific. General tools remain useful for expert work, but they deserve stronger containment because their action space is much larger.

### Example

Use a repository file-update function for a known path rather than giving a content agent an unrestricted shell.

### Check

The chosen tool exposes no broad capability merely because it was easier to integrate.

### Limits

- Over-fragmented tools can make workflows brittle; use the narrowest interface that still represents the real job.

### Evidence and sources

- supports: OWASP recommends avoiding open-ended agent extensions where more granular task-specific functionality can be used. — RS-EEEBBCF69A614717. General tools may be justified for expert workflows, but they require stronger containment and authorization. (Avoid open-ended extensions)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

No review details supplied.

---

## Run agent actions in the user's authorization context

ID: MHC-D-RESEARCH-0311 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/run-agent-actions-in-the-user-s-authorization-context

A helpful assistant should not quietly become a shared administrator.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI acts on behalf of different users against shared downstream systems.

### Avoid when

- User-context execution does not make malicious or mistaken user requests safe; policy and high-impact approval may still apply.

### Explanation

Carry the initiating user's identity and allowed scope into downstream calls instead of using one generic high-privilege service identity for everyone. Let the destination system enforce what that user can access. This keeps the agent from turning a reasoning mistake into cross-user authority.

### Example

A repository assistant uses the user's OAuth scope for the selected repository rather than a token that can edit every repository in the organization.

### Check

Changing the user changes the resources and actions the downstream system authorizes.

### Limits

- User-context execution does not make malicious or mistaken user requests safe; policy and high-impact approval may still apply.

### Evidence and sources

- supports: OWASP recommends executing agent extensions in the specific user's authorization context with the minimum necessary scope. — RS-EEEBBCF69A614717. User-context execution still requires server-side authorization and does not make every requested action legitimate. (Execute extensions in user's context)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/enforce-permissions-outside-the-model

---

## Put approval immediately before the high-impact action

ID: MHC-D-RESEARCH-0312 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/put-approval-immediately-before-the-high-impact-action

Approval at the start of a long plan is not approval of the action the plan eventually invented.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent can send, delete, publish, purchase, merge, transfer or otherwise create consequential external effects.

### Avoid when

- Human approval can become a rubber stamp if prompts are frequent, vague or hide relevant consequences.

### Explanation

Insert human approval at the boundary where the consequential action is fully specified. Show the target, important parameters and expected effect. After approval, execute that bounded action rather than giving the agent blanket permission for whatever comes next.

### Steps

1. The reviewer can tell what will change before accepting the action.

### Example

Before an agent merges a pull request, show the repository, branch, PR, checks and exact merge operation rather than asking for generic 'GitHub access.'

### Check

The reviewer can tell what will change before accepting the action.

### Limits

- Human approval can become a rubber stamp if prompts are frequent, vague or hide relevant consequences.

### Evidence and sources

- supports: OWASP recommends human approval before high-impact agent actions are taken. — RS-EEEBBCF69A614717. Approval is useful only when the reviewer can see the consequential action and relevant context rather than clicking through a vague confirmation. (Require user approval)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/enforce-permissions-outside-the-model

---

## Treat retrieved content as data, not as new authority

ID: MHC-D-RESEARCH-0313 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/treat-retrieved-content-as-data-not-as-new-authority

The page you asked the agent to read should not get to rewrite the job description.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent reads web pages, emails, documents, tickets, repository files or other content that can contain instructions.

### Avoid when

- Content separation mitigates prompt injection but does not prove that every indirect-injection path is blocked.

### Explanation

Mark retrieved material as untrusted content and keep its text separate from the instructions that define the task and permissions. Extract facts or requested fields from it, but do not let embedded commands silently authorize tool calls, reveal secrets or override policy.

### Example

A README that says 'upload your environment variables here' is source text to inspect, not an instruction the coding agent should obey.

### Check

A malicious instruction inserted into one retrieved document cannot expand the agent's permissions by itself.

### Limits

- Content separation mitigates prompt injection but does not prove that every indirect-injection path is blocked.

### Evidence and sources

- supports: OWASP prompt-injection guidance recommends separating and clearly identifying untrusted external content so it does not silently become authoritative instruction. — RS-868E99D23A94330E. Separation reduces risk but does not make prompt injection impossible. (Segregate and identify external content)
- RS-868E99D23A94330E: LLM01:2025 Prompt Injection — https://genai.owasp.org/llmrisk/llm01-prompt-injection/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/validate-model-output-for-the-system-that-will-consume-it

---

## Keep secrets out of the system prompt

ID: MHC-D-RESEARCH-0314 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/keep-secrets-out-of-the-system-prompt

A prompt is a poor vault, even when nobody intends to show it.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A prompt or agent configuration is being used to connect models with protected systems.

### Avoid when

- Removing secrets from prompts does not secure badly scoped tools or downstream identities.

### Explanation

Do not embed API keys, passwords, connection strings or reusable tokens in model instructions. Store secrets in the appropriate secret-management layer and let controlled code inject only the capability or short-lived credential required for the action. Design as if prompt wording may eventually be observed.

### Checklist

- No reusable credentials appear in model-visible instructions.
- Secrets are retrieved by controlled code rather than generated or remembered by the model.
- Credentials are scoped and short-lived where the surrounding system supports it.
- Prompt disclosure would reveal wording, not a reusable secret.

### Example

The agent receives a `search_customer` tool, not the database password that makes the tool possible.

### Check

A copy of the full system prompt contains no credential that grants independent access.

### Limits

- Removing secrets from prompts does not secure badly scoped tools or downstream identities.

### Evidence and sources

- supports: OWASP states that sensitive data such as credentials and connection strings should not be stored in system prompts. — RS-A650FC1839E382FF. Secrets also require appropriate storage, rotation, scoping and access controls outside the model. (Separate Sensitive Data from System Prompts)
- RS-A650FC1839E382FF: LLM07:2025 System Prompt Leakage — https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/enforce-permissions-outside-the-model

---

## Enforce permissions outside the model

ID: MHC-D-RESEARCH-0315 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/enforce-permissions-outside-the-model

A sentence in a prompt is guidance. An authorization check is a control.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A workflow relies on prompt instructions such as 'never access other users' data' or 'only admins may delete.'

### Avoid when

- Deterministic authorization still needs correct policy, testing, auditability and secure identity handling.

### Explanation

Put access rules in deterministic downstream code or policy enforcement points that validate every protected request. The model can help choose an action, but it should not be the component that decides whether the caller is allowed to perform it.

### Example

A delete API checks the user's role and record scope even when the agent confidently claims the deletion is allowed.

### Check

Changing or bypassing the model prompt cannot bypass the downstream authorization rule.

### Limits

- Deterministic authorization still needs correct policy, testing, auditability and secure identity handling.

### Evidence and sources

- supports: OWASP recommends enforcing critical authorization and privilege-separation controls independently from the LLM in deterministic, auditable systems. — RS-A650FC1839E382FF. External controls still need correct policy design and testing; moving a decision out of the prompt does not make it correct automatically. (Ensure security controls are enforced independently from the LLM)
- RS-A650FC1839E382FF: LLM07:2025 System Prompt Leakage — https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/

No review details supplied.

---

## Validate model output for the system that will consume it

ID: MHC-D-RESEARCH-0316 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/validate-model-output-for-the-system-that-will-consume-it

Readable text becomes a different risk when the next component treats it as code.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- LLM output will be executed, rendered or interpreted by another component.

### Avoid when

- Validation rules must match the actual sink and can themselves contain vulnerabilities.

### Explanation

Treat model output as untrusted input at the downstream boundary. Validate allowed fields and ranges, use parameterized interfaces where relevant, encode for the destination context, and reject output that requests effects outside the contract. A generic 'valid JSON' check is not enough when the payload later becomes SQL, HTML, a path or a shell argument.

### Steps

1. Output becomes SQL or another query language.: Use parameterized operations and allowlisted semantics rather than string execution.
2. Output is rendered in a browser or message.: Apply the required context-aware encoding and content controls.
3. Output controls a tool call.: Validate the function, arguments, resource scope and authorization before execution.

### Example

Parse an agent's file-edit request into an allowlisted path and operation instead of interpolating its text into a shell command.

### Check

Malformed or out-of-contract model output is rejected before the downstream component acts on it.

### Limits

- Validation rules must match the actual sink and can themselves contain vulnerabilities.

### Evidence and sources

- supports: OWASP recommends treating model output as untrusted input to downstream components and validating or encoding it for the destination context. — RS-88391B56559EEFCF. Validation must match the sink; a JSON shape check does not sanitize a shell command, SQL fragment or HTML payload. (Prevention and Mitigation Strategies)
- RS-88391B56559EEFCF: LLM05:2025 Improper Output Handling — https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/log-what-the-agent-did-not-only-what-it-said

---

## Log what the agent did, not only what it said

ID: MHC-D-RESEARCH-0317 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/log-what-the-agent-did-not-only-what-it-said

The chat transcript is not the audit trail if the real effects happened elsewhere.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An agent can take actions through tools and failures need to be investigated or contained.

### Avoid when

- Logging creates its own privacy and security obligations; collect only what is needed and protect access.

### Explanation

Record the consequential tool call, bounded arguments, authorization context, result and downstream effect needed for review. Link actions to the initiating request and approval where relevant. Keep the log inspectable without copying sensitive payloads that are unnecessary for diagnosis.

### Checklist

- The initiating request or workflow run can be identified.
- Each consequential tool call has a timestamp and result.
- Relevant authorization and approval context is traceable.
- Sensitive payloads are minimized or redacted according to policy.
- Downstream effects can be reconciled with the tool record.

### Example

For a repository agent, keep the PR or commit IDs it created, not just 'Done' in the conversation.

### Check

After a failure, you can identify which external actions occurred and which did not.

### Limits

- Logging creates its own privacy and security obligations; collect only what is needed and protect access.

### Evidence and sources

- supports: OWASP recommends logging and monitoring agent-extension and downstream activity to detect undesirable actions. — RS-EEEBBCF69A614717. Logs should avoid unnecessary sensitive content and need retention and access controls of their own. (Damage-limitation controls: log and monitor activity)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

No review details supplied.

---

## Cross-check the fact before the agent acts on it

ID: MHC-D-RESEARCH-0318 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/cross-check-the-fact-before-the-agent-acts-on-it

Confidence is a writing style. Evidence is a different object.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A generated factual claim will drive a consequential decision, message, configuration or transaction.

### Avoid when

- A cited page can be wrong, stale or derivative; source quality and independence still matter.

### Explanation

Identify the factual proposition that matters to the action, then verify it against a trusted primary or independent source appropriate to the domain. Keep the source with the decision so later reviewers can see what was actually checked. Increase scrutiny with impact; low-stakes drafting does not need the same burden as a legal, financial, security or medical action.

### Steps

1. The consequential action no longer depends solely on an uncited model assertion.

### Example

Before an agent changes an API integration based on a remembered parameter, check the current vendor documentation for that endpoint.

### Check

The consequential action no longer depends solely on an uncited model assertion.

### Limits

- A cited page can be wrong, stale or derivative; source quality and independence still matter.

### Evidence and sources

- supports: OWASP recommends cross-checking critical LLM outputs against trusted external sources and using human oversight for sensitive information. — RS-A911B8F9CB685C39. Two sources are not independent merely because two pages repeat the same underlying claim. (Cross-Verification and Human Oversight)
- RS-A911B8F9CB685C39: LLM09:2025 Misinformation — https://genai.owasp.org/llmrisk/llm092025-misinformation/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/validate-model-output-for-the-system-that-will-consume-it

---

## Keep an eval set that can embarrass the agent

ID: MHC-D-RESEARCH-0319 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/keep-an-eval-set-that-can-embarrass-the-agent

A demo asks whether the system can succeed once. An eval asks where it reliably fails.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI workflow will be reused and quality cannot be judged from one polished demo.

### Avoid when

- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.

### Explanation

Maintain a versioned set of representative tasks, edge cases and known failure examples with explicit pass criteria. Record the model or system version, relevant configuration, metrics and evaluation tooling. Add important real failures after review so the set becomes harder in useful ways instead of merely larger.

### Steps

1. Cases represent the real task distribution, not only easy examples.
2. Known failure modes and boundary cases are included.
3. Each case has an observable pass criterion.
4. Model, prompt/tool configuration and evaluation version are recorded.
5. New cases are added deliberately when they expose a distinct risk or capability gap.

### Example

A document-extraction eval includes clean forms, missing fields, conflicting fields and a case where the correct answer is 'unknown.'

### Check

A new system version can be compared against the same defined cases without reconstructing the test from memory.

### Limits

- Passing a finite eval set does not prove safety, general intelligence or performance on unseen conditions.

### Evidence and sources

- supports: NIST AI RMF calls for documenting AI test sets, metrics and evaluation tools. — RS-89634FBC4E1347EC. A documented evaluation set can still be unrepresentative or weak; provenance and coverage must be judged separately. (Measure 2.1)
- RS-89634FBC4E1347EC: AI RMF Core — https://airc.nist.gov/airmf-resources/airmf/5-sec-core/

### Vedokrok editorial review

A regression set should remember failures without becoming the whole exam

A stable eval set is valuable because it makes regressions visible. The trap is using the same familiar cases as the only proof that a changing agent still works. Once those cases guide development, the team also needs evidence from work the system has not been tuned around.

#### Separate regression from generalization

Keep a stable core of representative tasks, edge cases and known failures. Its job is to answer a practical question: did this change break something we already know matters? NIST’s AI RMF supports documenting test sets, metrics and evaluation tools so results can be repeated and interpreted. But a regression suite and a fresh test answer different questions. The first protects known behavior; the second gives a better check on whether improvements extend beyond the cases the team has repeatedly seen.

#### Repeated reuse can turn the test into part of the development loop

This is not only a language-model problem. Work on adaptive data analysis shows that repeatedly using a holdout to choose what to try next can overfit the holdout itself. LLM evaluation adds another route: benchmark items may already appear in training data. A 2025 study of 83 software-engineering benchmarks found leakage was uneven rather than universal, but heavily leaked benchmarks existed and the authors reported that leakage affected measured model performance. A rising score therefore needs a provenance question as well as a scoring question.

#### Fresh cases help, but freshness is not a magic control

LiveBench responds to contamination risk by adding questions from recent sources and updating the benchmark over time. That is useful evidence for a rolling or held-out slice, not a reason to discard the stable regression set. Fresh data can also differ in difficulty or distribution. In the ImageNet replication study, accuracy fell on newly collected test sets, yet the authors concluded the drop was better explained by the new images being slightly harder than by adaptive overfitting. If a fresh slice scores worse, investigate both possibilities before declaring the model overfit.

#### Three questions before you trust the score

Should every production failure enter the eval? Add a failure when it represents a distinct, important risk or capability gap, but put it in the regression set; do not let all future measurement collapse into known cases. How large must the eval be? There is no universal number: coverage of real task types and uncertainty around the score matter more than a round target. Does 100% mean the agent is ready? Only for the defined cases, configuration and scoring rule. Keep a fresh or time-separated check when generalization matters, and record which set influenced development.

**Editorial takeaway:** Use two complementary views: a stable regression set that remembers important failures, and a fresh or protected slice that the development loop has not repeatedly optimized against. Version both, keep scoring rules explicit, and treat a score as evidence about those cases—not a certificate for unseen work.

#### Editorial evidence

- supports: NIST AI RMF Measure 2.1 calls for documenting test sets, metrics and details of tools used during test, evaluation, validation and verification. — nist-ai-rmf-core-measure. This supports versioned, repeatable evaluation records. NIST does not say that one finite set proves performance outside the tested conditions. (Measure 2.1)
- limits: Adaptive reuse of a holdout can overfit the holdout when observed results influence later hypotheses or choices. — dwork-holdout-2015. The paper is about adaptive data analysis rather than agent-eval operations, so the regression-versus-fresh-slice workflow here is an editorial application of the general reuse problem. (Abstract)
- challenges: Fresh CIFAR-10 and ImageNet test sets produced lower accuracy than the original test sets, but the authors attributed the drop mainly to the new data being slightly harder rather than to adaptive test-set overfitting. — recht-imagenet-2019. A worse fresh-set score is therefore not, by itself, proof that the development process overfit the old eval. Distribution and difficulty need checking too. (Abstract)
- contextualizes: LiveBench uses frequently updated questions from recent sources and objective scoring to reduce test-set contamination and keep the benchmark useful as models change. — livebench-iclr-2025. This is a benchmark-design example, not evidence that monthly refresh is the correct cadence for every internal agent eval. (Abstract)
- challenges: A 2025 analysis of 83 software-engineering benchmarks found benchmark leakage into StarCoder training data was uneven, with some benchmarks heavily affected, and reported that leakage materially affected evaluation results. — lessleak-2025. The study concerns one open-model training corpus and software-engineering benchmarks. Its leakage rates should not be generalized to unrelated models or private eval sets. (Abstract)
- nist-ai-rmf-core-measure: AI RMF Core — https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ — National Institute of Standards and Technology
- dwork-holdout-2015: Generalization in Adaptive Data Analysis and Holdout Reuse — https://proceedings.neurips.cc/paper/2015/hash/bad5f33780c42f2588878a9d07405083-Abstract.html — Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, Aaron Roth
- recht-imagenet-2019: Do ImageNet Classifiers Generalize to ImageNet? — https://proceedings.mlr.press/v97/recht19a.html — Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar
- livebench-iclr-2025: LiveBench: A Challenging, Contamination-Limited LLM Benchmark — https://proceedings.iclr.cc/paper_files/paper/2025/hash/e4a46394ba5378b3f9a186a5b4c650d1-Abstract-Conference.html
- lessleak-2025: LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks — https://arxiv.org/abs/2502.06215 — Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, David Lo

Editorial version: 1.0.0 · evidence checked 2026-09-25 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/test-the-agent-under-conditions-that-resemble-the-real-job

---

## Test the agent under conditions that resemble the real job

ID: MHC-D-RESEARCH-0320 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/test-the-agent-under-conditions-that-resemble-the-real-job

A perfect lab result can still be a poor rehearsal.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI workflow performs well in a clean sandbox but will face different data, latency, permissions or users in production.

### Avoid when

- Similarity to deployment is contextual; do not claim full production equivalence from a test environment.

### Explanation

List the conditions that materially shape the deployed task—data distribution, tool availability, permission boundaries, response time, interaction length and failure handling. Reproduce the important ones in evaluation rather than testing only the easiest environment. Record differences you cannot reproduce.

### Example

Test a support agent with realistic long threads, unavailable tools and restricted user permissions instead of only single-turn happy paths.

### Check

The evaluation report names the production conditions represented and the material gaps that remain.

### Limits

- Similarity to deployment is contextual; do not claim full production equivalence from a test environment.

### Evidence and sources

- supports: NIST AI RMF recommends measuring AI performance or assurance criteria under conditions similar to deployment settings. — RS-89634FBC4E1347EC. No test environment perfectly reproduces production, so material differences should be documented. (Measure 2.3)
- RS-89634FBC4E1347EC: AI RMF Core — https://airc.nist.gov/airmf-resources/airmf/5-sec-core/

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/monitor-the-agent-after-deployment

---

## Monitor the agent after deployment

ID: MHC-D-RESEARCH-0321 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/monitor-the-agent-after-deployment

Deployment is the first time the model meets all the mess your test set forgot.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An AI workflow is useful enough to run repeatedly in real conditions.

### Avoid when

- NIST notes that AI monitoring practice is still developing; a metric set should not be presented as a complete safety methodology.

### Explanation

Define a small set of post-deployment signals tied to real failure modes: invalid actions, corrections, refusals, escalation, unexpected tool use, latency or other task-specific outcomes. Review incidents and field behavior for conditions not represented in predeployment tests. Feed verified failures back into controls and evaluation rather than treating monitoring as a dashboard decoration.

### Steps

1. Choose signals linked to concrete failure or harm modes.
2. Capture enough context to investigate without collecting unnecessary sensitive data.
3. Review unexpected behavior and user corrections on a defined cadence or trigger.
4. Turn confirmed new failure modes into a control, test case or product decision.

### Example

Track rejected tool calls and manual corrections after an agent goes live; cluster repeated causes and add representative failures to the eval set.

### Check

A real-world failure has a route from detection to investigation and, when warranted, to a changed control or evaluation.

### Limits

- NIST notes that AI monitoring practice is still developing; a metric set should not be presented as a complete safety methodology.

### Evidence and sources

- supports: NIST recommends production monitoring of AI behavior, and its 2026 monitoring report explains why controlled pre-deployment evaluations cannot capture all real-world variability and unexpected consequences. — RS-06C6740A8538704F. The 2026 report explicitly notes that monitoring methods and terminology remain nascent and scattered. (Abstract)
- RS-06C6740A8538704F: Challenges to the monitoring of deployed AI systems: Center for AI Standards and Innovation — https://www.nist.gov/publications/challenges-monitoring-deployed-ai-systems-center-ai-standards-and-innovation

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/keep-an-eval-set-that-can-embarrass-the-agent

---

## Bound how much damage one agent run can do

ID: MHC-D-RESEARCH-0322 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/bound-how-much-damage-one-agent-run-can-do

Automation turns one mistake into throughput unless you design a brake.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An automated agent can repeat actions quickly or affect many records before a person notices.

### Avoid when

- Rate limits and stop controls reduce blast radius; they do not substitute for correct permissions, validation and approval.

### Explanation

Set limits on action count, affected resources, spend or batch size that match the task. Add a visible stop path for behavior outside expectations and make recovery possible where the downstream system supports rollback. The aim is not to make bad actions acceptable; it is to prevent one error from scaling silently.

### Checklist

- One run has an explicit maximum action or resource scope.
- Unusual action rates can be detected or stopped.
- A human can halt or suspend further actions.
- High-impact batches are split or sampled before full execution when practical.
- Rollback or compensating action is documented where available.

### Example

An agent updating customer records first changes a small bounded batch and pauses on error-rate thresholds instead of editing the full database in one pass.

### Check

You can state the maximum plausible effect of one unreviewed run under the configured controls.

### Limits

- Rate limits and stop controls reduce blast radius; they do not substitute for correct permissions, validation and approval.

### Evidence and sources

- supports: OWASP recommends rate limiting to reduce the number of undesirable agent actions, while NIST safety guidance includes the ability to shut down, modify or intervene in systems that deviate from expected behavior. — RS-EEEBBCF69A614717. A rate cap and stop path limit damage; they do not make an unsafe autonomous action acceptable. (Damage-limitation controls: rate limiting)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

### Vedokrok editorial review

Bound the total effect of one agent run

A slow mistake can still reach every record. Set a total limit as well as a rate limit, and decide how further actions will stop.

#### Set separate limits

Before a bulk task, name the maximum records, actions or spend allowed for one run. Set the action rate separately. A rate limit restricts actions within a time window; it does not by itself impose a total run budget. Enforce the chosen limits outside the model, alongside normal permissions and approval.

#### Test the stopping point

For a catalogue update, allow a small named batch and require review before more records can change. In a safe test, try one action beyond the batch limit and verify that it is blocked. Check who can stop further work and what happens to actions already sent. A stop signal does not undo completed changes; document recovery only where it is available.

**Editorial takeaway:** Before scaling up, verify the total cap, the rate limit and the stop path separately. A bounded run can still perform a harmful action, so keep permission, validation and approval checks in place.

#### Editorial evidence

- supports: OWASP describes rate limiting as a damage-limitation measure for agent actions within a time period. — RS-EEEBBCF69A614717. Selected passages read on 2026-10-04. The rate applies over time; the separate total-run cap and boundary test are editorial implementation advice. (Prevention and Mitigation Strategies; damage-limitation bullets after item 8)
- supports: NIST AI RMF includes shutdown, modification or human intervention when AI systems depart from expected behavior. — RS-37C3116E2E2D73CF. Selected passage read on 2026-10-04. NIST calls for context-specific safety measures, not a universal stopping or rollback mechanism. (AI RMF 1.0, section 3.2 Safe, final two paragraphs)
- limits: OWASP distinguishes damage-limitation measures from controls intended to prevent excessive agency. — RS-EEEBBCF69A614717. Selected passage read on 2026-10-04. A cap or stop path does not replace authorization, input validation or approval. The proposed check is not a measured efficacy claim. (Prevention and Mitigation Strategies; paragraph introducing logging and rate limiting)
- RS-EEEBBCF69A614717: LLM06:2025 Excessive Agency — https://genai.owasp.org/llmrisk/llm062025-excessive-agency/
- RS-37C3116E2E2D73CF: AI Risks and Trustworthiness — https://airc.nist.gov/airmf-resources/airmf/3-sec-characteristics/

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/put-approval-immediately-before-the-high-impact-action
Related (useful_with): https://vedokrok.com/knowledge/monitor-the-agent-after-deployment
