# Find the failure without creating a second incident

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Reduce a reproducible failure, locate its boundary and control the risks of retries, concurrent edits and recovery.
Collection kind: collection
Source: https://vedokrok.com/collections/find-the-failure-without-creating-a-second-incident
Version: 0.1.0

---

## Shrink the example until the same failure is hard to hide

ID: MHC-D-RESEARCH-0970 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/shrink-the-example-until-the-same-failure-is-hard-to-hide

A thousand-line attachment is evidence. A six-line reproducer is an invitation to solve the problem.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A large file, message or sequence fails, but nobody knows which part matters.

### Avoid when

- Do this in an isolated test environment. Intermittent failures need repeatability work first; smallest-looking is not the same as globally minimal.

### Explanation

Reduce the input without changing the failure you are investigating. Remove a chunk, rerun the same check and keep the smaller input only when the original failure remains. When large cuts stop working, try smaller ones.

### Steps

1. Freeze a safe copy, the environment and an exact failure signature.
2. Remove one chunk; distinguish the original failure from an unrelated validation error.
3. Save the reduced failing case and a nearby passing case.

### Example

A text import fails with hundreds of lines. Reduction leaves two lines and a separator; deleting that separator makes the same test pass.

### Check

Another person can reproduce the specified failure with the reduced input.

### Limits

- Do this in an isolated test environment. Intermittent failures need repeatability work first; smallest-looking is not the same as globally minimal.

### Evidence and sources

- supports: Delta debugging removes parts of an input while preserving a specified failure; a 1-minimal result need not be globally smallest. — RS-DAF12B69E9E3B42E. The practical reduction loop assumes a meaningful, sufficiently repeatable test and preserves the original failure signature. (Delta Debugging; 1-minimality discussion)
- RS-DAF12B69E9E3B42E: Reducing Failure-Inducing Inputs — https://www.debuggingbook.org/html/DeltaDebugger.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/bisect-the-version-history-instead-of-inspecting-every-change

---

## Bisect the version history instead of inspecting every change

ID: MHC-D-RESEARCH-0971 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/bisect-the-version-history-instead-of-inspecting-every-change

The last edit is a suspect, not automatically the culprit.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- The same test passed in an earlier version and fails now.

### Avoid when

- Repeated introduction and removal of a bug, changing dependencies or flaky tests can invalidate a simple one-transition story.

### Explanation

Choose a reproducibly passing version and a failing version. Test a version between them, then keep the half containing the transition. Continue until the boundary is narrow enough to inspect. Git bisect automates this search for code history.

### Steps

1. Use the same input, test and relevant environment across versions.
2. Record untestable versions as unknown, not as failures.
3. Confirm the candidate boundary with a fresh run and inspect the actual change.

### Example

A mapping worked in release A and breaks in H. Testing D and then F narrows the investigation without reading every transport first.

### Check

The report names the tested boundary and any skipped versions that prevent a precise conclusion.

### Limits

- Repeated introduction and removal of a bug, changing dependencies or flaky tests can invalidate a simple one-transition story.

### Evidence and sources

- supports: Git bisect narrows a known-good to known-bad version interval through repeated tests; skipped versions can leave an ambiguous boundary. — RS-BB2489EEC3B17FAA. A first failing version is an investigation lead, not proof of a complete root cause. (Description; Bisect skip)
- RS-BB2489EEC3B17FAA: git-bisect: Use binary search to find the commit that introduced a bug — https://git-scm.com/docs/git-bisect

No review details supplied.

---

## Find the first boundary where the record becomes wrong

ID: MHC-D-RESEARCH-0972 · Version: 0.1.0 · Kind: template
Source: https://vedokrok.com/knowledge/find-the-first-boundary-where-the-record-becomes-wrong

Follow one record, not five dashboards reporting different populations.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Several systems passed a record around and each team says its own step succeeded.

### Avoid when

- An absent observation may be a logging gap. Do not label a team or component as the root cause until the transformation is checked.

### Explanation

Compare the same business object at successive boundaries. Check the value against the transformation expected at that boundary; not every difference is a defect. The first unjustified divergence narrows the investigation without pretending to finish it.

### Template

Object and run: [identity]. Boundary: [step]. Input evidence: [before]. Expected transformation: [rule]. Output evidence: [after]. First unexplained difference: [gap].

### Example

A partner role exists in the sender and outbound message but is absent after inbound mapping. Investigate that mapping boundary before resending the whole population.

### Check

The compared artifacts refer to the same object, version and processing attempt.

### Limits

- An absent observation may be a logging gap. Do not label a team or component as the root cause until the transformation is checked.

### Evidence and sources

- supports: Google's troubleshooting guidance recommends examining successive components of a stack or data pipeline and narrowing the suspect boundary. — RS-C7E43CE158CCDA2E. The same-object boundary table is an authored application; transformations must be judged against their intended contracts. (Divide and conquer)
- RS-C7E43CE158CCDA2E: Effective Troubleshooting — https://sre.google/sre-book/effective-troubleshooting/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/shrink-the-example-until-the-same-failure-is-hard-to-hide

---

## Keep every rerun result when the failure is intermittent

ID: MHC-D-RESEARCH-0973 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/keep-every-rerun-result-when-the-failure-is-intermittent

Rerunning until green edits the story, not the software.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An unchanged test alternates between passing and failing.

### Avoid when

- No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk.

### Explanation

Preserve the sequence of outcomes and the conditions of each attempt. First establish that the result varies without the intended code change. Then investigate candidate dependencies such as concurrency, test order or external services instead of treating the final pass as a repair.

### Steps

1. Record version, input, environment, attempt and outcome together.
2. Compare failing and passing attempts for a specific condition that changed.
3. Test that condition deliberately in a safe environment and retain failures in the report.

### Example

A batch test passes alone but fails after another test leaves shared state behind. An isolated green run did not cover that sequence.

### Check

The evidence contains unsuccessful attempts and distinguishes a suspected condition from a confirmed cause.

### Limits

- No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk.

### Evidence and sources

- supports: A flaky test can pass and fail on the same code, so one successful rerun does not establish that the underlying problem was fixed. — RS-B476D6C7D31BC624. Repeated attempts reveal variability but do not by themselves identify its cause or certify reliability. (Opening account of nondeterministic tests)
- RS-B476D6C7D31BC624: Where do our flaky tests come from? — https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/shrink-the-example-until-the-same-failure-is-hard-to-hide

---

## Trace the request by identity, not by nearby timestamps

ID: MHC-D-RESEARCH-0974 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/trace-the-request-by-identity-not-by-nearby-timestamps

Two messages appearing at 10:03 do not necessarily belong to the same story.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Logs from multiple services need to be connected to one operation.

### Avoid when

- A shared identifier can be misused or reused incorrectly. A trace shows what was instrumented, not a guaranteed complete history.

### Explanation

Use propagated request or trace identifiers to connect work across services. Preserve parent or causal links when asynchronous processing starts a separate trace. Timestamps help order observations, but proximity alone cannot establish that one message caused another.

### Checklist

- Identify the initiating operation and its correlation identifier.
- Follow the identifier or explicit causal link through each boundary.
- Mark missing instrumentation and avoid copying sensitive payloads into a tracing field.

### Example

A queue consumer starts minutes after the sender finishes. The recorded causal link, not clock proximity, connects their attempts.

### Check

Each claimed connection has an identifier or documented linking rule; gaps remain visible.

### Limits

- A shared identifier can be misused or reused incorrectly. A trace shows what was instrumented, not a guaranteed complete history.

### Evidence and sources

- supports: OpenTelemetry uses propagated trace context and span links to associate work across synchronous and asynchronous boundaries. — RS-11A3596487BDCEAE. Recorded links describe instrumented relationships, not necessarily the complete system history. (Context Propagation; Span Links)
- RS-11A3596487BDCEAE: Traces — https://opentelemetry.io/docs/concepts/signals/traces/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/find-the-first-boundary-where-the-record-becomes-wrong

---

## Treat a timeout as an unknown outcome before repeating the action

ID: MHC-D-RESEARCH-0975 · Version: 0.1.0 · Kind: pattern
Source: https://vedokrok.com/knowledge/treat-a-timeout-as-an-unknown-outcome-before-repeating-the-action

No reply is not the same as no action.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A send, create or update request times out after leaving your application.

### Avoid when

- Reconciliation against stale or incomplete data can also mislead. Do not invent an automatic retry rule for an undocumented interface.

### Explanation

The receiver may have completed the operation while its response was lost. Preserve an unknown or pending state and reconcile using the operation's identity before creating a fresh request. Where a documented idempotent retry is available, follow that contract.

### Recognition

No reply is not the same as no action.

### Example

An import request times out. Its run identifier reveals that the job is already processing, so a second import is not started.

### Check

The status distinguishes confirmed failure from an unconfirmed outcome.

### Limits

- Reconciliation against stale or incomplete data can also mislead. Do not invent an automatic retry rule for an undocumented interface.

### Evidence and sources

- supports: A request that receives no response may already have created its intended resource; blindly retrying can create additional effects. — RS-ABFC60C5149B88E0. Reconciliation requires an authoritative receiver view and enough identity information to distinguish this operation from another. (Retrying and side effects)
- RS-ABFC60C5149B88E0: Making retries safe with idempotent APIs — https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/reuse-the-operation-identity-when-retrying-the-same-intent

---

## Reuse the operation identity when retrying the same intent

ID: MHC-D-RESEARCH-0976 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/reuse-the-operation-identity-when-retrying-the-same-intent

A retry should not introduce itself as a brand-new business request.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- An interface supports idempotency and transient failures require retries.

### Avoid when

- A token field alone does nothing unless the receiver enforces it. Expired keys, changed scope and partially completed workflows need explicit handling.

### Explanation

An idempotency key identifies one intended operation, not merely a network attempt. Retain that key when retrying the same operation under the receiver's documented rules. A genuinely new intent needs its own identity, even when its payload happens to look identical.

### Steps

1. Confirm that this operation implements deduplication and read its scope and retention rules.
2. Store the operation key with the original parameters before sending.
3. Reuse the key for permitted retries; reject accidental parameter changes instead of silently reusing it.

### Example

Retrying one account-creation request preserves its key. Creating a second account deliberately uses a different key.

### Check

A controlled retry test produces one intended effect, and a separate intent remains possible.

### Limits

- A token field alone does nothing unless the receiver enforces it. Expired keys, changed scope and partially completed workflows need explicit handling.

### Evidence and sources

- supports: Supported EC2 operations use client tokens to recognize repeated requests, with operation-specific scope and parameter rules. — RS-665898252AE611BF. Adding a token to an arbitrary request has no protective effect unless the receiver implements the corresponding contract. (Client tokens; Types of idempotency)
- RS-665898252AE611BF: Ensuring idempotency in Amazon EC2 API requests — https://docs.aws.amazon.com/ec2/latest/devguide/ec2-api-idempotency.html

No review details supplied.
Related (compare_with): https://vedokrok.com/knowledge/make-the-write-conditional-on-the-version-you-reviewed

---

## Make the write conditional on the version you reviewed

ID: MHC-D-RESEARCH-0977 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/make-the-write-conditional-on-the-version-you-reviewed

A careful review of yesterday's value does not authorize overwriting today's change.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Another person or process can edit the same record before your update reaches it.

### Avoid when

- Reading twice is not atomic protection. The check must be enforced by the receiving system, with the correct version scope.

### Explanation

Bind the update to the version that informed it. With a supported HTTP interface, an entity tag and If-Match can provide this check. If the version changed, reread and reconcile the difference instead of forcing the old update through.

### Steps

1. Read the value and its server-provided version together.
2. Submit the change with an enforced version precondition.
3. On conflict, inspect the new state and prepare a fresh, reviewed update.

### Example

While you correct an address, another user changes the contact details. A version conflict prevents your stale full-record payload from silently erasing that work.

### Check

A test that changes the record between read and write is rejected or handled as a documented conflict.

### Limits

- Reading twice is not atomic protection. The check must be enforced by the receiving system, with the correct version scope.

### Evidence and sources

- supports: HTTP If-Match can make a state-changing request conditional on a matching representation version and prevent lost updates. — RS-8500D155C07EFEF9. A client-side read followed by an unconditional write is not an atomic version check. (RFC 9110, section 13.1.1)
- RS-8500D155C07EFEF9: RFC 9110: HTTP Semantics — https://www.rfc-editor.org/rfc/rfc9110.html

### Vedokrok editorial review

Make the server check the version before writing

A second read does not close the gap between reading a record and overwriting it. The receiver must enforce the version condition when it handles the write.

#### Use the reviewed version

For an HTTP API that supports this contract, read the representation and its strong entity tag, or ETag, together. Send that tag unchanged in If-Match with the intended update. Tags marked W/ are weak and cannot satisfy strong comparison. If-Match: * checks that a current representation exists; it does not pin your update to the version you reviewed.

#### Treat a conflict as new information

On a failed precondition, fetch the current record and reconcile changes before preparing another update. Do not strip the condition to force a stale write through. For example, a new emergency-contact number entered during your address correction must survive the correction. Test this race with safe data and check that the stale write is blocked. Follow documented handling if the receiver reports the change was already applied.

**Editorial takeaway:** Before relying on a version condition, confirm that the receiving API enforces it on the representation you reviewed. Use its documented conflict handling rather than forcing a stale update.

#### Editorial evidence

- supports: If-Match requires strong entity-tag comparison; its wildcard tests existence, and a false condition forbids performing the requested method. — RS-8500D155C07EFEF9. Selected normative passages inspected on 2026-10-04. The server may report success when it determines that the requested change was already applied. No API behavior was tested in this review. (Sections 8.8.3 and 13.1.1)
- limits: For a state-changing request whose If-Match condition fails, RFC 9110 permits a success response when the requested change appears to have already been applied. — RS-8500D155C07EFEF9. Rechecked on 2026-10-04. A failed comparison need not always surface as HTTP 412; recovery must interpret the receiving API’s documented outcome without forcing another write. (13.1.1 If-Match, paragraphs following condition evaluation: already-applied 2xx response)
- RS-8500D155C07EFEF9: RFC 9110: HTTP Semantics — https://www.rfc-editor.org/rfc/rfc9110.html#section-13.1.1

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/reverse-the-business-effect-without-restoring-stale-history

---

## Reverse the business effect without restoring stale history

ID: MHC-D-RESEARCH-0978 · Version: 0.1.0 · Kind: question
Source: https://vedokrok.com/knowledge/reverse-the-business-effect-without-restoring-stale-history

Undo is a business decision when other people have moved on.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A workflow partly completed and a simple rollback could overwrite legitimate later work.

### Avoid when

- Some actions cannot be fully reversed. Compensation must respect domain rules, audit history and authorization; never test it on live transactions casually.

### Explanation

Plan compensation around the effect that must be corrected. Restoring an old snapshot can erase concurrent changes. A compensating action may therefore differ from the original action in reverse, and some consequences require manual resolution rather than a pretend rollback.

### Question

Which completed effect needs correction, and which later changes must remain? · Who can authorize the compensating action? · What evidence proves it completed, and what happens if compensation also fails?

### Example

A partially completed reservation is cancelled through its supported cancellation process; unrelated customer updates are not replaced with an old database snapshot.

### Check

The recovery plan names the intended post-recovery business state, not merely an old technical state.

### Limits

- Some actions cannot be fully reversed. Compensation must respect domain rules, audit history and authorization; never test it on live transactions casually.

### Evidence and sources

- supports: A compensating transaction reverses business effects while accounting for concurrent work rather than simply restoring an old snapshot. — RS-3F83EE739EFB9029. Not every effect is reversible; the appropriate compensation and its approval are domain-specific. (Solution)
- RS-3F83EE739EFB9029: Compensating Transaction pattern — https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-the-canary-a-fair-comparison-and-a-real-stopping-rule

---

## Give the canary a fair comparison and a real stopping rule

ID: MHC-D-RESEARCH-0979 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/give-the-canary-a-fair-comparison-and-a-real-stopping-rule

Five easy successes can be a very convincing test of the wrong workload.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A limited rollout is being used to decide whether a change should reach everyone.

### Avoid when

- A canary still exposes real work to risk. Stateful replay, shared caches and duplicated side effects require isolation, not just a smaller sample.

### Explanation

Choose exposure that can reveal the relevant failure, then compare it with an appropriate unchanged control. Define the harmful signals and response before starting. Keep shared infrastructure in view: a bad canary can affect the control and hide the difference you expected to measure.

### Checklist

- Include the important workload types and enough time for delayed effects to appear.
- Compare attributable outcome metrics, not just a green deployment status.
- Assign a stop or rollback action and the person authorized to trigger it.

### Example

A new batch transformation is tried on representative record types, including exceptions, while the unchanged path supplies a comparison.

### Check

The rollout decision states which workloads and observation period were actually covered.

### Limits

- A canary still exposes real work to risk. Stateful replay, shared caches and duplicated side effects require isolation, not just a smaller sample.

### Evidence and sources

- supports: Canary evaluation needs representative exposure, attributable metrics and attention to shared failure domains between canary and control. — RS-E781EA1C862F0DE3. The source does not justify a universal safe sample size, duration or rollout percentage. (Canary population; Metrics Should Be Representative and Attributable)
- RS-E781EA1C862F0DE3: Canarying Releases — https://sre.google/workbook/canarying-releases/

No review details supplied.
