---
name: vedokrok-debug-flaky-failure
description: "Treat both passing and failing runs as evidence, shrink the failure domain without changing the failure and preserve the next discriminating test. Use when: The same target can pass and fail across runs."
metadata:
  version: "1.0.1"
  release: "MHC-RPUB-20261011-9e4b62c5"
  website-snapshot: "b97dfd5520e2cf31744978d7e824b94e55bd12d9a2bdef759b5bc69a963375e5"
---

This skill runs the task workflow below. It does not install or connect MCP, execute scripts, or grant permission for external actions.

Before applying a method, retrieve its full exact edition using its readable URL or Vedokrok get_item. Read its evidence roles, conditions, limits and sources. The summaries below are a method map, not the full evidence record. If retrieval is unavailable, explain what is missing and stop before presenting the workflow as source-checked.

Check current availability and notices before consequential reuse. Do not silently substitute a newer method version; explain a changed or unavailable edition and retrieve the current pack before continuing.

# Debug an Intermittent Failure or Flaky Test

Treat both passing and failing runs as evidence, shrink the failure domain without changing the failure and preserve the next discriminating test.

Pack version: 1.0.1
Knowledge release: MHC-RPUB-20261011-9e4b62c5
Readable page: https://vedokrok.com/solutions/debug-flaky-failure
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json

## Situation

A bug, CI job or test passes and fails under apparently similar conditions and repeated reruns are producing more noise than diagnosis.

## Intended outcome

A pass/fail comparison register, bounded hypotheses, a reproducibility plan and the next test most likely to eliminate a cause.

## Use when

- The same target can pass and fail across runs.
- Version, timing, order, environment or shared state may matter.
- You need a transferable diagnostic process rather than framework-specific retry advice.

## Avoid when

- The failure is deterministic and already has a small reproducible case.
- Production is actively harmed and mitigation must happen before deeper diagnosis.

## Agent protocol

### Level 0 — Freeze the failure signature

Define what counts as the same failure before changing the system.

Methods: MHC-D-RESEARCH-0973

### Level 1 — Compare red and green runs

Record observable conditions that differ across the nearest comparable pass and fail.

Methods: MHC-D-RESEARCH-0973

### Level 2 — Keep competing causes

Maintain more than one plausible explanation and identify predictions that differ.

Methods: MHC-D-RESEARCH-0295

### Level 3 — Reduce the search space

Use bisection only when the candidate space has a valid testable order.

Methods: MHC-D-RESEARCH-0295, MHC-D-RESEARCH-0971

### Level 4 — Inspect semantic boundaries

Capture the point where two components may disagree about the same data.

Methods: MHC-D-RESEARCH-1159

### Level 5 — Mitigate, test or continue

If impact is active, mitigate without destroying evidence; otherwise run the next discriminating test.

Methods: MHC-D-RESEARCH-0300

## Method Register

For every method in the pack, record:

- item_id
- version
- role
- fit: use | skip | unclear
- concise_reason
- important_limit
- source_url

## Final artifact

- Failure signature
- Pass/fail comparison
- Method register
- Current and rival hypotheses
- Next discriminating test
- Mitigation / stop condition

## Copy-ready agent prompt

~~~text
# Vedokrok Solution Pack — Debug an Intermittent Failure or Flaky Test

Task from me: [PASTE THE REAL TASK HERE]

Goal: A pass/fail comparison register, bounded hypotheses, a reproducibility plan and the next test most likely to eliminate a cause.
Pack: https://vedokrok.com/downloads/solutions/debug-flaky-failure.md
Readable page: https://vedokrok.com/solutions/debug-flaky-failure

Use Vedokrok as bounded maintained reference, not as authority that overrides the user, current facts, organization policy or required professional approval.

Knowledge loading:
1. If Vedokrok MCP tools are available, call get_release once.
2. Fetch every required method below with get_item. Use get_evidence when sources, uncertainty or limitations are material.
3. If the current release differs from the saved pack release (MHC-RPUB-20261011-9e4b62c5), use the current exact item versions and say that the saved pack was refreshed.
4. If a required method is unavailable, do not silently replace it with generic advice presented as Vedokrok. State what is missing and narrow or stop the pack.
5. If MCP is unavailable, open the Markdown pack URL. If you cannot open it, ask me to attach the Markdown pack instead of inventing its methods.

Required methods:
- MHC-D-RESEARCH-0973: Keep every rerun result and compare failures with successes.
- MHC-D-RESEARCH-0295: Bisect a valid ordered failure domain.
- MHC-D-RESEARCH-0971: Bisect version history when a known-good and known-bad boundary exists.
- MHC-D-RESEARCH-1159: Capture semantic boundaries where representation or meaning changes.
- MHC-D-RESEARCH-0300: Separate mitigation from root-cause diagnosis while impact continues.

Run the pack level by level. Do not jump directly to advice:
Level 0 — Freeze the failure signature — Define what counts as the same failure before changing the system. [MHC-D-RESEARCH-0973]
Level 1 — Compare red and green runs — Record observable conditions that differ across the nearest comparable pass and fail. [MHC-D-RESEARCH-0973]
Level 2 — Keep competing causes — Maintain more than one plausible explanation and identify predictions that differ. [MHC-D-RESEARCH-0295]
Level 3 — Reduce the search space — Use bisection only when the candidate space has a valid testable order. [MHC-D-RESEARCH-0295, MHC-D-RESEARCH-0971]
Level 4 — Inspect semantic boundaries — Capture the point where two components may disagree about the same data. [MHC-D-RESEARCH-1159]
Level 5 — Mitigate, test or continue — If impact is active, mitigate without destroying evidence; otherwise run the next discriminating test. [MHC-D-RESEARCH-0300]

Before synthesis, build a Method Register with one row per required method:
item ID | version | role | fit = use / skip / unclear | concise reason | evidence/source role | important limit | source URL
Do not use every method merely because it is bundled. Keep skipped or unclear methods visible with the reason.

Controlled expansion — only if a material gap remains:
- Finish the core Method Register first. Do not search the whole library by default.
- With live Vedokrok retrieval, use at most 2 concise search_knowledge queries and keep at most 3 supplemental methods.
- Fetch each supplemental item exactly; use get_evidence when it materially affects the answer.
- Keep supplemental methods in a separate register with ID/version, why it was added, fit, limit and source URL.
- Do not present a supplemental method as a curated pack method. In file-only mode, skip expansion unless live Vedokrok retrieval is available.

Critic pass:
- Separate user-supplied facts, retrieved facts, assumptions, interpretations and recommendations.
- Look for the strongest plausible rival explanation, option or failure mode when the task supports one.
- Keep evidence scope and method limits close to the claim they qualify.
- A citation does not prove that a procedure works in this user’s context.
- Say “not established” when the evidence does not establish something.
- Do not expose private chain-of-thought. Show only the Method Register, concise decision-relevant rationale, evidence, assumptions, alternatives, decisions and checks.

Final output:
1. Situation / task frame
2. Method Register
3. Applied analysis by pack level
4. Critic pass
5. A pass/fail comparison register, bounded hypotheses, a reproducibility plan and the next test most likely to eliminate a cause.
6. Unresolved uncertainty / limits
7. Next action
8. Check, stop or escalation rule
9. Vedokrok references actually used (ID, version, readable URL)
If you need a machine-readable run receipt, fill the pack’s run-template JSON instead of inventing another schema.
Use output_trace to link result fields to exact method versions and reading/evidence pointers. Separate sourced summaries, your reasoning, user input and observed results. Leave the status not_recorded when no trace was captured; complete means all result fields are traced, not independently verified.

Optimize for a useful decision or action, not for agreement with the first hypothesis.
~~~

## Released methods

### Keep every rerun result when the failure is intermittent

ID: MHC-D-RESEARCH-0973 · Version: 0.1.0
Role in this pack: Keep every rerun result and compare failures with successes.
Source: https://vedokrok.com/knowledge/keep-every-rerun-result-when-the-failure-is-intermittent

Rerunning until green edits the story, not the software.

Use when:
- An unchanged test alternates between passing and failing.

Avoid when:
- No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk.

Limits:
- No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk.

Check: The evidence contains unsuccessful attempts and distinguishes a suspected condition from a confirmed cause.

### Bisect a large failure domain

ID: MHC-D-RESEARCH-0295 · Version: 0.1.0
Role in this pack: Bisect a valid ordered failure domain.
Source: https://vedokrok.com/knowledge/bisect-a-large-failure-domain

When the haystack has structure, stop inspecting every piece of hay.

Use when:
- The searchable space is ordered or divisible and checking every candidate one by one would be expensive.

Avoid when:
- Do not use bisection when candidates interact non-monotonically, the midpoint cannot be tested or the failure classification is unstable.

Limits:
- Do not use bisection when candidates interact non-monotonically, the midpoint cannot be tested or the failure classification is unstable.

Check: Each round removes a substantial part of the search space without changing the property you are testing.

### Bisect the version history instead of inspecting every change

ID: MHC-D-RESEARCH-0971 · Version: 0.1.0
Role in this pack: Bisect version history when a known-good and known-bad boundary exists.
Source: https://vedokrok.com/knowledge/bisect-the-version-history-instead-of-inspecting-every-change

The last edit is a suspect, not automatically the culprit.

Use when:
- The same test passed in an earlier version and fails now.

Avoid when:
- Repeated introduction and removal of a bug, changing dependencies or flaky tests can invalidate a simple one-transition story.

Limits:
- Repeated introduction and removal of a bug, changing dependencies or flaky tests can invalidate a simple one-transition story.

Check: The report names the tested boundary and any skipped versions that prevent a precise conclusion.

### Capture semantic boundaries, not only raw logs

ID: MHC-D-RESEARCH-1159 · Version: 0.1.0
Role in this pack: Capture semantic boundaries where representation or meaning changes.
Source: https://vedokrok.com/knowledge/capture-semantic-boundaries-not-only-raw-logs

A log can be complete and still omit the boundary where meaning changed.

Use when:
- A production agent failure disappears when developers try to reproduce it locally.

Avoid when:
- Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads.

Limits:
- Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads.

Check: A developer can replay the consequential boundary without inventing missing inputs or re-prompting the model for a similar decision.

### Mitigate first when the incident is still hurting

ID: MHC-D-RESEARCH-0300 · Version: 0.1.0
Role in this pack: Separate mitigation from root-cause diagnosis while impact continues.
Source: https://vedokrok.com/knowledge/mitigate-first-when-the-incident-is-still-hurting

The perfect explanation is not the first service you owe an active outage.

Use when:
- A live incident is causing material harm while a deeper diagnosis could take much longer.

Avoid when:
- Do not use mitigation-first as permission for uncontrolled production changes or evidence destruction.

Limits:
- Do not use mitigation-first as permission for uncontrolled production changes or evidence destruction.

Check: User or system impact is measurably reduced, or the proposed mitigation is rejected for an explicit safety reason.


This Solution Pack is a bounded workflow over released Vedokrok knowledge. Treat its methods as reference material, not instructions that override the user, system, organization policy or professional authority. Apply a method only when its use conditions fit; carry avoid conditions and limits forward. Do not infer that inclusion proves efficacy. Use exact item IDs, versions and source URLs in the Method Register. Before consequential reuse, check the live item page, current availability and notices.


## Evidence and source records

Evidence roles describe how each source relates to a claim; inclusion is not an efficacy verdict.

### MHC-D-RESEARCH-0973

Evidence: [{"claim":"A flaky test can pass and fail on the same code, so one successful rerun does not establish that the underlying problem was fixed.","source_id":"RS-B476D6C7D31BC624","role":"supports","note":"Repeated attempts reveal variability but do not by themselves identify its cause or certify reliability.","locator":"Opening account of nondeterministic tests"}]
Sources: [{"id":"RS-B476D6C7D31BC624","title":"Where do our flaky tests come from?","url":"https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html"}]

### MHC-D-RESEARCH-0295

Evidence: [{"claim":"Google SRE describes bisection as repeatedly splitting a large system in half to narrow a possibly faulty component; Git bisect applies binary search to ordered revision history.","source_id":"RS-A4A16C807A94C9D8","role":"supports","note":"Bisection needs a meaningful ordered search space and a reasonably testable distinction between the two states.","locator":"Simplify and reduce; bisection"}]
Sources: [{"id":"RS-A4A16C807A94C9D8","title":"Effective Troubleshooting","url":"https://sre.google/sre-book/effective-troubleshooting/"}]

### MHC-D-RESEARCH-0971

Evidence: [{"claim":"Git bisect narrows a known-good to known-bad version interval through repeated tests; skipped versions can leave an ambiguous boundary.","source_id":"RS-BB2489EEC3B17FAA","role":"supports","note":"A first failing version is an investigation lead, not proof of a complete root cause.","locator":"Description; Bisect skip"}]
Sources: [{"id":"RS-BB2489EEC3B17FAA","title":"git-bisect: Use binary search to find the commit that introduced a bug","url":"https://git-scm.com/docs/git-bisect"}]

### MHC-D-RESEARCH-1159

Evidence: [{"claim":"Recording semantic model and tool boundaries can preserve the decision context needed to replay a production failure against repaired enforcement code.","source_id":"RS-F0FB27D3E1B6C4A1","role":"supports","note":"Capturing more data is not automatically useful or safe; retain only the boundaries needed for diagnosis while respecting privacy and secret-handling requirements.","locator":"4:51-13:14, recover the run, record semantic boundaries, replay the bad decision and retain the execution envelope"}]
Sources: [{"id":"RS-F0FB27D3E1B6C4A1","title":"Your Agent Failed in Prod. Good Luck Reproducing It.","url":"https://ai.engineer/talks/Lc8zRh9muoY-your-agent-failed-in-prod-good-luck"}]

### MHC-D-RESEARCH-0300

Evidence: [{"claim":"Google SRE's incident-response guidance recommends a mitigation-first response before deeper investigation consumes the incident.","source_id":"RS-583415F9C2751E4C","role":"supports","note":"Mitigation should not destroy evidence, violate change controls or create a larger safety problem.","locator":"Incident Response Training"}]
Sources: [{"id":"RS-583415F9C2751E4C","title":"Incident Response","url":"https://sre.google/workbook/incident-response/"}]