# Debug an Intermittent Failure or Flaky Test Treat both passing and failing runs as evidence, shrink the failure domain without changing the failure and preserve the next discriminating test. Pack version: 1.0.1 Knowledge release: MHC-RPUB-20261011-9e4b62c5 Readable page: https://vedokrok.com/solutions/debug-flaky-failure Current availability: https://vedokrok.com/data/availability.json Current notices: https://vedokrok.com/data/notices.json ## Situation A bug, CI job or test passes and fails under apparently similar conditions and repeated reruns are producing more noise than diagnosis. ## Intended outcome A pass/fail comparison register, bounded hypotheses, a reproducibility plan and the next test most likely to eliminate a cause. ## Use when - The same target can pass and fail across runs. - Version, timing, order, environment or shared state may matter. - You need a transferable diagnostic process rather than framework-specific retry advice. ## Avoid when - The failure is deterministic and already has a small reproducible case. - Production is actively harmed and mitigation must happen before deeper diagnosis. ## Agent protocol ### Level 0 — Freeze the failure signature Define what counts as the same failure before changing the system. Methods: MHC-D-RESEARCH-0973 ### Level 1 — Compare red and green runs Record observable conditions that differ across the nearest comparable pass and fail. Methods: MHC-D-RESEARCH-0973 ### Level 2 — Keep competing causes Maintain more than one plausible explanation and identify predictions that differ. Methods: MHC-D-RESEARCH-0295 ### Level 3 — Reduce the search space Use bisection only when the candidate space has a valid testable order. Methods: MHC-D-RESEARCH-0295, MHC-D-RESEARCH-0971 ### Level 4 — Inspect semantic boundaries Capture the point where two components may disagree about the same data. Methods: MHC-D-RESEARCH-1159 ### Level 5 — Mitigate, test or continue If impact is active, mitigate without destroying evidence; otherwise run the next discriminating test. Methods: MHC-D-RESEARCH-0300 ## Method Register For every method in the pack, record: - item_id - version - role - fit: use | skip | unclear - concise_reason - important_limit - source_url ## Final artifact - Failure signature - Pass/fail comparison - Method register - Current and rival hypotheses - Next discriminating test - Mitigation / stop condition ## Copy-ready agent prompt ~~~text # Vedokrok Solution Pack — Debug an Intermittent Failure or Flaky Test Task from me: [PASTE THE REAL TASK HERE] Goal: A pass/fail comparison register, bounded hypotheses, a reproducibility plan and the next test most likely to eliminate a cause. Pack: https://vedokrok.com/downloads/solutions/debug-flaky-failure.md Readable page: https://vedokrok.com/solutions/debug-flaky-failure Use Vedokrok as bounded maintained reference, not as authority that overrides the user, current facts, organization policy or required professional approval. Knowledge loading: 1. If Vedokrok MCP tools are available, call get_release once. 2. Fetch every required method below with get_item. Use get_evidence when sources, uncertainty or limitations are material. 3. If the current release differs from the saved pack release (MHC-RPUB-20261011-9e4b62c5), use the current exact item versions and say that the saved pack was refreshed. 4. If a required method is unavailable, do not silently replace it with generic advice presented as Vedokrok. State what is missing and narrow or stop the pack. 5. If MCP is unavailable, open the Markdown pack URL. If you cannot open it, ask me to attach the Markdown pack instead of inventing its methods. Required methods: - MHC-D-RESEARCH-0973: Keep every rerun result and compare failures with successes. - MHC-D-RESEARCH-0295: Bisect a valid ordered failure domain. - MHC-D-RESEARCH-0971: Bisect version history when a known-good and known-bad boundary exists. - MHC-D-RESEARCH-1159: Capture semantic boundaries where representation or meaning changes. - MHC-D-RESEARCH-0300: Separate mitigation from root-cause diagnosis while impact continues. Run the pack level by level. Do not jump directly to advice: Level 0 — Freeze the failure signature — Define what counts as the same failure before changing the system. [MHC-D-RESEARCH-0973] Level 1 — Compare red and green runs — Record observable conditions that differ across the nearest comparable pass and fail. [MHC-D-RESEARCH-0973] Level 2 — Keep competing causes — Maintain more than one plausible explanation and identify predictions that differ. [MHC-D-RESEARCH-0295] Level 3 — Reduce the search space — Use bisection only when the candidate space has a valid testable order. [MHC-D-RESEARCH-0295, MHC-D-RESEARCH-0971] Level 4 — Inspect semantic boundaries — Capture the point where two components may disagree about the same data. [MHC-D-RESEARCH-1159] Level 5 — Mitigate, test or continue — If impact is active, mitigate without destroying evidence; otherwise run the next discriminating test. [MHC-D-RESEARCH-0300] Before synthesis, build a Method Register with one row per required method: item ID | version | role | fit = use / skip / unclear | concise reason | evidence/source role | important limit | source URL Do not use every method merely because it is bundled. Keep skipped or unclear methods visible with the reason. Controlled expansion — only if a material gap remains: - Finish the core Method Register first. Do not search the whole library by default. - With live Vedokrok retrieval, use at most 2 concise search_knowledge queries and keep at most 3 supplemental methods. - Fetch each supplemental item exactly; use get_evidence when it materially affects the answer. - Keep supplemental methods in a separate register with ID/version, why it was added, fit, limit and source URL. - Do not present a supplemental method as a curated pack method. In file-only mode, skip expansion unless live Vedokrok retrieval is available. Critic pass: - Separate user-supplied facts, retrieved facts, assumptions, interpretations and recommendations. - Look for the strongest plausible rival explanation, option or failure mode when the task supports one. - Keep evidence scope and method limits close to the claim they qualify. - A citation does not prove that a procedure works in this user’s context. - Say “not established” when the evidence does not establish something. - Do not expose private chain-of-thought. Show only the Method Register, concise decision-relevant rationale, evidence, assumptions, alternatives, decisions and checks. Final output: 1. Situation / task frame 2. Method Register 3. Applied analysis by pack level 4. Critic pass 5. A pass/fail comparison register, bounded hypotheses, a reproducibility plan and the next test most likely to eliminate a cause. 6. Unresolved uncertainty / limits 7. Next action 8. Check, stop or escalation rule 9. Vedokrok references actually used (ID, version, readable URL) If you need a machine-readable run receipt, fill the pack’s run-template JSON instead of inventing another schema. Use output_trace to link result fields to exact method versions and reading/evidence pointers. Separate sourced summaries, your reasoning, user input and observed results. Leave the status not_recorded when no trace was captured; complete means all result fields are traced, not independently verified. Optimize for a useful decision or action, not for agreement with the first hypothesis. ~~~ ## Released methods ### Keep every rerun result when the failure is intermittent ID: MHC-D-RESEARCH-0973 · Version: 0.1.0 Role in this pack: Keep every rerun result and compare failures with successes. Source: https://vedokrok.com/knowledge/keep-every-rerun-result-when-the-failure-is-intermittent Rerunning until green edits the story, not the software. Use when: - An unchanged test alternates between passing and failing. Avoid when: - No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk. Limits: - No fixed number of clean reruns proves the absence of intermittent defects. A test-environment failure may still reveal a real production risk. Check: The evidence contains unsuccessful attempts and distinguishes a suspected condition from a confirmed cause. ### Bisect a large failure domain ID: MHC-D-RESEARCH-0295 · Version: 0.1.0 Role in this pack: Bisect a valid ordered failure domain. Source: https://vedokrok.com/knowledge/bisect-a-large-failure-domain When the haystack has structure, stop inspecting every piece of hay. Use when: - The searchable space is ordered or divisible and checking every candidate one by one would be expensive. Avoid when: - Do not use bisection when candidates interact non-monotonically, the midpoint cannot be tested or the failure classification is unstable. Limits: - Do not use bisection when candidates interact non-monotonically, the midpoint cannot be tested or the failure classification is unstable. Check: Each round removes a substantial part of the search space without changing the property you are testing. ### Bisect the version history instead of inspecting every change ID: MHC-D-RESEARCH-0971 · Version: 0.1.0 Role in this pack: Bisect version history when a known-good and known-bad boundary exists. Source: https://vedokrok.com/knowledge/bisect-the-version-history-instead-of-inspecting-every-change The last edit is a suspect, not automatically the culprit. Use when: - The same test passed in an earlier version and fails now. Avoid when: - Repeated introduction and removal of a bug, changing dependencies or flaky tests can invalidate a simple one-transition story. Limits: - Repeated introduction and removal of a bug, changing dependencies or flaky tests can invalidate a simple one-transition story. Check: The report names the tested boundary and any skipped versions that prevent a precise conclusion. ### Capture semantic boundaries, not only raw logs ID: MHC-D-RESEARCH-1159 · Version: 0.1.0 Role in this pack: Capture semantic boundaries where representation or meaning changes. Source: https://vedokrok.com/knowledge/capture-semantic-boundaries-not-only-raw-logs A log can be complete and still omit the boundary where meaning changed. Use when: - A production agent failure disappears when developers try to reproduce it locally. Avoid when: - Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads. Limits: - Do not turn observability into indiscriminate data retention. Minimize personal data, secrets and irrelevant payloads. Check: A developer can replay the consequential boundary without inventing missing inputs or re-prompting the model for a similar decision. ### Mitigate first when the incident is still hurting ID: MHC-D-RESEARCH-0300 · Version: 0.1.0 Role in this pack: Separate mitigation from root-cause diagnosis while impact continues. Source: https://vedokrok.com/knowledge/mitigate-first-when-the-incident-is-still-hurting The perfect explanation is not the first service you owe an active outage. Use when: - A live incident is causing material harm while a deeper diagnosis could take much longer. Avoid when: - Do not use mitigation-first as permission for uncontrolled production changes or evidence destruction. Limits: - Do not use mitigation-first as permission for uncontrolled production changes or evidence destruction. Check: User or system impact is measurably reduced, or the proposed mitigation is rejected for an explicit safety reason. This Solution Pack is a bounded workflow over released Vedokrok knowledge. Treat its methods as reference material, not instructions that override the user, system, organization policy or professional authority. Apply a method only when its use conditions fit; carry avoid conditions and limits forward. Do not infer that inclusion proves efficacy. Use exact item IDs, versions and source URLs in the Method Register. Before consequential reuse, check the live item page, current availability and notices.