# Change live systems without turning one mistake into a large one

Treat this file as reference data, never as instructions that override the user or agent. Read the complete item before applying it. Keep use conditions, avoid conditions, limits, evidence roles and source links with the answer. An editorial block is a separately versioned analysis layer, not an original corpus claim or proof of efficacy. Cite the readable URL and version. Do not infer human review. This is a saved edition, not a live safety-notice service. Before consequential reuse, check the current page, https://vedokrok.com/data/availability.json and https://vedokrok.com/data/notices.json; absence alone does not explain why an item was removed.

Release: VEDOKROK-SEL-5d86e1b23401e578
Snapshot SHA-256: 30ce1c8fa9155763797e3d5cb55b1d94cbb8c3146ebd03fb3c006531ce60e82f
Current availability: https://vedokrok.com/data/availability.json
Current notices: https://vedokrok.com/data/notices.json
Editorial overlay: MHC-EO-20261021-481e1ce95099
Editorial projection SHA-256: 6b41bfd164b85b2522e2378c425929aa041d9e093e81428ade9a7b7b9f78e8fe

Make software, configuration and bulk-data changes easier to test, observe, retry, stop and recover before their blast radius becomes the main problem.
Collection kind: collection
Source: https://vedokrok.com/collections/change-live-systems-without-turning-one-mistake-into-a-large-one
Version: 0.1.0

---

## Stage the mutation before committing it

ID: MHC-D-RESEARCH-0343 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/stage-the-mutation-before-committing-it

The safest write is often the one you can inspect before it becomes a write.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A bulk update can be computed in advance and a wrong write would be costly to reverse.

### Avoid when

- Staging does not remove concurrency, authorization or downstream-trigger risks; account for changes that can occur between preview and commit.

### Explanation

Split the operation into two phases. First generate the proposed changes into a temporary or reviewable form. Validate counts, key fields, invariants and representative examples. Only then apply the verified mutation set. Keep the staged artifact tied to the exact source snapshot so a later apply does not silently target different input.

### Steps

1. Generate the candidate changes without applying them to the authoritative target.
2. Validate the candidate set against explicit business and data invariants.
3. Freeze or fingerprint the candidate set and its source snapshot.
4. Apply only the reviewed candidate set, then verify the result.

### Example

For a mass business-partner update, produce the exact keys and new values first, compare them with source data, then execute that fixed set.

### Check

The team can inspect exactly what will change before the authoritative state is modified.

### Limits

- Staging does not remove concurrency, authorization or downstream-trigger risks; account for changes that can occur between preview and commit.

### Evidence and sources

- supports: Google SRE describes two-phase mutation as storing proposed mutations temporarily, validating them, and applying them only after verification. — RS-5D6BA8DCC62D671F. Two-phase mutation adds complexity and may not fit every storage system or transaction model. (Idempotent and Two-Phase Mutations)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/reconcile-what-changed-after-a-bulk-write

---

## Canary the batch before scaling it

ID: MHC-D-RESEARCH-0344 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/canary-the-batch-before-scaling-it

Do not make the millionth write your first realistic test.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A change must touch many records, jobs, instances or users and real conditions matter.

### Avoid when

- A canary cannot expose failures that only appear at scale, in rare data or after long delays; design later gates for those risks.

### Explanation

Apply the real change to a deliberately small, representative slice first. Observe the same correctness and health signals that matter at full scale, including delayed processing where relevant. Expand only when the canary remains healthy for the required observation window. Choose the slice to reveal risk, not merely to make the success rate look good.

### Steps

1. The full rollout waits for evidence from a bounded real-world slice rather than only preproduction confidence.

### Example

Update 50 representative records across several data shapes before starting the remaining 50,000.

### Check

The full rollout waits for evidence from a bounded real-world slice rather than only preproduction confidence.

### Limits

- A canary cannot expose failures that only appear at scale, in rare data or after long delays; design later gates for those risks.

### Evidence and sources

- supports: Google SRE recommends canarying a data pipeline on a subset of real production-shaped data or in dry-run mode before full rollout. — RS-5D6BA8DCC62D671F. A representative canary lowers exposure but can still miss rare data shapes or delayed effects. (Canarying data pipelines)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/

### Vedokrok editorial review

Canary the batch before scaling it

A small first batch needs a decision rule. Define what would let it grow and what would stop it before making the first live change.

#### Choose the slice and the gate

Write down the records or users included, the correctness and health signals, the stop condition, the observation window and the next expansion step. Include important data shapes rather than selecting only easy cases. For example, a customer update can cover both ordinary records and records with delayed replication. Keep the canary results visible separately from the rest of the system.

#### Wait for the relevant result

Apply the bounded change and check it through the processing cycle that matters. Expand only when the declared checks pass; stop and investigate when they fail or remain inconclusive. A dry run can inspect proposed output, but it does not exercise every live-write effect. A healthy small slice also leaves scale-only, rare-input and later failures unresolved.

**Editorial takeaway:** Before expansion, show which checks passed, which risks the slice did not cover and who can stop the next stage. A small batch without that decision is only a small batch.

#### Editorial evidence

- supports: Google SRE defines canarying as bounded deployment plus evaluation integrated into the rollout decision. — RS-5620B924E8D134E5. Selected official passages read on 2026-10-04. The five-part gate is an editorial application, not a tested universal canary size or duration. (Definition; Requirements of a Canary Process; Choosing a Canary Population and Duration)
- supports: Pipeline canary verification may need a complete processing cycle and comparison of candidate and live output. — RS-5D6BA8DCC62D671F. Selected official passages read on 2026-10-04. The chapter permits production-data dry runs that skip storage writes; these are not equivalent to live mutation. (Map Your Development Lifecycle > Canarying)
- limits: A canary needs representative traffic, suitable metrics and attention to shared failure domains. — RS-5620B924E8D134E5. Selected official passages read on 2026-10-04. No guarantee of finding every defect or independent causal attribution is made. (Choosing a Canary Population and Duration; Dependencies and Isolation)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/
- RS-5620B924E8D134E5: Canarying Releases — https://sre.google/workbook/canarying-releases/

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/dry-run-the-real-shaped-input-without-the-real-write

---

## Write the rollback trigger before deployment

ID: MHC-D-RESEARCH-0345 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/write-the-rollback-trigger-before-deployment

'We will know when to roll back' is not a rollback condition.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A change can degrade service and people may hesitate to roll back once effort has been invested.

### Avoid when

- Thresholds can be noisy or incomplete; allow human escalation when evidence is serious but the predefined metric misses it.

### Explanation

Before deployment, define observable signals that mean the change should stop or reverse: error rate, failed reconciliation, latency, business invariant, unexpected side effect or another relevant threshold. Also define who can call the rollback. This converts a stressful judgment into a prepared decision and reduces the temptation to wait for certainty while impact grows.

### Checklist

- The rollback condition is observable during the rollout.
- The signal is tied to the change's real failure modes.
- An owner has authority to stop or reverse the rollout.
- The condition distinguishes rollback from a tolerable transient effect.
- The recovery path is known before the trigger fires.

### Example

Rollback if more than 0.5% of updated records fail the post-write invariant or if replication backlog exceeds the agreed threshold for ten minutes.

### Check

A responder can decide whether the trigger fired without inventing a new rule during the incident.

### Limits

- Thresholds can be noisy or incomplete; allow human escalation when evidence is serious but the predefined metric misses it.

### Evidence and sources

- supports: AWS recommends documenting rollback criteria and the rollback or fix-forward plan before deploying a change. — RS-0B17D9040762A762. Rollback is not always safer than fixing forward, particularly for irreversible data migrations; the strategy must match the change. (Implementation guidance)
- RS-0B17D9040762A762: OPS06-BP01 Plan for unsuccessful changes — https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_mit_deploy_risks_plan_for_unsucessful_changes.html

### Vedokrok editorial review

Define the stop signal and recovery choice

Before deployment, decide what failure will stop the rollout and what happens next. Stopping new changes and reversing completed changes are different decisions.

#### Make the trigger observable

Name the signal, its threshold or clear failure condition, the observation period and the person or automation allowed to act. For a data migration, a broken required relationship can trigger a halt while the affected records are examined. Choose values for the actual failure mode; an example percentage is not a general safety threshold.

#### Choose a recovery path that fits

State when to restore the previous version and when to fix the current state. A rollback must remain compatible with changed data and dependencies. If reversing the change is unsafe or incomplete, specify containment and a tested fix-forward plan. Keep an escalation route for serious evidence that the chosen metrics miss.

**Editorial takeaway:** Walk through one simulated breach: can the responder identify the trigger, stop further exposure and choose a viable recovery path without inventing a rule?

#### Editorial evidence

- supports: AWS calls for documented, tested recovery plans and explicit requirements for rollback. — RS-0B17D9040762A762. Selected official passages read on 2026-10-04. Signal selection and the simulated breach are editorial review actions, not experimentally validated thresholds. (Implementation guidance; Implementation steps 1 and 5)
- limits: AWS explicitly allows fix-forward circumstances and discusses changes that cannot be rolled back. — RS-0B17D9040762A762. Selected official passages read on 2026-10-04. A failure signal does not establish that reversal is the safest recovery action. (Implementation guidance; Implementation steps 1.1 and 4)
- RS-0B17D9040762A762: OPS06-BP01 Plan for unsuccessful changes — https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_mit_deploy_risks_plan_for_unsucessful_changes.html

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (use_before): https://vedokrok.com/knowledge/make-the-rollout-stop-itself-on-a-known-bad-signal

---

## Test the rollback before you need it

ID: MHC-D-RESEARCH-0346 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/test-the-rollback-before-you-need-it

A rollback plan that exists only on paper is a hypothesis.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A release plan says 'rollback available' but nobody has recently executed the recovery path.

### Avoid when

- A rehearsal cannot reproduce every live dependency; data migrations and external side effects may require a different recovery design.

### Explanation

Exercise the rollback in a safe environment using the same artifacts, permissions and sequence intended for production. Verify that the old version or state actually returns, dependencies remain compatible and operators know what cannot be reversed. Record the measured recovery time and any manual step that still depends on memory.

### Steps

1. Execute the documented rollback with production-like artifacts and permissions.
2. Verify the known-good state after rollback, not only that the command completed.
3. Record irreversible parts and required fix-forward steps.
4. Update the runbook when the rehearsal exposes missing access, timing or dependency assumptions.

### Example

Before a high-risk configuration rollout, deploy the new version in staging, roll it back using the production runbook and verify the old behavior.

### Check

Someone can demonstrate the recovery path and its limitations with evidence from a recent rehearsal.

### Limits

- A rehearsal cannot reproduce every live dependency; data migrations and external side effects may require a different recovery design.

### Evidence and sources

- supports: AWS recommends validating rollback procedures before live deployment rather than discovering the recovery path during failure. — RS-0B17D9040762A762. A nonproduction rollback test may not reproduce every production dependency or data state. (Documented and tested recovery plan)
- RS-0B17D9040762A762: OPS06-BP01 Plan for unsuccessful changes — https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_mit_deploy_risks_plan_for_unsucessful_changes.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/write-the-rollback-trigger-before-deployment

---

## Give a retryable mutation one stable operation ID

ID: MHC-D-RESEARCH-0347 · Version: 0.1.0 · Kind: pattern
Source: https://vedokrok.com/knowledge/give-a-retryable-mutation-one-stable-operation-id

A retry should mean 'finish this operation,' not 'please create another one.'

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A create or update request may be retried after a timeout, dropped connection or transient service error.

### Avoid when

- Do not assume idempotency exists where the service does not guarantee it, and do not reuse one key for materially different requests.

### Explanation

Attach a stable idempotency or request key to one logical mutation and reuse it for retries of that same intent. Generate a new key only for a genuinely new operation. Store enough context to recognize ambiguous responses and reconcile them. This is especially valuable for payments, job creation, provisioning and bulk actions where duplicates are expensive.

### Recognition

A retry should mean 'finish this operation,' not 'please create another one.'

### Example

If a create-order call times out, retry with the same operation ID rather than issuing a second independent create request.

### Check

Repeated delivery of the same logical request cannot silently create multiple intended-once effects under the API contract.

### Limits

- Do not assume idempotency exists where the service does not guarantee it, and do not reuse one key for materially different requests.

### Evidence and sources

- supports: Amazon and Stripe document idempotency keys or request identifiers as a way to safely retry a mutation without unintentionally performing the same logical operation twice. — RS-56B6FA675EBAA1EF. Idempotency semantics depend on the API contract and retention window; callers must not assume every endpoint supports them. (Client request identifiers and retry safety)
- RS-56B6FA675EBAA1EF: Making retries safe with idempotent APIs — https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

### Vedokrok editorial review

Keep the operation key within its contract

Reuse one key for retries of the same intended change. First check what the API promises: a repeated key only helps when the service still recognizes it.

#### Keep identity and intent together

Store the key with the request parameters and result or unresolved status. Retry with that same identity after an ambiguous response; do not silently generate a second create request. Give a genuinely new operation a new key, and never reuse a key for changed parameters. Check which endpoint, account or other scope the service uses to recognize duplicates.

#### Check the lifetime before replay

Find the service's retention rule and what happens after it expires. Stripe, for example, may treat reuse of a pruned key as a new request; it also returns saved failures for recognized retries. A stable key therefore does not promise eventual success or indefinite duplicate protection. If recognition is uncertain, reconcile the original outcome before deciding whether to issue another mutation.

**Editorial takeaway:** For one timed-out request, locate its saved key and parameters, verify the service's recognition window, and choose a same-operation retry or an authoritative status check.

#### Editorial evidence

- supports: Amazon describes caller-provided request identifiers and parameter checks to distinguish a retry from a different intent. — RS-56B6FA675EBAA1EF. Selected official passages read on 2026-10-04. This is an API contract pattern, not a guarantee for endpoints that do not implement it. (Reducing client complexity; Same client request ID, different intent)
- limits: Retention requirements for recognizing earlier requests vary by service and resource. — RS-56B6FA675EBAA1EF. Selected official passages read on 2026-10-04. The card does not transfer EC2 retention semantics to other APIs. (Late arriving requests and the life span of unique client request identifiers)
- limits: Stripe returns a stored result for a recognized key, including failures, and may create a new request after the original key is pruned. — RS-BB8881D82498DEB5. Official API page selected passages read on 2026-10-04. Stripe-specific behavior is an example; readers must inspect the contract of their own endpoint. (Idempotent requests: saved results, key pruning and parameter comparison)
- RS-56B6FA675EBAA1EF: Making retries safe with idempotent APIs — https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/
- RS-BB8881D82498DEB5: Idempotent requests — https://docs.stripe.com/api/idempotent_requests

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/treat-a-timeout-as-an-unknown-outcome

---

## Retry slower, with randomness, and stop

ID: MHC-D-RESEARCH-0348 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/retry-slower-with-randomness-and-stop

A retry storm is an outage trying to help.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A remote dependency fails transiently and many clients may retry at the same time.

### Avoid when

- Backoff reduces amplification but may increase completion latency; tune it to the service and user-facing deadline.

### Explanation

Use a bounded retry policy: wait longer between attempts, add jitter so clients do not synchronize, and stop after a defined budget. Combine this with timeouts and idempotency for mutating operations. Retry only errors that can plausibly recover; validation failures and permanent authorization errors need a different response.

### Steps

1. The retry policy cannot continue indefinitely or synchronize a fleet into repeated load spikes.

### Example

A worker retries a throttled API after randomized exponential delays instead of immediately looping hundreds of requests.

### Check

The retry policy cannot continue indefinitely or synchronize a fleet into repeated load spikes.

### Limits

- Backoff reduces amplification but may increase completion latency; tune it to the service and user-facing deadline.

### Evidence and sources

- supports: AWS reliability guidance recommends limiting retries and using exponential backoff with jitter rather than retrying immediately and indefinitely. — RS-3C71607C882BB63A. Retries are appropriate mainly for transient failures and can worsen overload when used without budgets, timeouts and idempotency. (Implementation guidance)
- RS-3C71607C882BB63A: REL05-BP03 Control and limit retry calls — https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_limit_retries.html

### Vedokrok editorial review

Give retries a budget and one owner

Random delays spread retries; they do not make retry traffic harmless. Bound the work and check which layer is already retrying before adding another loop.

#### Write a complete retry policy

List recoverable errors, the maximum attempts, an overall time budget, growing delays with random variation, and the action after the budget runs out. Leave room for the caller's deadline. Check SDK and application settings so nested loops do not multiply calls. For a write, verify the API's duplicate-handling contract and preserve the operation identity.

#### Test the failure path

In a controlled test, make the dependency fail and record actual attempt times and counts. Check that retries spread out, stop at the budget and report a useful failure. Then try a permanent validation or permission error: it should take the defined error path rather than wait through the transient-failure policy. Randomization can still produce coincident attempts; watch total retry load as well as timing.

**Editorial takeaway:** Count the downstream calls made by one failed user request, including hidden SDK attempts. If that count or total elapsed time exceeds the policy, fix the retry boundary before increasing traffic.

#### Editorial evidence

- supports: AWS recommends exponential backoff, randomized intervals, retry limits and testing retry scenarios. — RS-3C71607C882BB63A. Selected official passages read on 2026-10-04. No fixed attempt count, delay or time budget is claimed to suit every workload. (Implementation guidance; Common anti-patterns)
- limits: Retries at multiple layers can multiply load, and permanent errors or non-idempotent calls need different handling. — RS-3C71607C882BB63A. Selected official passages read on 2026-10-04. The impossibility of coincident random retries is not claimed; the proposed test checks a configured policy, not future outage prevention. (Common anti-patterns: multiple layers, error codes and non-idempotent calls)
- RS-3C71607C882BB63A: REL05-BP03 Control and limit retry calls — https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_limit_retries.html

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/give-a-retryable-mutation-one-stable-operation-id

---

## Treat a timeout as an unknown outcome

ID: MHC-D-RESEARCH-0349 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/treat-a-timeout-as-an-unknown-outcome

No response is not the same thing as no effect.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A mutating remote request times out after it was sent.

### Avoid when

- Some services cannot reveal a definitive state immediately; preserve the uncertainty instead of inventing certainty.

### Explanation

After a timeout, keep both possibilities alive: the server may have applied the change and lost the response, or it may not have applied it. Use the service's request identity, status endpoint, reconciliation data or another authoritative check before deciding to issue a new logical mutation. This avoids duplicate actions and false recovery.

### Example

A payment call times out; query or safely retry the same idempotent operation instead of creating a second payment.

### Check

The recovery procedure does not equate communication failure with application failure.

### Limits

- Some services cannot reveal a definitive state immediately; preserve the uncertainty instead of inventing certainty.

### Evidence and sources

- supports: Amazon's idempotency guidance treats a timed-out mutating request as potentially having completed, which is why a retry needs stable request identity rather than a new logical operation. — RS-56B6FA675EBAA1EF. The exact state must be checked using the specific service's contract; timeout does not mean either success or failure by itself. (Retry scenarios where the response is lost)
- RS-56B6FA675EBAA1EF: Making retries safe with idempotent APIs — https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

No review details supplied.

---

## Reconcile what changed after a bulk write

ID: MHC-D-RESEARCH-0350 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/reconcile-what-changed-after-a-bulk-write

'Job finished' is a process status, not a data-quality result.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A large update reports technical completion but business correctness still needs proof.

### Avoid when

- Reconciliation is only as strong as the intended-state specification; an incorrect source file can reconcile perfectly to the wrong goal.

### Explanation

Compare the authoritative post-write state with the intended mutation set. Check totals, missing keys, unexpected keys, critical fields and a sample of downstream effects. Classify mismatches instead of merely counting them. Preserve the reconciliation artifact so the team can repair the exact exceptions rather than rerun the whole batch blindly.

### Steps

1. Every intended key is accounted for.
2. Unexpected changed keys are detected.
3. Critical before/after fields match the approved mutation.
4. Exceptions are classified with enough detail to repair selectively.
5. Downstream replication or derived state is checked where it is part of the intended outcome.

### Example

After 20,000 customer updates, compare the target table and downstream partner data with the approved input rather than trusting the success message.

### Check

You can state exactly how many records matched, failed, were missed or changed unexpectedly.

### Limits

- Reconciliation is only as strong as the intended-state specification; an incorrect source file can reconcile perfectly to the wrong goal.

### Evidence and sources

- supports: Google SRE recommends comparing canary or dry-run pipeline results with the live pipeline to check for data differences before expansion. — RS-5D6BA8DCC62D671F. Reconciliation only catches differences represented by the comparison logic; missing invariants remain missing. (Verification of canary or preproduction environment)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/

### Vedokrok editorial review

Reconcile the result of a bulk write

A finished job is not an account of what changed. Compare the intended update with authoritative results and leave a repairable list of exceptions.

#### Account for each intended change

Keep the approved keys and expected values. After the write, compare them with the target state and classify matches, missing changes, incorrect values and unexpected changes. Check downstream results when they form part of the intended outcome. For example, a customer update may finish in the main table while replication is still pending; record that state rather than treating it as confirmed failure.

#### Check the comparison itself

This is an editorial workflow for bulk updates. Google SRE discusses expected-versus-actual data checks and comparisons during pipeline canaries; it does not specify this complete post-write repair procedure. Define when the comparison is valid and account for concurrent edits. A perfect match to an incorrect input still produces the wrong business result.

**Editorial takeaway:** Produce a key-level exception list with the expected value, observed state and next check. Resolve those exceptions before choosing a selective repair or another run.

#### Editorial evidence

- supports: Google SRE describes checking expected and actual pipeline output and monitoring discrepancies. — RS-5D6BA8DCC62D671F. Selected official passages read on 2026-10-04. This supports the comparison principle, not every detail of the card's post-write repair workflow. (Define and Measure Service Level Objectives > Data correctness)
- contextualizes: The chapter compares canary or dry-run output with the live pipeline before rollout expansion. — RS-5D6BA8DCC62D671F. Selected official passages read on 2026-10-04. A pre-expansion comparison is context for, rather than direct validation of, a post-write reconciliation procedure. (Map Your Development Lifecycle > Canarying)
- limits: Data correctness depends on the product and may require a separately defined expected result. — RS-5D6BA8DCC62D671F. Selected official passages read on 2026-10-04. The exception-list design is an editorial recommendation; no production success or comprehensive correctness is established. (Define and Measure Service Level Objectives > Data correctness; End-to-end measurement)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/

Editorial version: 1.0.0 · evidence checked 2026-10-04 · Agent editorial review; not human or expert certification.

No review details supplied.

---

## Write invariants before the migration

ID: MHC-D-RESEARCH-0351 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/write-invariants-before-the-migration

A migration can be perfectly formatted and completely wrong.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A transformation can produce syntactically valid output while breaking business rules.

### Avoid when

- Missing invariants create false confidence; involve domain owners where correctness depends on rules the implementation team may not know.

### Explanation

Before running the change, write the properties that must remain true across it: uniqueness, required relationships, totals, allowed status transitions, referential links, permission boundaries or another domain invariant. Validate them on staged output and again after application. Prefer checks that fail loudly over manual visual confidence.

### Checklist

- Each invariant states what must remain true, not how the script happens to work.
- The invariant can be checked automatically or with a defined inspection method.
- Pre-change baseline values are captured when comparison matters.
- Staged output is checked before commit.
- Authoritative post-change state is checked again.

### Example

A business-partner migration may require every active sales-area assignment to keep exactly one valid partner role mapping.

### Check

A technically successful transformation still fails the gate when a declared business invariant is violated.

### Limits

- Missing invariants create false confidence; involve domain owners where correctness depends on rules the implementation team may not know.

### Evidence and sources

- supports: Google SRE's two-phase mutation pattern explicitly separates generating candidate changes from validating their correctness before applying them. — RS-5D6BA8DCC62D671F. Validation rules must encode the invariants that actually matter; a passing schema check can still permit a semantically wrong change. (Two-Phase Mutations)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/stage-the-mutation-before-committing-it

---

## Shrink the change before shrinking the review

ID: MHC-D-RESEARCH-0352 · Version: 0.1.0 · Kind: principle
Source: https://vedokrok.com/knowledge/shrink-the-change-before-shrinking-the-review

A giant change does not become small because the ticket fits on one screen.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A planned deployment or migration bundles many independent changes because one release window is available.

### Avoid when

- Small is not automatically safe; a one-line permission or schema change can still have wide consequences.

### Explanation

Reduce the number of independent things that can fail together. Split changes along reversible, testable boundaries and ship or migrate them in smaller units when dependencies permit. Smaller increments make failures easier to localize, limit impact and simplify rollback. Keep tightly coupled changes together when splitting would create an invalid intermediate state.

### Example

Separate a configuration change from an unrelated data cleanup instead of combining both into one production window.

### Check

A failure implicates a smaller set of changes and a smaller affected scope than the original bundled plan.

### Limits

- Small is not automatically safe; a one-line permission or schema change can still have wide consequences.

### Evidence and sources

- supports: AWS recommends reducing release size to reduce the potential business impact and shorten recovery from unsuccessful changes. — RS-0B17D9040762A762. Smaller changes can still be high risk if they alter a critical invariant or irreversible state. (Reduce the potential impact by making the change smaller)
- RS-0B17D9040762A762: OPS06-BP01 Plan for unsuccessful changes — https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_mit_deploy_risks_plan_for_unsucessful_changes.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/canary-the-batch-before-scaling-it

---

## Let the change bake before stacking the next one

ID: MHC-D-RESEARCH-0353 · Version: 0.1.0 · Kind: heuristic
Source: https://vedokrok.com/knowledge/let-the-change-bake-before-stacking-the-next-one

If you change the experiment before the result arrives, you inherit an explanation problem.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A first change has completed technically but important effects may appear with traffic, queues or delayed jobs.

### Avoid when

- Do not delay urgent remediation just to preserve a clean experiment; incident response can justify immediate follow-on action.

### Explanation

Choose an observation window long enough to expose the failure modes you care about before adding another unrelated change. Watch leading and delayed signals during that time. The point is not idle waiting; it is preserving causal visibility long enough to know whether the current state is healthy.

### Example

After changing replication parallelism, observe at least one full processing cycle before also changing retry settings.

### Check

When a metric moves, the team still has a reasonably interpretable set of recent changes rather than a stack of simultaneous experiments.

### Limits

- Do not delay urgent remediation just to preserve a clean experiment; incident response can justify immediate follow-on action.

### Evidence and sources

- supports: AWS identifies rapid follow-on deployments without sufficient bake time as a deployment anti-pattern. — RS-948FC7FCF8F1FE56. Required observation time depends on traffic, delayed jobs and the failure modes that matter. (Common anti-patterns)
- RS-948FC7FCF8F1FE56: REL08-BP05 Deploy changes with automation — https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_tracking_change_management_automated_changemgmt.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/canary-the-batch-before-scaling-it

---

## Version the configuration that changes behavior

ID: MHC-D-RESEARCH-0354 · Version: 0.1.0 · Kind: checklist
Source: https://vedokrok.com/knowledge/version-the-configuration-that-changes-behavior

If configuration can change production, it deserves a history as serious as code.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Runtime behavior can change without a code deployment because configuration is mutable.

### Avoid when

- Secrets may need separate protected storage; version references rather than exposing sensitive values in ordinary configuration history.

### Explanation

Store important configuration as versioned data with validation, change identity and a known previous version. Record which version each running environment uses. Make promotion and rollback explicit rather than editing values in place without a trace. This turns 'nothing changed in code' from a dead end into an inspectable statement.

### Checklist

- Behavior-affecting configuration has a stable version or change identity.
- Changes pass validation before promotion.
- The running version can be observed for each relevant environment.
- A previous known-good version remains recoverable.
- The configuration history names what changed and why.

### Example

Version an AI agent's tool policy, thresholds and prompt configuration together so a behavior shift can be tied to a concrete configuration snapshot.

### Check

Given an incident timestamp, you can identify the exact configuration version that was active.

### Limits

- Secrets may need separate protected storage; version references rather than exposing sensitive values in ordinary configuration history.

### Evidence and sources

- supports: Current AWS Agentic AI guidance recommends centralized versioned configuration with validation so behavior can be traced and rolled back by version. — RS-2DF67D7590E081DA. Centralization can create its own failure domain; configuration systems still need redundancy and access control. (Desired outcome; configuration versioning and validation)
- RS-2DF67D7590E081DA: AGENTREL08-BP01 Establish consistent configuration management practices — https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentrel08-bp01.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/detect-configuration-drift-instead-of-assuming-sameness

---

## Detect configuration drift instead of assuming sameness

ID: MHC-D-RESEARCH-0355 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/detect-configuration-drift-instead-of-assuming-sameness

'Same deployment' does not guarantee the same state.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Multiple instances, environments or agents are supposed to run the same configuration.

### Avoid when

- Not all divergence is drift; legitimate canaries and experiments need explicit identities so monitoring does not 'repair' them away.

### Explanation

Regularly compare the effective configuration of peers that are expected to match. Flag differences against the approved version and classify them as planned experiment, staged rollout or unintended drift. Correlate behavior anomalies with version differences before treating them as random noise.

### Steps

1. Define which instances or environments are expected to match.
2. Collect the effective configuration version or fingerprint.
3. Compare against the approved target and rollout plan.
4. Classify differences as intentional or unintended.
5. Remediate unintended drift and preserve evidence of the divergence.

### Example

If one agent instance produces different tool behavior, check whether its policy or model configuration differs before debugging the prompt text alone.

### Check

Unexpected behavior can be tested against an objective configuration difference rather than relying on deployment assumptions.

### Limits

- Not all divergence is drift; legitimate canaries and experiments need explicit identities so monitoring does not 'repair' them away.

### Evidence and sources

- supports: Current AWS Agentic AI guidance recommends detecting configuration drift across running instances and correlating behavior with configuration versions. — RS-2DF67D7590E081DA. Not every difference is harmful drift; approved experiments and staged rollouts must remain distinguishable from unintended divergence. (Drift detection)
- RS-2DF67D7590E081DA: AGENTREL08-BP01 Establish consistent configuration management practices — https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentrel08-bp01.html

No review details supplied.
Related (useful_with): https://vedokrok.com/knowledge/start-with-what-changed-not-with-certainty-about-it

---

## Dry-run the real-shaped input without the real write

ID: MHC-D-RESEARCH-0356 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/dry-run-the-real-shaped-input-without-the-real-write

Real data can teach you something before it is allowed to change anything.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- Test fixtures are too clean and production data shape is a major source of risk.

### Avoid when

- A dry run may miss write-side triggers, locking, permissions and downstream side effects; test those separately before assuming full equivalence.

### Explanation

Run the transformation or decision logic on production-shaped input while suppressing authoritative writes. Capture proposed outputs, errors, counts and performance, then compare them with the live or expected path. This is especially useful for data pipelines, migrations and agent workflows whose failures depend on real input diversity.

### Steps

1. The dry run receives representative production-shaped input.
2. Authoritative writes and external side effects are disabled or redirected safely.
3. Outputs and failures are captured for comparison.
4. Differences are reviewed before live mutation is enabled.
5. Sensitive production data remains protected under the same access rules.

### Example

Run a customer transformation against current records into a temporary table and compare the proposed target fields before enabling update mode.

### Check

The team has evidence from realistic input without yet exposing the authoritative target to the change.

### Limits

- A dry run may miss write-side triggers, locking, permissions and downstream side effects; test those separately before assuming full equivalence.

### Evidence and sources

- supports: Google SRE describes dry-run or canary pipeline execution that uses production data while skipping writes to production storage. — RS-5D6BA8DCC62D671F. A no-write path may not exercise all write-side constraints, triggers or downstream effects. (Canarying data pipelines; two-phase mutation)
- RS-5D6BA8DCC62D671F: Improve and Optimize Data Processing Pipelines — https://sre.google/workbook/data-processing/

No review details supplied.

---

## Make the rollout stop itself on a known bad signal

ID: MHC-D-RESEARCH-0357 · Version: 0.1.0 · Kind: protocol
Source: https://vedokrok.com/knowledge/make-the-rollout-stop-itself-on-a-known-bad-signal

A safety metric is more useful when it can pull the brake.

Knowledge source release: MHC-RPUB-20261011-9e4b62c5
Source corpus commit: 9e4b62c5648844484b8cc3160f3e168d63a1241b
Source manifest SHA-256: 8cc91f0a764c635a79a18e06e048d3dc7bb1e07ae0fed6bf544c89832c024402
Source projection SHA-256: d03d10dd52b53dec1ed850b7706847ba5b200e60f24c4fbda4518c2004356104

### Use when

- A progressive deployment can be observed automatically and waiting for a human response would increase impact.

### Avoid when

- Poor thresholds can cause harmful flapping or false rollback; automate only signals whose meaning and recovery action are understood.

### Explanation

Connect a small set of high-confidence failure signals to automatic halt or rollback behavior. Define the condition before rollout and test that the automation acts on the intended version. Keep a human override for ambiguous cases and record every automatic intervention for review.

### Steps

1. The stop signal maps to a real unacceptable outcome.
2. The threshold is defined before deployment.
3. The rollback or halt targets the correct change version.
4. The automatic path has been tested safely.
5. A human can inspect and override when the signal is ambiguous.
6. Every automatic stop creates an auditable event.

### Example

Pause a configuration rollout automatically when error rate or a critical data-integrity check crosses the predefined boundary.

### Check

A known bad signal limits further exposure without depending on someone noticing a dashboard in time.

### Limits

- Poor thresholds can cause harmful flapping or false rollback; automate only signals whose meaning and recovery action are understood.

### Evidence and sources

- supports: AWS recommends automated rollback when predefined tests or desired-outcome thresholds indicate that a deployed change is unsuccessful. — RS-1877C4E72975A9D2. Automatic rollback needs carefully chosen signals; a false alarm can itself create disruption. (Pre-defined conditions and automated rollback)
- RS-1877C4E72975A9D2: OPS06-BP04 Automate testing and rollback — https://docs.aws.amazon.com/wellarchitected/latest/framework/ops_mit_deploy_risks_auto_testing_and_rollback.html

No review details supplied.
