Protocol
Reconcile what changed after a bulk write
'Job finished' is a process status, not a data-quality result.
When it fits
- A large update reports technical completion but business correctness still needs proof.
When to avoid it
- Reconciliation is only as strong as the intended-state specification; an incorrect source file can reconcile perfectly to the wrong goal.
Why it matters
Compare the authoritative post-write state with the intended mutation set. Check totals, missing keys, unexpected keys, critical fields and a sample of downstream effects. Classify mismatches instead of merely counting them. Preserve the reconciliation artifact so the team can repair the exact exceptions rather than rerun the whole batch blindly.
Steps
- Every intended key is accounted for.
- Unexpected changed keys are detected.
- Critical before/after fields match the approved mutation.
- Exceptions are classified with enough detail to repair selectively.
- Downstream replication or derived state is checked where it is part of the intended outcome.
An example
After 20,000 customer updates, compare the target table and downstream partner data with the approved input rather than trusting the success message.
Check your result
You can state exactly how many records matched, failed, were missed or changed unexpectedly.
Keep this limit in mind
- Reconciliation is only as strong as the intended-state specification; an incorrect source file can reconcile perfectly to the wrong goal.
Evidence and sources
Google SRE recommends comparing canary or dry-run pipeline results with the live pipeline to check for data differences before expansion.
Reconciliation only catches differences represented by the comparison logic; missing invariants remain missing.
Improve and Optimize Data Processing Pipelines · Verification of canary or preproduction environment