Protocol
Canary the batch before scaling it
Do not make the millionth write your first realistic test.
When it fits
- A change must touch many records, jobs, instances or users and real conditions matter.
When to avoid it
- A canary cannot expose failures that only appear at scale, in rare data or after long delays; design later gates for those risks.
Why it matters
Apply the real change to a deliberately small, representative slice first. Observe the same correctness and health signals that matter at full scale, including delayed processing where relevant. Expand only when the canary remains healthy for the required observation window. Choose the slice to reveal risk, not merely to make the success rate look good.
Steps
- The full rollout waits for evidence from a bounded real-world slice rather than only preproduction confidence.
An example
Update 50 representative records across several data shapes before starting the remaining 50,000.
Check your result
The full rollout waits for evidence from a bounded real-world slice rather than only preproduction confidence.
Keep this limit in mind
- A canary cannot expose failures that only appear at scale, in rare data or after long delays; design later gates for those risks.
Connected ideas
Useful withDry-run the real-shaped input without the real write
Evidence and sources
Google SRE recommends canarying a data pipeline on a subset of real production-shaped data or in dry-run mode before full rollout.
A representative canary lowers exposure but can still miss rare data shapes or delayed effects.
Improve and Optimize Data Processing Pipelines · Canarying data pipelines