Protocol
Retry slower, with randomness, and stop
A retry storm is an outage trying to help.
When it fits
- A remote dependency fails transiently and many clients may retry at the same time.
When to avoid it
- Backoff reduces amplification but may increase completion latency; tune it to the service and user-facing deadline.
Why it matters
Use a bounded retry policy: wait longer between attempts, add jitter so clients do not synchronize, and stop after a defined budget. Combine this with timeouts and idempotency for mutating operations. Retry only errors that can plausibly recover; validation failures and permanent authorization errors need a different response.
Steps
- The retry policy cannot continue indefinitely or synchronize a fleet into repeated load spikes.
An example
A worker retries a throttled API after randomized exponential delays instead of immediately looping hundreds of requests.
Check your result
The retry policy cannot continue indefinitely or synchronize a fleet into repeated load spikes.
Keep this limit in mind
- Backoff reduces amplification but may increase completion latency; tune it to the service and user-facing deadline.
Connected ideas
Useful withGive a retryable mutation one stable operation ID
Evidence and sources
AWS reliability guidance recommends limiting retries and using exponential backoff with jitter rather than retrying immediately and indefinitely.
Retries are appropriate mainly for transient failures and can worsen overload when used without budgets, timeouts and idempotency.
REL05-BP03 Control and limit retry calls · Implementation guidance