Retry Logic Does Not Always Make a System More Reliable
Retries can recover from transient failures, but careless retries can also turn a small slowdown into a retry storm.
0xNN · · 8 min read
Retries help with transient failures, but they are not free. Every retry adds load, latency, and queue pressure. On a struggling dependency, that extra optimism can become an amplifier.
A retry storm often starts small: one service slows down, callers time out, every caller retries three times, and the dependency receives more traffic exactly when it is least able to handle it.
What good retry logic needs
• Exponential backoff so retries get less aggressive over time
• Jitter so clients do not retry in lockstep
• Retry budgets so one request does not live forever
When not to retry
Do not retry permanent errors, non-idempotent operations, or overloaded dependencies without guardrails. Sometimes failing fast is healthier than pretending the system is more available than it is.
Recommended reading
• AWS Builders' Library - Timeouts, retries, and backoff with jitter
• Google SRE Book
• Stripe Engineering Blog
• Designing Data-Intensive Applications
Retries are useful tools. They are just dangerous when used as reflex instead of design.
---
*Written after watching a service that only needed breathing room get overwhelmed by well-meaning retries from every caller.*