When a service fails and everyone upstream retries at once, a small outage can balloon into a stack-wide incident. Here's how Uber tamed that "retry storm" problem. Title: How Uber Protects Against Retry Storms URL: To stop retry storms — where traffic amplifies exponentially as every upstream service retries simultaneously against a degraded downstream — Uber built a system that distinguishes who actually owns an error. Highlights 🔍 Only the error-owning service claims the error A service "claims" an error only when its downstream call succeeded but it still returned an error itself; if the downstream is at fault, it "unclaims" instead. Only the actual owner of the problem retries. 🔁 Retry budgets and propagation cap further amplification Retry budgets cap retries at each hop to a fixed percentage, turning exponential growth into multiplicative growth. When a retry fails, the "unclaimed" status propagates upstream to prevent further amplification higher in the chain. 📊 Stopped 9.5 million requests during a real outage In a November 18, 2025 incident, traffic to a degraded service would have jumped 46–135% without this system. Instead, it blocked 9.5 million spurious requests plus up to 200,000 more from immediate callers, and cut the max retry-storm radius across all user-facing APIs from 25 hops to 3. What stands out is how this runs as automatic, shared infrastructure with no per-service tuning needed — that's exactly what makes it practical at Uber's scale. #ReliabilityEngineering# #DistributedSystems#