A retry is another unit of work, not an automatic improvement in reliability. It can recover from a transient failure, but can also repeat an unsafe operation or add demand to an overloaded dependency. Decide whether the operation is eligible, how much total effort is allowed, and when that effort should happen before choosing a delay formula.

Bound attempts before spreading them

Distinguish attempts from retries: three total attempts means one initial call and at most two retries. Also choose which layer owns recovery. If three nested layers each allow three attempts, one outer action can cause twenty-seven calls to a persistently failing leaf. Adding jitter changes their timing, not that multiplication.

Google SRE describes both per-request and wider retry budgets. A per-request cap alone still allows a large population of failing clients to generate substantial extra work. Measure retry traffic separately from original demand and coordinate with existing SDK, proxy, and application policies.

Exponential backoff sets a ceiling

A capped exponential schedule increases its ceiling until reaching a maximum. Without jitter, clients that fail together can calculate the same next time, including the same capped interval. Waiting longer does not necessarily break their synchronization.

Full jitter samples between zero and the current ceiling. Because the draw is random, the next realized wait can be shorter than the previous one even while the ceiling grows. It is inaccurate to say every individual wait must increase.

AWS’s comparison distinguishes full, equal, and decorrelated jitter and reports results from a particular contention simulation. Those results are useful evidence, not a promise that one policy minimizes every workload’s latency or prevents all bursts. Correlated random seeds, timer granularity, and many clients can still create clusters.

Use one end-to-end deadline

The budget must include execution, waiting, and subsequent attempts. Creating a fresh full timeout each time silently expands the original caller’s allowed lifetime. Propagate the remaining budget to downstream operations and recheck it before doing more work.

Use a monotonic clock for elapsed-time accounting within a process. A client timeout is not proof that the server stopped or never committed. Cancellation and remote deadlines need real transport support, and the application still needs its retry or idempotency contract.

A server-directed wait such as Retry-After adds another constraint. Do not sample an earlier retry than that boundary merely because a local jitter formula permits it. If the required wait cannot fit, abandon that retry under the current deadline instead of shortening the requirement.

When it breaks

Permanent validation errors do not become valid through repetition. A nonidempotent operation with an uncertain result can duplicate its effect. Select retriable failure classes deliberately; a blanket catch-and-retry loop hides programming errors and may convert a small incident into an overload cycle.

A backoff cap without an attempt or elapsed-time bound still permits endless work. Conversely, a deadline that is too short can reject every expensive legitimate operation. Observe completed work, retry counts, total latency, and abandonment reasons together; a falling error rate alone can hide rising retry cost.

Check it quickly

Save the block as retry_check.py and run python3 retry_check.py with Python 3.10 or newer. This review ran it with Python 3.12.14 using only the standard library. The clock and downstream calls are deterministic models; the script does not sleep, start servers, or contact a dependency. Its retry policy handles timeouts and 503 only for GET or HEAD.

from collections import Counter
import random


class Clock:
    def __init__(self):
        self.now = 0.0

    def advance(self, seconds):
        self.now += seconds


class ScriptedCall:
    def __init__(self, clock, steps):
        self.clock, self.steps, self.timeouts = clock, iter(steps), []

    def __call__(self, timeout):
        self.timeouts.append(timeout)
        status, duration = next(self.steps)
        self.clock.advance(min(duration, timeout))
        if duration >= timeout:
            raise TimeoutError("the cooperative fixture used the remaining budget")
        return status


def retry(call, clock, budget, sample, method="GET", max_attempts=3):
    deadline = clock.now + budget
    for attempt in range(1, max_attempts + 1):
        remaining = deadline - clock.now
        if remaining <= 0:
            return "deadline", attempt - 1
        try:
            status = call(remaining)
        except TimeoutError:
            status = None
        if clock.now >= deadline:
            return "deadline", attempt
        if status == 200:
            return "success", attempt
        if status not in (None, 503) or method not in ("GET", "HEAD"):
            return "not-retryable", attempt
        if attempt == max_attempts:
            return "attempt-limit", attempt
        ceiling = 2.0 ** min(attempt - 1, 3)
        delay = ceiling * sample()
        assert 0 <= delay <= ceiling
        if delay >= deadline - clock.now:
            return "wait-exceeds-budget", attempt
        clock.advance(delay)
    raise AssertionError("positive attempt limit required")


clock = Clock()
call = ScriptedCall(clock, [(503, 0.25), (503, 0.25), (200, 0.25)])
assert retry(call, clock, 5, lambda: 0.5) == ("success", 3)
assert clock.now == 2.25 and call.timeouts == [5, 4.25, 3]
print("three attempts finish at 2.25; propagated timeouts shrink from 5 to 4.25 to 3")

class EarlyTimeout:
    def __init__(self, clock):
        self.clock, self.timeouts = clock, []

    def __call__(self, timeout):
        self.timeouts.append(timeout)
        self.clock.advance(0.25)
        if len(self.timeouts) == 1:
            raise TimeoutError("a transient failure before the deadline")
        return 200


clock = Clock()
call = EarlyTimeout(clock)
assert retry(call, clock, 5, lambda: 0.5) == ("success", 2)
assert clock.now == 1.0 and call.timeouts == [5, 4.25]
print("an early timeout can recover without resetting the shared deadline")

for method, status in (("POST", 503), ("GET", 400)):
    clock = Clock()
    call = ScriptedCall(clock, [(status, 0.25)])
    assert retry(call, clock, 5, lambda: 0.5, method) == ("not-retryable", 1)
clock = Clock()
call = ScriptedCall(clock, [(503, 0.25)] * 4)
assert retry(call, clock, 5, lambda: 0.5) == ("attempt-limit", 3)
assert len(call.timeouts) == 3
print("POST, permanent failure, and the total-attempt cap stop further work")

clock = Clock()
call = ScriptedCall(clock, [(503, 0.25)] * 7)
assert retry(call, clock, 100, lambda: 1.0, max_attempts=7) == ("attempt-limit", 7)
assert clock.now == 32.75  # Seven 0.25 calls plus waits 1, 2, 4, 8, 8, 8.
print("the exponential ceiling stops at 8 rather than growing without bound")

clock = Clock()
call = ScriptedCall(clock, [(503, 0.1)])
assert retry(call, clock, 0.5, lambda: 1.0) == ("wait-exceeds-budget", 1)
assert clock.now == 0.1 and len(call.timeouts) == 1
clock = Clock()
call = ScriptedCall(clock, [(503, 99)])
assert retry(call, clock, 0.5, lambda: 0.5) == ("deadline", 1)
assert clock.now == 0.5
clock = Clock()
call = ScriptedCall(clock, [])
assert retry(call, clock, 0, lambda: 0.5) == ("deadline", 0) and call.timeouts == []
print("a wait that cannot fit is abandoned; cooperative execution cannot consume a fresh budget per attempt")

rng = random.Random(7)  # Repeatable samples for this experiment, not a production seed.
delays = [rng.random() for _ in range(100)]
bins = Counter(int(delay * 10) for delay in delays)
fixed_bins = Counter(int(1.0 * 10) for _ in range(100))
assert all(0 <= delay < 1 for delay in delays) and sum(bins.values()) == 100
assert max(bins.values()) < 100
assert max(fixed_bins.values()) == 100
print("first-retry 0.1-second-bin peak: fixed delay", max(fixed_bins.values()), "; full-jitter sample", max(bins.values()))

leaf_calls = 0


class Unavailable(Exception):
    pass


def nested(depth):
    global leaf_calls
    if depth == 0:
        leaf_calls += 1
        raise Unavailable
    for _ in range(3):
        try:
            return nested(depth - 1)
        except Unavailable:
            pass
    raise Unavailable


try:
    nested(3)
except Unavailable:
    pass
assert leaf_calls == 27
print("three retrying layers, each allowing three total attempts, invoke the failing leaf 27 times")

Two failures followed by success finish at modeled time 2.25. The propagated timeouts shrink from five to 4.25 to three seconds; execution does not receive a fresh budget. An early timeout also recovers within that shared budget. The exponent is capped before evaluation. Separate assertions cover the total-attempt limit, an eight-second ceiling, permanent errors, POST exclusion, a wait that cannot fit, and an already exhausted deadline.

One hundred fixed first-retry delays land in one 0.1-second bin. The reproducible full-jitter sample has a peak bin count of thirteen. This compares calculated schedules, not actual network arrival times or successful throughput. The fixed random seed exists only to reproduce the experiment, not to synchronize production clients.

Finally, three nested retry loops produce twenty-seven leaf invocations before failure. The example deliberately shows that amplification rather than recommending it. Wider retry quotas, server-directed delay parsing, real timers, transport cancellation, distributed clocks, and production latency remain outside this model’s tests.

Further reading

Sources