Chapter 8 · Integrations / Automation Engineer

Retry with exponential backoff

retry logic · exponential backoff · delay sequencesAUT-40152 of 84

Anything that talks to an external service — an API, a database, a queue — will eventually hit a transient failure: a timeout, a dropped connection, a momentary 503. The amateur move is to give up on the first error. The professional move is to retry.

But you don't retry instantly, and you don't retry forever. Exponential backoff spaces the attempts out, doubling the wait each time: base_delay * (2 attempt). Attempt 0 waits base_delay, attempt 1 waits 2×, attempt 2 waits 4×. You cap** it with max_delay so the waits don't grow to minutes, and you stop after max_attempts.

The doubling matters: it backs off fast when a service is struggling, instead of hammering it with a tight retry loop and making things worse.

A few rules of thumb:

Only retry transient errors (timeouts, 5xx, rate-limit responses). A 400 Bad Request or 404 won't fix itself — retrying is just wasted time.
Add a little jitter (a small random amount) in production so a thousand clients retrying at once don't all fire on the exact same tick — but jitter makes output non-deterministic, so the examples below leave it out.

In a real script the wait is time.sleep(delay). Here we *compute* the delay sequence and record it — same logic, fully reproducible, no actual waiting.

Syntax

delay = min(base_delay * (2 ** attempt), max_delay)

for attempt in range(max_attempts):
    try:
        return fn()              # success → stop
    except Exception:
        delay = min(base_delay * (2 ** attempt), max_delay)
        # time.sleep(delay)      # real script waits here
# all attempts exhausted

Worked examples

The delay doubles each attempt, then caps

def calc_delay(attempt, base_delay, max_delay):
    return min(base_delay * (2 ** attempt), max_delay)

print(calc_delay(0, 1.0, 60.0))   # 1.0
print(calc_delay(1, 1.0, 60.0))   # 2.0
print(calc_delay(2, 1.0, 60.0))   # 4.0
print(calc_delay(10, 1.0, 10.0))  # 10.0  (capped)
# Output:
# 1.0
# 2.0
# 4.0
# 10.0

The whole delay sequence for a run

def calc_delay(attempt, base, cap):
    return min(base * (2 ** attempt), cap)

def get_delay_sequence(max_attempts, base, cap):
    return [calc_delay(i, base, cap) for i in range(max_attempts)]

print(get_delay_sequence(4, 1.0, 60.0))   # [1.0, 2.0, 4.0, 8.0]
print(get_delay_sequence(4, 1.0, 5.0))    # [1.0, 2.0, 4.0, 5.0]
# Output:
# [1.0, 2.0, 4.0, 8.0]
# [1.0, 2.0, 4.0, 5.0]

Retry until success, recording the delays

def retry_with_backoff(fn, max_attempts, base, cap):
    delays = []
    for attempt in range(max_attempts):
        try:
            value = fn()
            return {'result': value, 'attempts': attempt + 1, 'delays': delays}
        except Exception:
            delays.append(min(base * (2 ** attempt), cap))
    return {'result': None, 'attempts': max_attempts, 'delays': delays}

calls = [0]
def flaky():
    calls[0] += 1
    if calls[0] < 3:
        raise RuntimeError('not yet')
    return 'connected'

print(retry_with_backoff(flaky, 5, 1.0, 60.0))
# Output:
# {'result': 'connected', 'attempts': 3, 'delays': [1.0, 2.0]}

Common mistakes & gotchas

Retrying errors that will never succeed

Backoff is for *transient* failures. A 400, 401, or 404 is a permanent error — the request itself is wrong, and retrying it five times just wastes five delays before failing anyway. Retry on timeouts and 5xx; fail fast on 4xx.

Forgetting to cap the delay

Without min(..., max_delay) the delay doubles unbounded — attempt 10 of a 1-second base is 2 ** 10 = 1024 seconds (17 minutes). Always cap it. The cap is why calc_delay(10, 1.0, 10.0) returns 10.0, not 1024.0.

No jitter means a thundering herd

If many clients fail at the same instant and all use the identical backoff schedule, they retry in perfect sync and stampede the recovering service on every tick. Production code adds a small random jitter to spread them out. The examples omit it only to stay deterministic.

Why it matters

Networks are unreliable by nature — packets drop, services restart, load spikes. Retry-with-backoff is the single most common resilience pattern in production code: it turns a transient blip that would have crashed your script into a half-second pause nobody notices.