Failure-mode design
Production systems don't aim to never fail — at scale, something always fails. They aim to fail well. The architect's job is to classify each failure and route it appropriately.
The core distinction is transient vs permanent:
- A transient error (timeout, rate-limit, a brief network blip) is likely to succeed if you just try again. So you retry it, up to a limit.
- A permanent error (a malformed request, an auth failure, a 400) will fail identically every time. Retrying it wastes time and money — so you don't retry; you send it straight to a dead-letter queue for a human to inspect.
And over the top of both: a fallback. If retries are exhausted, or a request is dead-lettered, you don't crash — you return a safe default so the system stays up. That's graceful degradation: degraded, not down. A cached answer, a cheaper model's answer, an honest "[unavailable]" — anything better than a 500.
The routing logic in resilient_call: loop up to max_retries times, return on success, continue (retry) on a transient error until the last attempt then fall back, and on a permanent error append the work item to dead_letters and return the fallback immediately — no retry. This composes the retry and dead-letter patterns into one failure-routing policy. In production the dead-letter is a real queue (an SQS DLQ, a dead_letters table) and the fallback is a cached or cheaper response.
Syntax
class TransientError(Exception): pass
class PermanentError(Exception): pass
def resilient_call(fn, arg, max_retries, fallback, dead_letters):
for attempt in range(max_retries):
try:
return fn(arg) # success -> done
except TransientError:
if attempt == max_retries - 1:
return fallback # out of retries -> degrade
continue # otherwise retry
except PermanentError:
dead_letters.append(arg) # never retry; park for inspection
return fallback
return fallbackWorked examples
A transient error is retried until it succeeds
class TransientError(Exception): pass
class PermanentError(Exception): pass
def resilient_call(fn, arg, max_retries, fallback, dead_letters):
for attempt in range(max_retries):
try:
return fn(arg)
except TransientError:
if attempt == max_retries - 1:
return fallback
continue
except PermanentError:
dead_letters.append(arg)
return fallback
return fallback
state = {"n": 0}
def flaky(x):
state["n"] += 1
if state["n"] < 3:
raise TransientError("blip") # fails twice, then works
return f"ok:{x}"
dl = []
print(resilient_call(flaky, "job", 5, "SAFE", dl))
print("attempts:", state["n"])
# Output:
# ok:job
# attempts: 3A permanent error is dead-lettered immediately — no retry
class TransientError(Exception): pass
class PermanentError(Exception): pass
def resilient_call(fn, arg, max_retries, fallback, dead_letters):
for attempt in range(max_retries):
try:
return fn(arg)
except TransientError:
if attempt == max_retries - 1:
return fallback
continue
except PermanentError:
dead_letters.append(arg)
return fallback
return fallback
def bad(x):
raise PermanentError("400") # would fail identically every time
dl = []
print(resilient_call(bad, "job2", 5, "SAFE", dl))
print("dead_letters:", dl)
# Output:
# SAFE
# dead_letters: ['job2']Exhausted retries degrade to the fallback (not dead-lettered)
class TransientError(Exception): pass
class PermanentError(Exception): pass
def resilient_call(fn, arg, max_retries, fallback, dead_letters):
for attempt in range(max_retries):
try:
return fn(arg)
except TransientError:
if attempt == max_retries - 1:
return fallback
continue
except PermanentError:
dead_letters.append(arg)
return fallback
return fallback
count = {"n": 0}
def always_transient(x):
count["n"] += 1
raise TransientError("t") # never recovers
dl = []
print(resilient_call(always_transient, "x", 3, "SAFE", dl))
print("attempts:", count["n"], "| dead_letters:", dl)
# Output:
# SAFE
# attempts: 3 | dead_letters: []Common mistakes & gotchas
Retrying a permanent error
A 400 / auth failure won't change on the next attempt — retrying it just burns your retry budget, your token budget, and time, then fails anyway. Permanent errors must skip the retry loop entirely: dead-letter and return at once. Only transient errors earn a retry.
Off-by-one in the retry count
for attempt in range(max_retries) makes exactly max_retries total attempts (attempt 0 through max_retries-1). The 'give up' check is attempt == max_retries - 1 — the LAST index, not max_retries. Get this wrong and you either retry one too few times or fall off the loop into an unintended path.
Dead-lettering transient failures too
When transient retries are exhausted you return the fallback but should NOT dead-letter — a transient failure isn't a broken request, it's bad luck, and parking it for human inspection just floods the queue with noise. Only permanent errors belong in the dead-letter queue; transient exhaustion just degrades.
Why it matters
Every system that touches a network will see failures, and the difference between a fragile product and a robust one is entirely in how those failures are handled. Classifying transient vs permanent, retrying the recoverable, parking the broken, and always having a safe fallback is the resilience playbook — it's why a well-built AI service degrades to a slightly worse answer instead of showing your users an error page.