Chapter 8 · Integrations / Automation Engineer

Dead-letter handling

error recovery · dead-letter queue · partial failureAUT-40556 of 84

Process a thousand items and one of them is malformed. The naive loop crashes on item 400 — and the other 600 good items never get processed. That's a total failure caused by a partial problem, and it's a terrible way to run a pipeline.

The resilient pattern: wrap each item's processing in try/except. On success, collect the result. On failure, don't crash — route that item to a dead-letter list (often called a dead-letter queue, or DLQ) and carry on with the rest.

At the end you have two lists:

succeeded — everything that worked, with its result.
dead_letter — everything that failed, with the error captured as a string so you can inspect it later.

This gives you partial-failure tolerance: one bad item costs you that one item, not the whole run. And because the failures are captured (not lost in a stack trace), you can fix the cause and retry just the dead-letter items later — re-running the failures without re-doing the successes.

This is exactly how SQS dead-letter queues, Celery's failure handling, and every mature job processor work.

Syntax

succeeded, dead_letter = [], []
for item in items:
    try:
        result = process(item)
        succeeded.append({'item': item, 'result': result})
    except Exception as e:
        dead_letter.append({'item': item, 'error': str(e)})

# retry later: re-run only the dead_letter items
retry_items = [entry['item'] for entry in dead_letter]

Worked examples

One item's outcome — success or captured error

def process_item(item, process_fn):
    try:
        result = process_fn(item)
        return {'ok': True, 'item': item, 'result': result}
    except Exception as e:
        return {'ok': False, 'item': item, 'error': str(e)}

print(process_item(5, lambda x: x * 2))
print(process_item('x', lambda x: int(x)))
# Output:
# {'ok': True, 'item': 5, 'result': 10}
# {'ok': False, 'item': 'x', 'error': "invalid literal for int() with base 10: 'x'"}

A bad item doesn't sink the whole run

def process_item(item, fn):
    try:
        return {'ok': True, 'item': item, 'result': fn(item)}
    except Exception as e:
        return {'ok': False, 'item': item, 'error': str(e)}

def run_pipeline(items, fn):
    succeeded, dead_letter = [], []
    for item in items:
        outcome = process_item(item, fn)
        (succeeded if outcome['ok'] else dead_letter).append(outcome)
    return {'succeeded': succeeded, 'dead_letter': dead_letter, 'total': len(items)}

def positive_only(x):
    if x < 0:
        raise ValueError('negative')
    return x * 10

out = run_pipeline([1, -1, 2, -2, 3], positive_only)
print(len(out['succeeded']), 'ok,', len(out['dead_letter']), 'dead, total', out['total'])
# Output:
# 3 ok, 2 dead, total 5

Retry only the failures (after the cause is fixed)

def run_pipeline(items, fn):
    succeeded, dead_letter = [], []
    for item in items:
        try:
            succeeded.append({'ok': True, 'item': item, 'result': fn(item)})
        except Exception as e:
            dead_letter.append({'ok': False, 'item': item, 'error': str(e)})
    return {'succeeded': succeeded, 'dead_letter': dead_letter, 'total': len(items)}

def retry_dead_letter(dead_letter, fn):
    items = [entry['item'] for entry in dead_letter]
    return run_pipeline(items, fn)

first = run_pipeline(['10', 'oops', '20'], int)   # 'oops' fails
fixed = retry_dead_letter(first['dead_letter'], lambda s: 0)   # fallback
print('first pass failures:', len(first['dead_letter']))
print('after retry:', len(fixed['succeeded']), 'recovered')
# Output:
# first pass failures: 1
# after retry: 1 recovered

Common mistakes & gotchas

Catching the exception but losing the detail

except Exception: with a bare 'error': 'failed' throws away *why* it failed. Capture str(e) (or the repr) so the dead-letter entry tells you what went wrong — otherwise you can't diagnose or fix the cause.

Storing the live exception object instead of a string

Keeping the raw exception ('error': e) means it can't be cleanly serialised to JSON for a real queue, and the object may hold a whole traceback's worth of references. Convert to text with str(e) when you route to the dead-letter store.

Too broad an except hides real bugs

except Exception will also swallow a KeyError from your own typo, quietly dead-lettering items because of a bug in *your* code, not the data. Keep the processing function small, and when debugging, log the dead-letter errors — a sudden pile-up usually means a code fault, not bad input.

Why it matters

In any batch of real data, some items will be malformed, and a pipeline that dies on the first bad one is useless at scale. Dead-letter handling is what turns "the whole job failed" into "998 succeeded, 2 need a look" — partial failure is survivable, isolated, and retryable instead of catastrophic.