Chapter 6 · Senior Developer

Generators and lazy evaluation

generators · yield · lazy evaluation · iterator protocolDEV-50138 of 84

A generator is a function that uses yield instead of return. Each time you ask it for a value, it runs until the next yield, hands that value back, and *pauses* — keeping all its local state. Ask again and it resumes right where it left off.

The payoff is lazy evaluation: values are produced one at a time, on demand, instead of all at once. A normal function that builds a list of a billion rows needs memory for a billion rows. A generator that yields them one at a time needs memory for *one*. That's the difference between streaming a huge file and crashing trying to load it.

Under the hood this is the iterator protocol: a generator is an iterator, so next() pulls the next value and a for loop calls next() for you until the generator is exhausted, at which point it raises StopIteration (the loop catches this silently).

For a quick one-off, a generator expression — (x * 2 for x in data) — is the same idea in brackets: like a list comprehension but lazy. Reach for a generator when the sequence is large, infinite, or you'll only walk it once; reach for a list when you need to index it, re-use it, or know its length.

Syntax

def gen(n):
    for i in range(n):
        yield i          # pause here, hand back i, resume next time

g = gen(3)
next(g)                  # 0
next(g)                  # 1
for x in gen(3): ...     # for-loop drives next() until StopIteration

squares = (i * i for i in range(5))   # generator expression (lazy)

Worked examples

yield pauses and resumes — pull values with next()

def squares(n):
    for i in range(n):
        yield i * i

g = squares(4)
print(next(g))   # 0
print(next(g))   # 1
print(list(g))   # [4, 9]  (the rest, from where it paused)
# Output:
# 0
# 1
# [4, 9]

A generator expression is lazy — far smaller in memory

import sys

eager = [x for x in range(100000)]   # builds the whole list
lazy = (x for x in range(100000))    # builds nothing yet

print(sys.getsizeof(eager) > sys.getsizeof(lazy))
# Output:
# True

The iterator protocol: next() and StopIteration

it = iter([10, 20])
print(next(it))   # 10
print(next(it))   # 20
try:
    next(it)
except StopIteration:
    print("exhausted")
# Output:
# 10
# 20
# exhausted

Common mistakes & gotchas

A generator is single-use

Once you've walked a generator to the end, it's spent — looping over it again yields nothing. g = squares(4); list(g) works; a second list(g) is []. If you need the values twice, store them in a list or call the generator function again to get a fresh one.

Calling it doesn't run the body

gen(3) returns a generator object without executing a single line of the body — the code only runs as you pull values. So a print() or side effect inside the generator won't happen until you iterate. This surprises people debugging "why didn't my generator do anything?".

You can't len() or index a generator

len(g) and g[0] both raise TypeError — a generator doesn't know its length and has no positions, because the values don't exist yet. If you need those, convert with list(g) first (which, of course, gives up the laziness).

Why it matters

Lazy evaluation is how Python handles data too big to fit in memory — log files, database cursors, streamed API results. Every file object, `range`, and `csv.reader` in the standard library is built on this. Once you think in generators, you write pipelines that scale from ten rows to ten million without changing a line.