Rate limiting and batching
Every real API has a rate limit — "max 100 requests per minute", "1000 rows per call". Fire 10,000 items through a tight loop and you'll get throttled, blocked, or your account suspended. The professional pattern is to batch.
Chunking splits a long list into sub-lists of at most batch_size. You process one chunk at a time, and in a live system you pause (time.sleep) between chunks to stay under the cap.
The core trick is slicing in steps: range(0, len(items), batch_size) gives the start of each chunk (0, batch_size, 2*batch_size, ...), and items[i:i + batch_size] grabs that slice. Python's slicing automatically stops at the end of the list, so the last chunk is just whatever's left — no special-casing needed.
A simple rate cap is a count-per-window: track how many you've sent in the current time window, and once you hit the cap, wait for the window to roll over before sending more.
Why batch rather than send one at a time? Two reasons: many APIs accept a whole batch in one call (far fewer round-trips), and batching gives you a natural place to pause and respect the limit.
Syntax
# Chunk a list into sub-lists of at most batch_size
for i in range(0, len(items), batch_size):
batch = items[i:i + batch_size]
process(batch)
# time.sleep(interval) # real script pauses between batches
# number of batches: len(chunks)
# last batch size: len(chunks[-1])Worked examples
Chunk a list (last batch is the remainder)
def chunk(items, batch_size):
result = []
for i in range(0, len(items), batch_size):
result.append(items[i:i + batch_size])
return result
print(chunk([1, 2, 3, 4, 5, 6], 2)) # even
print(chunk([1, 2, 3, 4, 5], 2)) # uneven — last is short
print(chunk([], 5)) # empty
# Output:
# [[1, 2], [3, 4], [5, 6]]
# [[1, 2], [3, 4], [5]]
# []Process each batch, collect the results
def chunk(items, n):
return [items[i:i + n] for i in range(0, len(items), n)]
def process_batches(items, batch_size, process_fn):
batches = chunk(items, batch_size)
results = [process_fn(b) for b in batches]
return {'batches': len(batches), 'results': results}
out = process_batches([1, 2, 3, 4, 5], 2, sum)
print(out)
# Output:
# {'batches': 3, 'results': [3, 7, 5]}A count-per-window rate cap
def send_all(items, cap_per_window):
sent_this_window = 0
windows = 1
for item in items:
if sent_this_window >= cap_per_window:
windows += 1 # real code: time.sleep(window)
sent_this_window = 0
sent_this_window += 1
return {'sent': len(items), 'windows_used': windows}
print(send_all(list(range(10)), cap_per_window=3))
# Output:
# {'sent': 10, 'windows_used': 4}Common mistakes & gotchas
Slicing past the end is safe — indexing isn't
items[i:i + batch_size] never raises even when i + batch_size runs past the end; it just returns what's left. But items[i + batch_size] (an index, not a slice) on the last chunk raises IndexError. Slice, don't index, when chunking.
len(chunks[-1]) blows up on an empty list
If items is empty, chunk returns [], and chunks[-1] raises IndexError: list index out of range. Guard the last-batch lookup: len(chunks[-1]) if chunks else 0.
batch_size of 0 is an infinite loop
range(0, len(items), 0) raises ValueError: range() arg 3 must not be zero. A batch_size of 0 (or negative) is never valid — validate it's >= 1 before chunking, or you'll crash (or, with a hand-rolled while, loop forever).
Why it matters
The difference between a script that quietly processes 50,000 records overnight and one that gets your API key banned at record 200 is batching. Respecting rate limits isn't optional politeness — it's the contract that keeps your integrations running, and chunked processing is how you honour it.