Chapter 10 · AI Architect

Scaling: batching and concurrency

batching · concurrency cap · throughput simulation · completion orderARCH-90470 of 84

When you need to make 10,000 model calls, you don't fire them all at once — you'd hit rate limits, exhaust memory, and get throttled. You run a bounded number concurrently (a *concurrency cap*) and process the rest in waves.

Capacity planning means being able to *compute* how long that takes before you run it. The model here is deliberately simple and deterministic:

Batch the work into waves of at most cap items (make_batches). The last wave may be smaller.
Items in the same wave run in parallel, so a wave takes as long as its slowest item (its max cost), not the sum.
Waves run one after another, so total time is the sum of the wave durations.
Completion order is wave order, then input order within a wave.

The key insight is *within a wave, costs overlap; across waves, they add up*. A cap of 1 is fully serial (every item is its own wave, so total time is the sum of all costs). A bigger cap means fewer, fatter waves and lower total time — up to the point your rate limit bites.

Pyodide has no real async I/O, so each item carries a known cost and we schedule on a wall-clock model. In production the cap is a semaphore or worker-pool size and cost is real latency.

Syntax

def make_batches(items, size):
    return [items[i:i + size] for i in range(0, len(items), size)]

def simulate(items, cap):
    total_time = 0
    completion_order = []
    for wave in make_batches(items, cap):
        total_time += max(cost for _id, cost in wave)   # parallel -> slowest item
        completion_order += [item_id for item_id, _cost in wave]
    return total_time, completion_order                  # waves add up serially

Worked examples

Batch a list into waves of at most N (last may be smaller)

def make_batches(items, size):
    return [items[i:i + size] for i in range(0, len(items), size)]

print(make_batches([1, 2, 3, 4, 5], 2))
# Output:
# [[1, 2], [3, 4], [5]]

Simulate throughput under a concurrency cap of 2

def make_batches(items, size):
    return [items[i:i + size] for i in range(0, len(items), size)]

def simulate(items, cap):
    total_time = 0
    order = []
    for wave in make_batches(items, cap):
        total_time += max(cost for _id, cost in wave)
        order += [item_id for item_id, _cost in wave]
    return total_time, order

reqs = [("r1", 3), ("r2", 1), ("r3", 2), ("r4", 5), ("r5", 1)]
# waves: [r1,r2]->max 3, [r3,r4]->max 5, [r5]->1  =>  3+5+1 = 9
total, order = simulate(reqs, cap=2)
print("total_time:", total)
print("order:", order)
# Output:
# total_time: 9
# order: ['r1', 'r2', 'r3', 'r4', 'r5']

A cap of 1 is fully serial — total time is the sum of all costs

def make_batches(items, size):
    return [items[i:i + size] for i in range(0, len(items), size)]

def simulate(items, cap):
    total_time = 0
    order = []
    for wave in make_batches(items, cap):
        total_time += max(cost for _id, cost in wave)
        order += [item_id for item_id, _cost in wave]
    return total_time, order

print(simulate([("a", 2), ("b", 3), ("c", 1)], cap=1))
# Output:
# (6, ['a', 'b', 'c'])

Common mistakes & gotchas

Summing costs within a wave

If you do total_time += sum(cost for _, cost in wave) you've modelled the wave as *serial* — but items in a wave run concurrently. The wave's duration is the max, not the sum. Summing within the wave silently inflates your capacity estimate (and defeats the point of concurrency).

Confusing the cap with the batch count

cap is how many items run *at once*, not how many waves there are. With 5 items and cap 2 you get 3 waves (2 + 2 + 1), not 2. The number of waves is ceil(len(items) / cap). Mixing these up gives wrong throughput numbers.

A real concurrency cap is not actually batching

This simulation uses fixed waves for determinism, but a true semaphore/worker-pool refills as soon as *any* slot frees — a fast item lets the next start immediately, rather than waiting for its whole wave. The wave model is a clean upper-bound approximation; don't mistake it for exact behaviour of a real async pool.

Why it matters

Scaling AI work is mostly about respecting limits — rate limits, memory, cost — while keeping throughput high. Being able to compute 'with a cap of N this takes T' before you run it is what separates capacity planning from hopeful guessing, and the batch-under-a-cap pattern is exactly what every job queue and worker pool implements.