Chapter 10 · AI Architect

Cost control and token budgeting

token estimation · price table · budget enforcement · model-tier downgradeARCH-90571 of 84

Token cost is a first-class architectural concern. At scale, the gap between a cheap model and a frontier model on every single call is the gap between a viable product and a bankrupt one. An architect estimates cost up front, enforces a budget, and downgrades to a cheaper tier rather than blowing it.

Four pieces:

Estimate tokens — a rough heuristic of ~1 token per 4 characters. This is honestly approximate: real tokenisers split on sub-words and vary by model, so treat len(text) // 4 as a ballpark for planning, not a billing figure.
Price table — cost per 1,000 tokens by tier. The numbers here (frontier 15.0, standard 3.0, mini 0.5) are *illustrative teaching bands*, not real provider prices.
Enforce a budget — before charging, check spent + cost against the budget. If it would exceed, don't spend — refuse and raise.
Downgrade — if the preferred (expensive) tier would blow the budget, fall back to a cheaper tier instead of failing the whole request. If even the cheap tier is over budget, *then* let it fail.

The ordering matters: cost is computed from tokens / 1000 * price, the check happens *before* the spend is recorded, and a refused call must leave spent untouched. In production you'd swap the heuristic for the provider's real tokeniser and the bands for the live price list.

Syntax

PRICES = {"frontier": 15.0, "standard": 3.0, "mini": 0.5}   # per 1000 tokens (illustrative)

def estimate_tokens(text):
    return max(1, len(text) // 4)        # rough heuristic, NOT exact tokenisation

class BudgetController:
    def __init__(self, budget):
        self.budget = budget
        self.spent = 0.0

    def cost_of(self, text, tier):
        return estimate_tokens(text) / 1000 * PRICES[tier]

    def charge(self, text, tier):
        cost = self.cost_of(text, tier)
        if self.spent + cost > self.budget:
            raise RuntimeError("over budget")   # refuse BEFORE spending
        self.spent += cost
        return cost

Worked examples

Estimate tokens (chars/4) and price against the table

PRICES = {"frontier": 15.0, "standard": 3.0, "mini": 0.5}

def estimate_tokens(text):
    return max(1, len(text) // 4)      # rough — real tokenisers differ

def cost_of(text, tier):
    return estimate_tokens(text) / 1000 * PRICES[tier]

print(estimate_tokens("x" * 4000))     # ~1000 tokens
print(cost_of("x" * 4000, "frontier")) # 1000/1000 * 15.0
print(cost_of("x" * 4000, "mini"))     # 1000/1000 * 0.5
# Output:
# 1000
# 15.0
# 0.5

An over-budget call is refused and spends nothing

PRICES = {"frontier": 15.0, "mini": 0.5}

class BudgetController:
    def __init__(self, budget):
        self.budget = budget
        self.spent = 0.0
    def cost_of(self, text, tier):
        return max(1, len(text) // 4) / 1000 * PRICES[tier]
    def charge(self, text, tier):
        cost = self.cost_of(text, tier)
        if self.spent + cost > self.budget:
            raise RuntimeError("over budget")
        self.spent += cost
        return cost

c = BudgetController(1.0)
try:
    c.charge("x" * 4000, "frontier")   # ~15.0, way over the 1.0 budget
except RuntimeError as e:
    print("blocked:", e, "| spent:", c.spent)
# Output:
# blocked: over budget | spent: 0.0

Downgrade to a cheaper tier instead of blowing the budget

PRICES = {"frontier": 15.0, "mini": 0.5}

class BudgetController:
    def __init__(self, budget):
        self.budget = budget
        self.spent = 0.0
    def cost_of(self, text, tier):
        return max(1, len(text) // 4) / 1000 * PRICES[tier]
    def charge(self, text, tier):
        cost = self.cost_of(text, tier)
        if self.spent + cost > self.budget:
            raise RuntimeError("over budget")
        self.spent += cost
        return cost
    def charge_with_downgrade(self, text, preferred, fallback):
        try:
            self.charge(text, preferred)
            return preferred
        except RuntimeError:
            self.charge(text, fallback)   # may itself raise if even this is too dear
            return fallback

ctrl = BudgetController(1.0)
used = ctrl.charge_with_downgrade("x" * 4000, "frontier", "mini")
print("charged at:", used)
print("spent: %.2f" % ctrl.spent)
# Output:
# charged at: mini
# spent: 0.50

Common mistakes & gotchas

Treating chars/4 as exact

len(text) // 4 is a planning heuristic, not real tokenisation — actual tokenisers split on sub-words and punctuation and vary per model, so the true count can be off by 20–30%. Fine for budgeting headroom; never quote it as a precise bill. In production, call the provider's real tokeniser.

Recording the spend before the budget check

If you do self.spent += cost first and check afterwards, an over-budget call still mutates your running total — your budget leaks. Check if self.spent + cost > self.budget: raise BEFORE touching self.spent, so a refused call costs nothing.

Swallowing the over-budget error on downgrade

charge_with_downgrade catches the preferred tier's RuntimeError and retries on the fallback — but if the fallback *also* can't afford it, that second RuntimeError must propagate, not be swallowed. Don't wrap the fallback charge in its own bare except; let genuine 'can't afford anything' failures surface.

Why it matters

An AI feature that works in a demo can quietly bankrupt you in production if every request hits the most expensive model. Estimating cost, enforcing a hard budget, and degrading to a cheaper tier under pressure are how you ship something that stays both useful and affordable — and it's a conversation you'll have with finance, not just engineering.