Chapter 10 · AI Architect

Observability and tracing

tracer · spans · parent links · trace tree · slowest-span analysisARCH-90672 of 84

When a multi-agent pipeline is slow or broken, you cannot debug it blind. You need tracing: every step records a span — its name, how long it took, whether it succeeded, and which step it ran *under* (its parent). Spans assemble into a trace tree, and the tree tells you where the time and the failures went.

A span is just a dict: {"name", "duration", "status", "parent"}. The parent link is what turns a flat list of spans into a tree — embed and retrieve both have parent request, so they're children of request.

Four operations on a tracer:

span — record a span (and return it, so the caller can hold a reference or nest under it).
children — the spans whose parent is a given name, in insertion order. Walk these to reconstruct the tree.
slowest — the name of the longest-duration span. *This is the single most useful number in ops* — it's where you optimise first.
failed — the names of every span whose status isn't ok. Your error list.

Here durations are passed in explicitly so tests are deterministic — no real clock. In production this is OpenTelemetry, with spans exported to Jaeger / Honeycomb / Datadog and durations measured for real, but the shape is identical.

Syntax

class Tracer:
    def __init__(self):
        self.spans = []

    def span(self, name, duration, status="ok", parent=None):
        record = {"name": name, "duration": duration, "status": status, "parent": parent}
        self.spans.append(record)
        return record

    def children(self, name):
        return [s["name"] for s in self.spans if s["parent"] == name]

    def slowest(self):
        if not self.spans:
            return None
        best = self.spans[0]
        for s in self.spans[1:]:
            if s["duration"] > best["duration"]:
                best = s
        return best["name"]

    def failed(self):
        return [s["name"] for s in self.spans if s["status"] != "ok"]

Worked examples

Spans with parent links form a tree; children walks it

class Tracer:
    def __init__(self):
        self.spans = []
    def span(self, name, duration, status="ok", parent=None):
        rec = {"name": name, "duration": duration, "status": status, "parent": parent}
        self.spans.append(rec)
        return rec
    def children(self, name):
        return [s["name"] for s in self.spans if s["parent"] == name]

t = Tracer()
t.span("request", 12.0)
t.span("embed", 2.0, parent="request")
t.span("retrieve", 3.0, parent="request")
print(t.children("request"))
# Output:
# ['embed', 'retrieve']

slowest points at the bottleneck; failed lists the errors

class Tracer:
    def __init__(self):
        self.spans = []
    def span(self, name, duration, status="ok", parent=None):
        self.spans.append({"name": name, "duration": duration, "status": status, "parent": parent})
    def slowest(self):
        if not self.spans:
            return None
        best = self.spans[0]
        for s in self.spans[1:]:
            if s["duration"] > best["duration"]:
                best = s
        return best["name"]
    def failed(self):
        return [s["name"] for s in self.spans if s["status"] != "ok"]

t = Tracer()
t.span("request", 12.0)
t.span("generate", 7.0, parent="request")
t.span("rerank", 1.0, parent="request", status="error")
print("slowest:", t.slowest())
print("failed:", t.failed())
# Output:
# slowest: request
# failed: ['rerank']

An empty tracer has no slowest span

class Tracer:
    def __init__(self):
        self.spans = []
    def slowest(self):
        if not self.spans:
            return None
        return self.spans[0]["name"]

print(Tracer().slowest())
# Output:
# None

Common mistakes & gotchas

Returning the whole span dict instead of its name

slowest tracks best as the full span dict while comparing durations — but callers want the *name*, so return best['name'], not best. Returning the dict leaks the internal shape and breaks any caller expecting a string.

Forgetting the empty-tracer case

best = self.spans[0] raises IndexError if no spans were recorded. Guard with if not self.spans: return None first. Aggregations over a possibly-empty collection (slowest, max, first) always need the empty case handled.

Comparing >= instead of > for the slowest

Using > means the FIRST span at the maximum duration wins ties — a stable, predictable result. Switching to >= makes the LAST equal span win, which quietly changes behaviour on ties and can surprise tests. Pick > for first-wins (as here) and be consistent.

Why it matters

You can't fix what you can't see. The first question on any slow or flaky AI pipeline is 'which step?' — and a trace answers it in seconds where guessing takes hours. Spans, parent links and slowest-span analysis are the literal foundation of OpenTelemetry and every production observability stack; building a tiny one by hand demystifies the whole category.