Chapter 5 · Data / ML Engineer

Evaluation metrics

accuracy · precision · recall · confusion matrixDAT-40535 of 84

A classifier that claims 95% accuracy can still be useless — if 95% of emails aren't spam, a model that says "never spam" scores 95% and catches nothing. To see the real story you need the confusion matrix and the metrics built from it.

For a binary classifier (label 'pos' / 'neg'), every prediction lands in one of four boxes:

TP (true positive) — predicted pos, actually pos ✓
FP (false positive) — predicted pos, actually neg ✗ (a false alarm)
FN (false negative) — predicted neg, actually pos ✗ (a miss)
TN (true negative) — predicted neg, actually neg ✓

From those four counts:

Accuracy = (TP + TN) / total — fraction of all predictions that were right.
Precision = TP / (TP + FP) — of everything you *called* positive, how much really was? (Punishes false alarms.)
Recall = TP / (TP + FN) — of all the *actual* positives, how many did you catch? (Punishes misses.)

Both ratios can divide by zero (no positive predictions → precision; no actual positives → recall) — guard each with a ... if denom > 0 else 0.0.

*In production this is sklearn.metrics.classification_report. The arithmetic below is exactly what it computes.*

Syntax

# Confusion counts from two aligned label lists
for pred, actual in zip(predictions, actuals):
    if   pred == 'pos' and actual == 'pos': tp += 1
    elif pred == 'pos' and actual == 'neg': fp += 1
    elif pred == 'neg' and actual == 'pos': fn += 1
    else:                                   tn += 1

accuracy  = (tp + tn) / total
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall    = tp / (tp + fn) if (tp + fn) > 0 else 0.0

Worked examples

Confusion counts + all three metrics

def confusion_counts(predictions, actuals):
    tp = fp = fn = tn = 0
    for pred, actual in zip(predictions, actuals):
        if pred == "pos" and actual == "pos":
            tp += 1
        elif pred == "pos" and actual == "neg":
            fp += 1
        elif pred == "neg" and actual == "pos":
            fn += 1
        else:
            tn += 1
    return {"TP": tp, "FP": fp, "FN": fn, "TN": tn}

def accuracy(predictions, actuals):
    c = confusion_counts(predictions, actuals)
    return round((c["TP"] + c["TN"]) / len(predictions), 4)

def precision_recall(predictions, actuals):
    c = confusion_counts(predictions, actuals)
    tp, fp, fn = c["TP"], c["FP"], c["FN"]
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return round(precision, 4), round(recall, 4)

preds = ["pos","pos","neg","pos","neg","neg","pos","neg","pos","neg"]
acts  = ["pos","neg","pos","pos","neg","pos","pos","neg","neg","neg"]
print(confusion_counts(preds, acts))
print(accuracy(preds, acts))
print(precision_recall(preds, acts))
# Output:
# {'TP': 3, 'FP': 2, 'FN': 2, 'TN': 3}
# 0.6
# (0.6, 0.6)

Worked example — a spam filter, step by step

# 6 emails. pred = filter's call, actual = the truth.
preds = ["pos", "pos", "pos", "neg", "neg", "neg"]
acts  = ["pos", "pos", "neg", "pos", "neg", "neg"]
# Pair them up:
#   pos/pos -> TP   pos/pos -> TP   pos/neg -> FP
#   neg/pos -> FN   neg/neg -> TN   neg/neg -> TN
# So: TP=2, FP=1, FN=1, TN=2  (total 6)
#
# accuracy  = (TP+TN)/total = (2+2)/6 = 0.6667
# precision = TP/(TP+FP)    = 2/(2+1) = 0.6667  (1 false alarm)
# recall    = TP/(TP+FN)    = 2/(2+1) = 0.6667  (1 spam missed)
tp, fp, fn, tn = 2, 1, 1, 2
print(round((tp + tn) / 6, 4))            # accuracy
print(round(tp / (tp + fp), 4))           # precision
print(round(tp / (tp + fn), 4))           # recall
# Output:
# 0.6667
# 0.6667
# 0.6667

Zero-denominator guard (no positive predictions)

def precision_recall(predictions, actuals):
    tp = sum(1 for p, a in zip(predictions, actuals) if p == "pos" and a == "pos")
    fp = sum(1 for p, a in zip(predictions, actuals) if p == "pos" and a == "neg")
    fn = sum(1 for p, a in zip(predictions, actuals) if p == "neg" and a == "pos")
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return round(precision, 4), round(recall, 4)

# Model never predicts pos → TP+FP == 0 → precision guarded to 0.0
print(precision_recall(["neg", "neg"], ["pos", "neg"]))
# Output:
# (0.0, 0.0)

Common mistakes & gotchas

Accuracy lies on imbalanced data

When one class dominates (99% 'neg'), a model that always predicts 'neg' scores 99% accuracy while catching zero positives — recall 0. Always look at precision AND recall, not accuracy alone, when the classes aren't roughly balanced.

Precision and recall trade off

Lower your threshold to catch more positives (recall up) and you also catch more false alarms (precision down) — and vice versa. You rarely max both. Pick which error costs more: missing a fraud (favour recall) or wrongly flagging a good customer (favour precision).

Dividing by zero on empty classes

TP/(TP+FP) blows up when the model predicts no positives; TP/(TP+FN) blows up when there are no actual positives. Both happen on small or skewed batches. Guard each ratio with if denom > 0 else 0.0 — a metric of 0.0 is meaningful, a crash isn't.

Why it matters

"Is the model any good?" has no single-number answer. Knowing precision from recall — and when accuracy is hiding the truth — is what separates someone who can ship a model from someone who just trained one. It's the vocabulary every ML conversation runs on.