Chapter 5 · Data / ML Engineer

Train → predict → evaluate

threshold learner · train/test split · evaluate pipelineDAT-407DAT-BOSS-437 of 84

Every real ML workflow is three phases on a loop:

Train — learn parameters from *labelled* data.
Predict — apply what you learned to *new* data.
Evaluate — measure how well it did, on data it never trained on.

The golden rule: split your data first, then only ever train on the training half and score on the held-out test half. Evaluating on data you trained on flatters the model — it's an open-book exam. Use a fixed random seed so the split is reproducible.

The learner here is the simplest possible: a threshold. Training tries every unique value as a cut-point, labels records 'pos' if value >= threshold else 'neg', and keeps the threshold that scores best on the training set. predict is then one comparison; evaluate runs predict over the test set and returns accuracy.

A real twist worth seeing: on a tiny test set the learned threshold may score *worse* than a hand-picked one — small samples are noisy. That's not a bug, it's why ML uses big datasets and cross-validation.

*In production: sklearn.model_selection.train_test_split plus a real estimator. The structure is identical — only the learner gets fancier.*

Syntax

import random

def train_test_split(records, test_fraction, seed):
    random.seed(seed)               # reproducible shuffle
    shuffled = records[:]
    random.shuffle(shuffled)
    n_test = round(len(shuffled) * test_fraction)
    return shuffled[:-n_test], shuffled[-n_test:]   # (train, test)

def train_threshold(train):
    best_t, best_acc = None, -1.0
    for t in sorted({r['spend'] for r in train}):
        acc = sum((r['spend'] >= t) == (r['label'] == 'pos') for r in train) / len(train)
        if acc > best_acc:
            best_acc, best_t = acc, t
    return best_t

def predict(record, threshold):
    return 'pos' if record['spend'] >= threshold else 'neg'

Worked examples

The whole loop, end to end

import random

DATASET = [
    {"spend": 450, "label": "pos"}, {"spend": 80,  "label": "neg"},
    {"spend": 320, "label": "pos"}, {"spend": 50,  "label": "neg"},
    {"spend": 210, "label": "pos"}, {"spend": 30,  "label": "neg"},
    {"spend": 500, "label": "pos"}, {"spend": 120, "label": "neg"},
    {"spend": 280, "label": "pos"}, {"spend": 90,  "label": "neg"},
]

def train_test_split(records, test_fraction, seed):
    random.seed(seed)
    shuffled = records[:]
    random.shuffle(shuffled)
    n_test = round(len(shuffled) * test_fraction)
    return shuffled[:-n_test], shuffled[-n_test:]

def train_threshold(train):
    best_t, best_acc = sorted({r["spend"] for r in train})[0], 0.0
    for t in sorted({r["spend"] for r in train}):
        correct = sum(1 for r in train if (r["spend"] >= t) == (r["label"] == "pos"))
        acc = correct / len(train)
        if acc > best_acc:
            best_acc, best_t = acc, t
    return best_t

def predict(record, threshold):
    return "pos" if record["spend"] >= threshold else "neg"

def evaluate(test, threshold):
    correct = sum(1 for r in test if predict(r, threshold) == r["label"])
    return round(correct / len(test), 4)

train, test = train_test_split(DATASET, 0.3, 42)
print(len(train), len(test))
t = train_threshold(train)
print("threshold:", t)
print("test accuracy:", evaluate(test, t))
# Output:
# 7 3
# threshold: 280
# test accuracy: 0.6667

A seeded split is reproducible

import random

def train_test_split(records, test_fraction, seed):
    random.seed(seed)
    shuffled = records[:]
    random.shuffle(shuffled)
    n_test = round(len(shuffled) * test_fraction)
    return shuffled[:-n_test], shuffled[-n_test:]

data = [{"spend": s} for s in range(10)]
a, _ = train_test_split(data, 0.3, 42)
b, _ = train_test_split(data, 0.3, 42)   # same seed
c, _ = train_test_split(data, 0.3, 99)   # different seed
print(a == b)   # True  — same seed, same split
print(a == c)   # False — different seed
# Output:
# True
# False

A perfect threshold scores 1.0 on a clean dataset

def predict(record, threshold):
    return "pos" if record["spend"] >= threshold else "neg"

def evaluate(test, threshold):
    correct = sum(1 for r in test if predict(r, threshold) == r["label"])
    return round(correct / len(test), 4)

clean = [
    {"spend": 450, "label": "pos"}, {"spend": 210, "label": "pos"},
    {"spend": 120, "label": "neg"}, {"spend": 30,  "label": "neg"},
]
print(evaluate(clean, 200))   # threshold 200 splits it perfectly
print(predict({"spend": 300}, 200), predict({"spend": 100}, 200))
# Output:
# 1.0
# pos neg

Common mistakes & gotchas

Never evaluate on training data

Score the model on the data it learned from and you measure memorisation, not skill — accuracy looks great and means nothing. Split first, train on one half, evaluate on the other. The held-out test set is the only honest scoreboard.

No seed = a different split every run

random.shuffle without random.seed(...) reorders differently each run, so your train/test split — and every metric downstream — changes between runs. Set a fixed seed for reproducible experiments; you can't compare two models if the data underneath them keeps moving.

Best-on-train isn't best-on-test

The threshold that maxes training accuracy can score worse on the test set — that's overfitting in miniature, and on a tiny test set it's almost guaranteed (here the learned 280 scores 0.6667 on 3 test rows). Real evaluation needs enough test data, or cross-validation, to be trustworthy.

Why it matters

Train → predict → evaluate is the heartbeat of every ML system, from a two-line threshold to a billion-parameter model. The discipline of holding out test data and measuring honestly is what stops you from shipping a model that aced its own homework and fails in the wild — and it's exactly the loop the boss-build pipeline wires together end to end.