Chapter 5 · Data / ML Engineer

Validating a batch

batch validation · reject/accept split · error reportingDAT-40232 of 84

A single-record validator is a building block. What a real pipeline needs is a batch validator: feed it the whole dataset, get back the clean rows separated from the broken ones — and ideally a note of *why* each broken row failed.

The shape is the accumulation pattern you already know. Start two empty lists (clean, rejected), loop the records, run validate_record on each, and append it to the right list. Return both as a tuple (clean, rejected).

validate_record itself is a chain of gates: run each check in order and bail out with False the moment one fails; if all pass, return True. Ordering the cheap checks first (presence, then type, then range) means you never call check_range on a value that isn't even a number.

For error reporting, don't return on the first failure — instead collect a list of error messages and keep checking. A row can have more than one thing wrong, and a good report tells you all of them at once.

Syntax

def validate_record(record):
    if not check_required(record, REQUIRED):
        return False
    if not check_types(record, TYPE_SCHEMA):
        return False
    if not check_range(record['age'], 18, 120):
        return False
    return True

def split_batch(records):
    clean, rejected = [], []
    for r in records:
        if validate_record(r):
            clean.append(r)
        else:
            rejected.append(r)
    return clean, rejected

Worked examples

Split a mixed batch into clean / rejected

def check_required(record, fields):
    return all(f in record for f in fields)
def check_types(record, schema):
    for f, t in schema.items():
        if f not in record or not isinstance(record[f], t):
            return False
    return True
def check_range(value, low, high):
    return low <= value <= high

REQUIRED = ["customer_id", "age", "spend"]
TYPE_SCHEMA = {"customer_id": str, "age": int, "spend": (int, float)}

def validate_record(r):
    if not check_required(r, REQUIRED):
        return False
    if not check_types(r, TYPE_SCHEMA):
        return False
    if not check_range(r["age"], 18, 120):
        return False
    if not check_range(r["spend"], 0, 1_000_000):
        return False
    return True

def split_batch(records):
    clean, rejected = [], []
    for r in records:
        (clean if validate_record(r) else rejected).append(r)
    return clean, rejected

batch = [
    {"customer_id": "C001", "age": 34, "spend": 250.0},   # valid
    {"customer_id": "C002", "age": 17, "spend": 80.0},    # age < 18
    {"customer_id": "C003", "age": 45, "spend": -10.0},   # negative spend
    {"customer_id": "C004", "age": 29, "spend": 430.5},   # valid
]
clean, rejected = split_batch(batch)
print(len(clean), len(rejected))
print([r["customer_id"] for r in clean])
print([r["customer_id"] for r in rejected])
# Output:
# 2 2
# ['C001', 'C004']
# ['C002', 'C003']

Collecting WHY a record failed (don't stop at the first error)

def validate_with_errors(record):
    errors = []
    for field in ["customer_id", "age", "spend"]:
        if field not in record:
            errors.append(f"missing field: {field}")
    if "age" in record and isinstance(record["age"], int):
        if not (18 <= record["age"] <= 120):
            errors.append("age out of range")
    return errors

bad = {"customer_id": "C002", "age": 17}
print(validate_with_errors(bad))
# Output:
# ['missing field: spend', 'age out of range']

Common mistakes & gotchas

Returning early kills error reporting

The pass/fail gate returns False on the first failure — great for a yes/no, useless for a report. If you want all the reasons, don't return early: append to an errors list and keep going. Pick the shape that matches what the caller needs.

Gate order matters

Run presence before type, and type before range. check_range(record['age'], ...) will raise KeyError if age is missing, or TypeError if it's a string. The earlier gates guarantee the later ones get something safe to work on.

A tuple isn't two return statements

return clean, rejected returns ONE value — a tuple (clean, rejected). The caller unpacks it with clean, rejected = split_batch(...). Forgetting to unpack (result = split_batch(...)) leaves you with the whole tuple, and result.append(...) then fails.

Why it matters

Real datasets are never clean. A batch validator that quarantines the bad rows — and reports why — lets the pipeline keep running on the good 95% instead of crashing on row 4,000. The clean/rejected split is the standard contract every ingestion stage in production exposes.