Chapter 5 · Data / ML Engineer

Schema validation

validate record · required fields · type + range guardsDAT-40131 of 84

Before any model — or any pipeline stage — touches your data, the data has to be valid. A record (a dict of fields) is valid when every field you need is present, each value is the right type, and the numbers fall in a sensible range. Catching a bad row at the door is far cheaper than debugging a corrupted training run later.

Validation breaks into three independent checks:

Required fields — is every key you depend on actually in the dict? (A field can be present with a value of None — present and missing are different things.)
Types — is age an int, is name a str? You check with isinstance(value, expected_type).
Range — is the number between sane bounds? low <= value <= high as a single chained comparison.

Each check is a small function returning True / False. Keeping them separate means you can test each one on its own and combine them however a given record demands.

*In production you'd reach for a library like Pydantic to declare a schema. Writing it by hand first makes the logic transparent — you can see exactly what "valid" means.*

Syntax

def check_required(record, required_fields):
    for field in required_fields:
        if field not in record:
            return False
    return True

def check_types(record, schema):  # schema: {"age": int, ...}
    for field, expected_type in schema.items():
        if field not in record:
            return False
        if not isinstance(record[field], expected_type):
            return False
    return True

def check_range(value, low, high):
    return low <= value <= high   # inclusive both ends

Worked examples

Required fields — present vs missing

def check_required(record, required_fields):
    for field in required_fields:
        if field not in record:
            return False
    return True

record = {"customer_id": "C001", "age": 34, "spend": 250.0}
print(check_required(record, ["customer_id", "age", "spend"]))  # True
print(check_required(record, ["customer_id", "email"]))         # False
# Output:
# True
# False

Type check — a type, or a tuple of allowed types

def check_types(record, schema):
    for field, expected_type in schema.items():
        if field not in record:
            return False
        if not isinstance(record[field], expected_type):
            return False
    return True

schema = {"age": int, "spend": (int, float)}   # spend may be int OR float
print(check_types({"age": 34, "spend": 250.0}, schema))   # True
print(check_types({"age": "34", "spend": 250.0}, schema)) # False (age is str)
print(check_types({"spend": 250.0}, schema))              # False (age missing)
# Output:
# True
# False
# False

Range check — bounds are inclusive

def check_range(value, low, high):
    return low <= value <= high

print(check_range(34, 18, 120))   # True
print(check_range(18, 18, 120))   # True  (low bound included)
print(check_range(17, 18, 120))   # False
# Output:
# True
# True
# False

Common mistakes & gotchas

Present-but-None is still present

field not in record is False when the key exists, even if its value is None. So check_required passes a record like {'age': None} — the field IS there. If you also want to reject None values, that's a *separate* check; don't fold it into the required-fields test.

bool is a subclass of int

In Python isinstance(True, int) is True — booleans are ints under the hood. So check_types({'age': True}, {'age': int}) passes. If a field must be a real number and not a bool, you have to exclude bool explicitly (e.g. type(v) is int), because isinstance won't.

Pass the type, not an instance

The schema maps to the type *object* — int, str, (int, float) — not an example value. {"age": 0} is wrong; isinstance(34, 0) raises TypeError: isinstance() arg 2 must be a type. Use the bare type name.

Why it matters

Garbage in, garbage out is the oldest law in data work. A model trained on rows where 'age' is sometimes the string "34" and sometimes 34 learns nonsense. Validation is the unglamorous gate that keeps every downstream stage — features, training, metrics — working on data it can trust.