Chapter 12 · AI Architect / Lead

Plausible-but-insecure

AI security reviewAIR-110381 of 84

AI reaches for the shortest one-liner that handles every case — and the shortest one-liner is often a security hole. Asked to 'restore a saved session from bytes', a model writes pickle.loads(raw) because pickle handles any Python object. It round-trips a real dict perfectly. It is also a remote code execution vulnerability.

Here is exactly why: pickle is not a data format, it's an *object-reconstruction* format. A pickle byte stream can carry a __reduce__ instruction that tells the unpickler which callable to invoke during loading. An attacker bakes (os.system, ('rm -rf ...',)) into a forged payload, and pickle.loads runs it while 'just loading a dict'. Never unpickle data you didn't create yourself. The same warning applies to eval, exec, and yaml.load (use yaml.safe_load) on untrusted input.

json is the safe swap for plain data. json.loads can only ever produce inert values — dict, list, str, int, float, bool, None. It has no hook to execute code, so a forged byte string is just garbage it rejects with a JSONDecodeError. Sessions are dicts of strings and numbers; they never needed pickle's power.

The review reflex: when AI code deserialises, evaluates, or executes anything that came from outside the program — a request, a file, a token — ask *what's the worst this input could do?* If the answer is 'run arbitrary code', reject it and reach for a data-only parser.

Syntax

# AI wrote this — works on real data, RUNS code on a forged payload:
import pickle
def load_session(raw):
    return pickle.loads(raw)   # arbitrary code execution on untrusted bytes

# Safe replacement — JSON parses DATA, never executes it:
import json
def load_session(raw):
    try:
        data = json.loads(raw)
    except Exception:
        raise ValueError("invalid session")
    if not isinstance(data, dict):
        raise ValueError("invalid session")
    return data

# Rule: never pickle.loads / eval / exec / yaml.load on untrusted input.

Worked examples

The safe JSON loader accepts real data, rejects junk

import json

def load_session(raw):
    try:
        data = json.loads(raw)
    except Exception:
        raise ValueError('invalid session')
    if not isinstance(data, dict):
        raise ValueError('invalid session')
    return data

raw = json.dumps({'user': 'sam', 'level': 4}).encode()
print(load_session(raw))   # {'user': 'sam', 'level': 4}

Malformed or wrong-shape input is rejected, never executed

import json

def load_session(raw):
    try:
        data = json.loads(raw)
    except Exception:
        raise ValueError('invalid session')
    if not isinstance(data, dict):
        raise ValueError('invalid session')
    return data

for bad in (b'not json {{{', json.dumps([1, 2, 3]).encode()):
    try:
        load_session(bad)
    except ValueError:
        print('rejected:', bad[:12])
# Output:
# rejected: b'not json {{{'
# rejected: b'[1, 2, 3]'

Why JSON is safe — it can only ever produce inert data

import json
# A string that, in a pickle, could carry an exploit. To JSON it is just text:
value = json.loads('"echo pwned"')
print(type(value), repr(value))   # <class 'str'> 'echo pwned'
# json has no __reduce__ hook — nothing here can run. It's data, not behaviour.

Common mistakes & gotchas

Treating pickle as a data format

pickle reconstructs arbitrary Python objects, including by *calling functions* during load via __reduce__. On data you created it's convenient; on anything from outside (a request, a cache, a file an attacker can touch) it is arbitrary code execution. Use it only on data you produced and never let cross a trust boundary.

Assuming a security flaw will show up as a failing test

A SQL-injection string or a malicious pickle often passes every functional test — the code 'works'. Security defects don't fail your happy-path suite; they require you to think adversarially: *what's the worst this input could do?* A green test is not a security clearance.

yaml.load and eval wear the same trap

AI also suggests yaml.load(s) and eval(s) for parsing — both execute code on hostile input. The safe forms are yaml.safe_load and, for data, json.loads / ast.literal_eval. Any 'parse this string' that can invoke code is the wrong tool for untrusted input.

Why it matters

AI optimises for the slick, universal one-liner — which is frequently the insecure one. The reviewer who knows *why* `pickle.loads` on untrusted bytes is code execution catches the breach the test suite never will. Security is the clearest case where verifying AI code is a hard skill no green checkmark replaces.