Chapter 9 · AI Engineer

Embeddings and cosine similarity

text → vector · cosine similarity · ranking by relevanceAIE-80361 of 84

An embedding is a piece of text turned into a list of numbers — a vector. The trick is that texts about similar things end up as vectors pointing in similar *directions*, so a computer can measure "how related are these two pieces of text?" with arithmetic.

The standard measure is cosine similarity: the cosine of the angle between two vectors. It ignores length and looks only at direction, which is exactly what you want for text. The scale is intuitive:

1.0 — same direction (as similar as it gets, e.g. a vector with itself).
0.0 — at right angles, sharing nothing.
(Negative values are possible in general, but bag-of-word counts are never negative, so here you stay between 0 and 1.)

The formula is the dot product divided by the product of the two magnitudes:

dot(a, b) = sum of a[i] * b[i] across every position.
magnitude(v) = sqrt(dot(v, v)) — the vector's length.
cosine(a, b) = dot(a, b) / (magnitude(a) * magnitude(b)).

Guard the divide: if either vector is all zeros its magnitude is 0, so return 0.0 rather than divide by zero. Real systems get vectors from a neural model over the network; the technique — and this exact formula — is identical whether the numbers come from a tiny word-count stub or a billion-parameter model.

Syntax

dot(a, b)        = sum(x * y for x, y in zip(a, b))
magnitude(v)     = sqrt(dot(v, v))
cosine(a, b)     = dot(a, b) / (magnitude(a) * magnitude(b))
# if either magnitude is 0 -> return 0.0 (no divide-by-zero)

Worked examples

Dot product and magnitude (the two primitives)

import math

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def magnitude(v):
    return math.sqrt(dot(v, v))

print(dot([1.0, 2.0, 3.0], [4.0, 5.0, 6.0]))  # 32.0  (4+10+18)
print(magnitude([3.0, 4.0]))                  # 5.0   (the 3-4-5 triangle)

Cosine similarity — verified: identical = 1.0, orthogonal = 0.0

import math

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def cosine_similarity(a, b):
    ma = math.sqrt(dot(a, a))
    mb = math.sqrt(dot(b, b))
    if ma == 0 or mb == 0:
        return 0.0
    return dot(a, b) / (ma * mb)

print(round(cosine_similarity([1.0, 1.0], [1.0, 1.0]), 4))  # 1.0  (identical direction)
print(round(cosine_similarity([1.0, 0.0], [0.0, 1.0]), 4))  # 0.0  (right angle)
print(round(cosine_similarity([1.0, 2.0, 2.0], [2.0, 0.0, 0.0]), 4))  # 0.3333  (2 / (3 * 2))

Embed text, then rank documents by relevance

import math

VOCAB = ["cat", "dog", "kitten", "puppy", "fish", "pet", "vet", "food"]

def embed(text):
    words = text.lower().split()
    return [float(words.count(term)) for term in VOCAB]

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    ma = math.sqrt(sum(x * x for x in a))
    mb = math.sqrt(sum(x * x for x in b))
    if ma == 0 or mb == 0:
        return 0.0
    return dot / (ma * mb)

def rank_by_relevance(query, docs):
    q = embed(query)
    scored = [(doc, cosine_similarity(q, embed(doc))) for doc in docs]
    return sorted(scored, key=lambda pair: pair[1], reverse=True)

docs = ["my dog loves the puppy", "the cat is a lovely pet", "fish need clean food"]
for doc, score in rank_by_relevance("cat pet", docs):
    print(round(score, 3), doc)
# Output:
# 1.0 the cat is a lovely pet
# 0.0 my dog loves the puppy
# 0.0 fish need clean food

Common mistakes & gotchas

Divide-by-zero on an empty / out-of-vocab vector

If a text contains no vocab words its vector is all zeros and its magnitude is 0, so the formula would divide by zero. Always guard: if ma == 0 or mb == 0: return 0.0. A zero-similarity is the correct, safe answer for "these share nothing" — it's also what powers the "I don't know" path later in RAG.

Cosine measures direction, not magnitude

Doubling every number in a vector doesn't change its cosine similarity to another — the angle is the same. That's a feature (a long document and a short one about the same topic still match) but a surprise if you expected longer = more similar. If raw magnitude matters to you, cosine is the wrong tool.

zip() stops at the shorter vector

dot() built on zip(a, b) silently truncates to the shorter list, so comparing vectors of different lengths gives a wrong-but-no-error answer. Embeddings must share one fixed dimension (here, the length of VOCAB). If your scores look bizarre, check the vectors are the same length first.

Why it matters

Embeddings are how computers do "find me things like this" — semantic search, recommendations, deduplication, and the retrieval half of RAG all rest on turning text into vectors and comparing them with cosine similarity. The model that produces the numbers is enormous; the maths that ranks them is the few lines above, and owning that formula means you understand what every vector database is doing under the hood.