Dot Product and Similarity

In ML, a vector is a fixed-length list where every position has a consistent meaning. Position 0 might mean "bass intensity" for every song in the database. Position 1 might mean "tempo" for every song. If you swap the positions, comparison breaks completely.

This is called a feature vector. In ML, you will represent customers, songs, words, images, and loan applications all as feature vectors. The dot product measures similarity between any two of them.

Think of it like a compatibility score between two people on a dating app. Both people rate 5 things on a scale of 1–10: outdoors, movies, cooking, travel, fitness. Person A: [8, 3, 7, 9, 6]. Person B: [7, 2, 8, 10, 5].

To find compatibility: multiply each matching pair (8×7, 3×2, 7×8, 9×10, 6×5) and add them up (56 + 6 + 56 + 90 + 30 = 238). A high number means they care about the same things. This IS the dot product.

The dot product rewards agreement. When both vectors are high at the same position, that position contributes a large product. When one is high and the other is low, the contribution is small.

import numpy as np

# ── Manual dot product — every step visible ───────────────────────────
hiphop   = [0.9, 0.7, 0.8, 0.1, 0.5]   # bass, tempo, energy, acoustic, vocal
similar  = [0.8, 0.6, 0.9, 0.2, 0.4]
classical= [0.1, 0.2, 0.1, 0.9, 0.8]

def dot_product_manual(a, b):
    """
    Step 1: multiply each matching position
    Step 2: sum all the products
    That is the entire dot product operation.
    """
    assert len(a) == len(b), "Vectors must be same length"
    products = [a[i] * b[i] for i in range(len(a))]
    print(f"  Products at each position: {[round(p, 3) for p in products]}")
    total = sum(products)
    return total

print("Dot product — hip-hop vs similar:")
score_similar   = dot_product_manual(hiphop, similar)
print(f"  Result: {score_similar:.3f}\n")

print("Dot product — hip-hop vs classical:")
score_classical = dot_product_manual(hiphop, classical)
print(f"  Result: {score_classical:.3f}\n")

print(f"Conclusion: similar score ({score_similar:.3f}) > classical score ({score_classical:.3f})")
print("→ Spotify recommends 'similar track' over 'classical piano'")

# ── NumPy way — what you use in production ─────────────────────────────
a = np.array(hiphop)
b = np.array(similar)
c = np.array(classical)

print(f"\nNumPy dot product:")
print(f"  np.dot(hiphop, similar):   {np.dot(a, b):.3f}")
print(f"  a @ b (same thing):        {a @ b:.3f}")  # @ is the dot product operator
print(f"  np.dot(hiphop, classical): {np.dot(a, c):.3f}")

Think of two arrows. Arrow A is 10cm long pointing northeast. Arrow B is 2cm long also pointing northeast. They point in exactly the same direction — they represent the same preferences. But if you measure their dot product, Arrow A wins simply because it is longer.

Cosine similarity fixes this by dividing the dot product by the length of both vectors — it strips away magnitude and measures only direction. Two arrows pointing northeast give cosine similarity = 1.0 regardless of how long they are.

||a|| is the "length" (norm) of vector a: the square root of the sum of all values squared. It is always a positive number.

import numpy as np

# ── The magnitude (norm) problem ──────────────────────────────────────
# Two users with same taste but different activity levels
user_active  = np.array([0.9, 0.8, 0.7, 0.1, 0.2]) * 10   # active user — bigger numbers
user_casual  = np.array([0.9, 0.8, 0.7, 0.1, 0.2])          # casual user — same direction!
restaurant   = np.array([0.8, 0.9, 0.6, 0.2, 0.3])

print("Raw dot product (confused by magnitude):")
print(f"  active user  · restaurant: {np.dot(user_active, restaurant):.2f}")
print(f"  casual user  · restaurant: {np.dot(user_casual, restaurant):.2f}")
print("  Active user scores 10× higher — but they have IDENTICAL taste!")

# ── Vector norm — what is the 'length' of a vector? ───────────────────
def vector_norm(v):
    """
    The length (magnitude) of a vector.
    = sqrt(v[0]² + v[1]² + v[2]² + ...)
    Called the L2 norm or Euclidean norm.
    """
    return np.sqrt(np.sum(v ** 2))

print(f"\nVector norms:")
print(f"  ||user_active||: {vector_norm(user_active):.4f}")
print(f"  ||user_casual||: {vector_norm(user_casual):.4f}")
print(f"  Ratio: {vector_norm(user_active) / vector_norm(user_casual):.1f}× — same direction, 10× length")

# ── Cosine similarity — fixes the magnitude problem ────────────────────
def cosine_similarity(a, b):
    """
    cosine_similarity = dot_product(a, b) / (norm(a) * norm(b))

    Step 1: compute the raw dot product
    Step 2: divide by the product of both norms
    Result: always between -1 and +1, magnitude removed
    """
    dot    = np.dot(a, b)
    norm_a = np.linalg.norm(a)   # same as vector_norm(a)
    norm_b = np.linalg.norm(b)
    return dot / (norm_a * norm_b)

print("\nCosine similarity (magnitude-independent):")
print(f"  active user  · restaurant: {cosine_similarity(user_active, restaurant):.4f}")
print(f"  casual user  · restaurant: {cosine_similarity(user_casual, restaurant):.4f}")
print("  Identical! Cosine similarity sees they have the same taste.")

# ── Real example: song similarity ─────────────────────────────────────
hiphop    = np.array([0.9, 0.7, 0.8, 0.1, 0.5])
similar   = np.array([0.8, 0.6, 0.9, 0.2, 0.4])
classical = np.array([0.1, 0.2, 0.1, 0.9, 0.8])

print("\nCosine similarity — song comparison:")
print(f"  hip-hop vs similar:   {cosine_similarity(hiphop, similar):.4f}  ← high, similar genre")
print(f"  hip-hop vs classical: {cosine_similarity(hiphop, classical):.4f}  ← low, different genre")

# ── sklearn version — what you use in production ──────────────────────
from sklearn.metrics.pairwise import cosine_similarity as sk_cosine

# sklearn expects 2D arrays (matrices), not 1D vectors
X = np.array([hiphop, similar, classical])
similarity_matrix = sk_cosine(X)
labels = ['hip-hop', 'similar', 'classical']

print("\nFull similarity matrix (sklearn):")
print(f"{'':12}", end='')
for l in labels: print(f"{l:12}", end='')
print()
for i, row_label in enumerate(labels):
    print(f"{row_label:12}", end='')
    for j in range(len(labels)):
        print(f"{similarity_matrix[i,j]:.4f}      ", end='')
    print()

import numpy as np

# ── Dot product in a neural network layer ────────────────────────────
# One neuron: takes an input vector, has a weight vector, computes dot product
# Think of weights as "how much does this neuron care about each input feature"

def neuron_forward(inputs, weights, bias):
    """
    One neuron's computation:
    1. dot product of inputs and weights (how much each input matters)
    2. add bias (shift the output)
    3. activation function applied later (not shown here)
    """
    return np.dot(inputs, weights) + bias

# A delivery time prediction neuron
# It has learned: distance and prep time matter most, traffic matters some
inputs  = np.array([5.2, 8.0, 22.0])   # distance_km, traffic_score, prep_min
weights = np.array([0.4, 0.15, 0.3])   # learned weights
bias    = 8.6

raw_output = neuron_forward(inputs, weights, bias)
print(f"Neuron output (predicted delivery time): {raw_output:.1f} min")

# ── Attention in transformers (simplified preview) ─────────────────────
# Each word has a Query vector (what it's looking for)
# and a Key vector (what it offers to others)
# Attention score = Query · Key

# Sentence: "The quick brown fox"
# Simplified 3D embeddings for illustration
words = ['The', 'quick', 'brown', 'fox']
# Query vectors (what each word is looking for)
queries = np.array([
    [0.1, 0.9, 0.2],   # 'The' — looks for: noun
    [0.8, 0.3, 0.1],   # 'quick' — looks for: what it modifies
    [0.7, 0.4, 0.2],   # 'brown' — looks for: what it modifies
    [0.2, 0.8, 0.9],   # 'fox' — looks for: its adjectives
])
# Key vectors (what each word offers)
keys = np.array([
    [0.1, 0.2, 0.3],   # 'The' — offers: article
    [0.6, 0.5, 0.1],   # 'quick' — offers: adjective
    [0.7, 0.4, 0.1],   # 'brown' — offers: adjective
    [0.3, 0.9, 0.8],   # 'fox' — offers: noun
])

# How much does 'fox' attend to each word?
fox_query = queries[3]
attention_scores = [np.dot(fox_query, keys[i]) for i in range(4)]
print(f"\nAttention scores for 'fox':")
for word, score in zip(words, attention_scores):
    bar = '█' * int(score * 20)
    print(f"  fox → {word:<8}: {bar} {score:.3f}")
# Fox should attend most to 'quick' and 'brown' (its adjectives)

# ── Embedding similarity search ────────────────────────────────────────
# Find most similar products to a query in an embedding space
np.random.seed(42)
product_names = [
    'Nike Running Shoes', 'Adidas Sneakers', 'Formal Oxford Shoes',
    'Flip Flops', 'Sports Socks', 'Running Shorts',
]
# Simulate product embeddings (in real life these come from a trained model)
product_embeddings = np.random.randn(6, 8)

# Query: user just looked at "Nike Running Shoes"
query_embedding = product_embeddings[0]

# Compute cosine similarity to all products
from sklearn.metrics.pairwise import cosine_similarity
sims = cosine_similarity(
    query_embedding.reshape(1, -1),
    product_embeddings
)[0]

print("\nProducts most similar to 'Nike Running Shoes':")
ranked = sorted(zip(product_names, sims), key=lambda x: x[1], reverse=True)
for name, score in ranked:
    marker = ' ← query' if name == 'Nike Running Shoes' else ''
    print(f"  {name:<25}: {score:.4f}{marker}")

Two cities on a map: New York (18.97°N, 72.83°E) and Boston (18.52°N, 73.85°E). The Euclidean distance between them as points on the map is roughly √((18.97−18.52)² + (72.83−73.85)²) — just like the Pythagorean theorem applied to their coordinate difference. This is Euclidean distance.

In ML, "distance" works the same way but in higher dimensions — instead of 2 coordinates you might have 512 coordinates (features). The formula is identical, just with more terms under the square root.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity, euclidean_distances

# ── Euclidean distance from scratch ───────────────────────────────────
def euclidean_distance(a, b):
    """
    Straight-line distance between two points in n-dimensional space.
    = sqrt( sum of (aᵢ - bᵢ)² for all i )
    Pythagorean theorem generalised to n dimensions.
    """
    differences = a - b
    squared     = differences ** 2
    summed      = np.sum(squared)
    return np.sqrt(summed)

# Customer feature vectors (after scaling to 0-1 range)
# [avg_order_value, orders_per_month, avg_rating, distance_preference]
customer_a = np.array([0.8, 0.9, 0.7, 0.3])   # high-value frequent customer
customer_b = np.array([0.7, 0.8, 0.8, 0.4])   # similar to A
customer_c = np.array([0.2, 0.1, 0.6, 0.9])   # very different — low value, rare

print("Euclidean distances:")
print(f"  A to B: {euclidean_distance(customer_a, customer_b):.4f}  ← small, similar customers")
print(f"  A to C: {euclidean_distance(customer_a, customer_c):.4f}  ← large, different customers")

print("\nCosine similarities (same customers):")
print(f"  A to B: {cosine_similarity(customer_a.reshape(1,-1), customer_b.reshape(1,-1))[0,0]:.4f}")
print(f"  A to C: {cosine_similarity(customer_a.reshape(1,-1), customer_c.reshape(1,-1))[0,0]:.4f}")

# ── When they disagree — an important case ─────────────────────────────
# Two customers with exactly the same PATTERN but different AMOUNTS
casual_customer = np.array([0.2, 0.2, 0.6, 0.3])       # same preferences, less active
active_customer = np.array([0.8, 0.8, 0.6, 0.3]) * 4   # 4× more active, same pattern

print("\nCasual vs active customer (same pattern, different magnitude):")
print(f"  Euclidean distance:  {euclidean_distance(casual_customer, active_customer):.4f}")
print(f"  Cosine similarity:   {cosine_similarity(casual_customer.reshape(1,-1), active_customer.reshape(1,-1))[0,0]:.4f}")
print("  Euclidean: they look far apart (because active has larger values)")
print("  Cosine:    they look identical (because they point the same direction)")
print("  → For segmentation by BEHAVIOUR: use cosine")
print("  → For segmentation by ACTIVITY LEVEL: use euclidean")

import numpy as np
from sklearn.preprocessing import normalize
from sklearn.metrics.pairwise import cosine_similarity

# Amazon electronics — simplified feature vectors
# Features: [price_normalised, brand_premium, battery_life, screen_size,
#            camera_quality, performance_score, weight_light]
products = {
    'iPhone 15':         [0.95, 1.0,  0.7,  0.6, 1.0,  0.95, 0.8],
    'Samsung Galaxy S24':[0.85, 0.9,  0.75, 0.65,0.95, 0.9,  0.75],
    'OnePlus 12':        [0.65, 0.7,  0.8,  0.7, 0.85, 0.88, 0.7],
    'Redmi Note 13':     [0.25, 0.4,  0.85, 0.72,0.65, 0.6,  0.65],
    'iPhone SE':         [0.55, 1.0,  0.5,  0.35,0.75, 0.8,  0.9],
    'iPad Air':          [0.75, 1.0,  0.9,  1.0, 0.85, 0.9,  0.5],
    'Samsung Tab S9':    [0.7,  0.85, 0.85, 0.95,0.8,  0.85, 0.45],
    'Boat Earbuds':      [0.1,  0.3,  0.9,  0.0, 0.0,  0.4,  1.0],
}

names       = list(products.keys())
embeddings  = np.array(list(products.values()))

# Normalise so cosine similarity is just a dot product (unit vectors)
# This is the standard approach for any similarity search
embeddings_norm = normalize(embeddings, norm='l2')

# Build the full similarity matrix
sim_matrix = cosine_similarity(embeddings_norm)

def get_similar_products(product_name, top_k=3):
    """Given a product name, return the top_k most similar products."""
    idx   = names.index(product_name)
    sims  = sim_matrix[idx]
    # Sort by similarity, exclude the product itself
    ranked = sorted(
        [(names[i], sims[i]) for i in range(len(names)) if i != idx],
        key=lambda x: x[1], reverse=True,
    )
    return ranked[:top_k]

print("Product similarity recommendations:\n")
for product in ['iPhone 15', 'Redmi Note 13', 'iPad Air', 'Boat Earbuds']:
    print(f"  Customer viewing: {product}")
    for rec, score in get_similar_products(product):
        bar = '█' * int(score * 20)
        print(f"    → {rec:<22}: {bar} {score:.3f}")
    print()

# ── Production note: in real Amazon, embeddings come from a trained
# neural network (not hand-crafted features). The similarity computation
# is identical — just replace the hand-crafted vectors with model embeddings.
print("In production:")
print("  Step 1: train a model to generate product embeddings")
print("  Step 2: store all embeddings in a vector database (FAISS, Pinecone)")
print("  Step 3: at query time: embed the viewed product, find nearest neighbours")
print("  Step 4: return top-k most similar products")

They agree in exactly one case: when both vectors are already unit length (norm equal to 1). Many embedding APIs — sentence-transformers, OpenAI's embedding endpoint, most production recommendation systems — deliberately output unit-normalised vectors precisely so that a raw dot product doubles as cosine similarity, skipping the division step entirely. But that is a deliberate design choice made by the model, not a mathematical law. Feed the dot product two un-normalised vectors — raw term-frequency counts, unscaled user activity vectors — and it reverts to being dominated by magnitude, exactly the active-vs-casual-user problem covered earlier in this module.

Similarity is a proxy for meaning, not a guarantee of it. Two sentences can land close together in embedding space because they share sentence structure, length, domain vocabulary, or formatting — not because they agree in meaning. 'I love this product' and 'I hate this product' can score surprisingly close in some embedding spaces because both are short first-person opinion statements about a product, and the model was never specifically trained to separate them along the sentiment axis. Treat a similarity threshold as something to validate against real examples for your specific model and domain, never as a universal cutoff.

Normalising is a deliberate choice to discard magnitude, and sometimes magnitude is exactly the signal you need. A document's raw term-frequency vector encodes how strongly it discusses a topic — a document mentioning 'kubernetes' forty times is plausibly more about Kubernetes than one mentioning it twice, even though cosine similarity would treat both the same after normalising away the count. Anywhere magnitude carries real information — activity level, confidence, intensity, raw counts — normalising before comparing throws that information away on purpose. Decide based on what magnitude means in your feature space, not by default habit.

In high dimensions, random vectors behave counterintuitively: pick two random directions in a space with hundreds of dimensions and their angle concentrates very close to ninety degrees — almost everything looks almost equally unrelated to everything else. This is the curse of dimensionality showing up in similarity search. Learned embeddings are not random — training pulls semantically related items together and pushes unrelated ones apart, which is what makes similarity search work at all — but the margin between 'related' and 'unrelated' scores is usually much smaller than two or three-dimensional intuition would suggest, which is exactly why production systems validate similarity thresholds empirically instead of guessing.

A single dot product is cheap — the cost problem is doing it against every stored vector. Brute-force search compares a query against all n stored embeddings, at a cost that grows directly with both n and the embedding dimension. That is fine at thousands of vectors and becomes untenable at hundreds of millions. Real vector databases (FAISS, Pinecone, Milvus) do not make the dot product itself faster — they build approximate nearest neighbour indexes (HNSW graphs, inverted-file plus product quantisation) that skip comparing against most of the vectors entirely, trading a small amount of recall for orders-of-magnitude faster lookups.

Cosine similarity when only direction should matter and magnitude is noise — text embeddings, recommendation profiles, anywhere two items with the same 'shape' of preferences should count as similar regardless of scale. Euclidean distance when actual spatial closeness is the thing you care about — K-Means and KNN on properly scaled features, physical measurements. One useful detail worth mentioning unprompted: if every vector is already unit-normalised, cosine similarity, the raw dot product, and squared Euclidean distance all produce the exact same ranking of nearest neighbours — they are monotonic transforms of each other in that special case, which is part of why so many embedding models normalise their output by default.

a times b equals the length of a, times the length of b, times the cosine of the angle between them. Multiplying matching components and summing them is just the computational recipe for that same geometric quantity — you never have to compute an angle directly to get the effect of comparing directions. That is why the dot product of two vectors pointing the same way is large and positive, two perpendicular vectors give zero, and two opposite vectors give a large negative number — the formula is silently encoding the cosine of the angle the whole time.

Normalising at generation time means downstream consumers can use a plain dot product and get cosine similarity for free, skipping the extra division at query time — which matters when you're running that comparison across millions of stored vectors per query. It also unlocks faster index structures built specifically for inner-product search, like FAISS's IndexFlatIP or IVF variants tuned for dot product, rather than needing a more general distance metric at search time. The normalisation cost is paid once, at write time, instead of on every single query.

Brute-force comparison against every stored vector does not scale to that size — the cost grows with both the number of vectors and their dimension, and a billion of them makes that prohibitive per query. In practice you reach for an approximate nearest neighbour index: an HNSW graph that lets you navigate toward close neighbours without visiting most of the dataset, or an inverted-file index combined with product quantisation (the FAISS IVF-PQ approach) that clusters vectors and compresses them to shrink both memory and search time. Both trade a small, tunable amount of recall for a large improvement in latency — the right tradeoff depends on how much a missed near-neighbour actually costs the product.

A few likely culprits. First, embedding spaces are often anisotropic — embeddings cluster inside a narrow cone of the full space rather than spreading out evenly, which inflates the baseline similarity between almost any two items from that model, so 0.95 might not be nearly as extreme relative to that model's typical score distribution as it looks in isolation. Second, the model may be keying off surface features it was trained to encode strongly — shared formatting, domain jargon, sentence length — rather than the semantic content a human reader cares about. The fix in both cases is the same: look at the distribution of similarity scores for your specific model and domain and calibrate a threshold against labelled examples, instead of trusting a raw absolute cosine value.