Vectors, Matrices and Tensors

This page teaches three things: what scalars, vectors, matrices, and tensors are (with pictures), what the key operations do and why ML needs them, and how to work with them in NumPy — the library all of ML is built on.

import numpy as np

# A scalar is just a regular Python number
loss = 0.34
learning_rate = 0.001
accuracy = 0.92

# Or a 0-dimensional NumPy array
scalar = np.array(7)
print(scalar.shape)   # ()  ← empty shape, no dimensions
print(scalar.ndim)    # 0

# One order = one vector
order = np.array([3.2, 2, 15, 7])
print(order.shape)   # (4,)  ← 4 features, 1 dimension
print(order.ndim)    # 1
print(order[0])      # 3.2   ← first feature (distance)

# Vector arithmetic — works element-by-element
order_a = np.array([3.2, 2, 15, 7])
order_b = np.array([1.1, 3, 10, 5])

diff = order_a - order_b
print(diff)   # [2.1, -1, 5, 2]  ← difference for each feature

# Scalar × vector — every element multiplied by the scalar
scaled = 2 * order_a
print(scaled)  # [6.4, 4, 30, 14]

# Build the feature matrix
np.random.seed(42)
n = 1000

X = np.column_stack([
    np.random.uniform(0.5, 8.0, n),   # distance_km
    np.random.randint(1, 5, n),        # time_of_day
    np.random.uniform(5, 25, n),       # prep_time
    np.random.uniform(1, 10, n),       # traffic
])

print(X.shape)    # (1000, 4)  ← rows × columns
print(X.ndim)     # 2

# Indexing
print(X[0])       # first order  → [3.74, 1, 21.4, 3.7]
print(X[0, 0])    # first order, first feature → 3.74
print(X[:, 0])    # ALL orders, first feature (distance column) → shape (1000,)
print(X[:5, :])   # first 5 orders, all features → shape (5, 4)

# Shape tells you: (rows, columns)
rows, cols = X.shape
print(f"{rows} orders, {cols} features each")

# 3D tensor — one RGB image (224×224 pixels, 3 colour channels)
image = np.random.randint(0, 256, size=(224, 224, 3))
print(image.shape)   # (224, 224, 3)
print(image.ndim)    # 3

# 4D tensor — batch of 32 images
batch = np.random.randint(0, 256, size=(32, 224, 224, 3))
print(batch.shape)   # (32, 224, 224, 3)

# The shape tells the whole story:
# (batch_size, height, width, channels)
#  32          224     224    3

# In PyTorch, channels come SECOND: (batch, channels, height, width)
# batch_pytorch = np.random.randn(32, 3, 224, 224)
# This is a common source of shape errors when switching between libraries.

features = np.array([3, 1, 4, 2])
weights  = np.array([0.5, 0.3, 0.8, 0.2])

# Method 1: np.dot()
result = np.dot(features, weights)
print(result)   # 5.4

# Method 2: @ operator (preferred in modern code)
result = features @ weights
print(result)   # 5.4

# This is literally what one neuron does:
# multiply each input by its weight, sum everything up
# then add a bias term: output = dot(features, weights) + bias

# 1000 training examples, 4 features each
X = np.random.randn(1000, 4)   # shape (1000, 4)

# Weight matrix for a layer with 64 neurons
W = np.random.randn(4, 64)     # shape (4, 64)
b = np.random.randn(64)        # bias for each neuron

# One matrix multiply = entire layer's output for ALL examples at once
output = X @ W + b
print(output.shape)   # (1000, 64)

# Shape rule check: (1000, 4) @ (4, 64) → inner dims match → (1000, 64) ✓

# The most common error in all of deep learning:
# X.shape = (1000, 4), W.shape = (64, 4)  ← WRONG ORDER
# output = X @ W  → ValueError: matmul dimensions don't match
# Fix: W = W.T   ← transpose W to (4, 64) then it works

A = np.array([[1, 2, 3],
              [4, 5, 6]])
print(A.shape)    # (2, 3)

AT = A.T           # or np.transpose(A)
print(AT.shape)   # (3, 2)

print(AT)
# [[1, 4],
#  [2, 5],
#  [3, 6]]

# Real use: fixing shape mismatches
# If model expects (batch, features) but you have (features, batch):
X_fixed = X.T    # flip it

# Normalise 1000 orders: subtract mean, divide by std
X = np.random.randn(1000, 4)   # 1000 orders, 4 features

mean = X.mean(axis=0)   # shape (4,)  ← one mean per feature column
std  = X.std(axis=0)    # shape (4,)

# Broadcasting: (1000,4) - (4,) works automatically
# NumPy treats (4,) as if it were (1,4) then tiles it 1000 times
X_norm = (X - mean) / std
print(X_norm.shape)        # (1000, 4)  ← same shape, now normalised
print(X_norm.mean(axis=0)) # [~0, ~0, ~0, ~0] ← means now near zero
print(X_norm.std(axis=0))  # [~1, ~1, ~1, ~1] ← stds now near one

# Without broadcasting you'd need:
# mean_matrix = np.tile(mean, (1000, 1))  ← wasteful copy

The most important debugging habit: whenever you get a shape error or unexpected result, immediately print the shape of every array involved. 90% of ML bugs come down to a wrong shape somewhere in the pipeline.

# What a real code review comment on this topic looks like
# PR: "Add prep-time feature to delivery model"

# Reviewer comment:
# "X_train.shape is (48000, 5) but you appended the new feature as
#  a separate array -- did you mean to np.column_stack it back into X
#  before calling model.fit()? Right now this line silently trains
#  on the OLD 4-feature matrix and the new column is never used."

import numpy as np

X_train = np.random.randn(48000, 4)          # existing 4 features
prep_time_feature = np.random.randn(48000)   # new column, shape (48000,)

# Wrong -- new feature never reaches the model
# model.fit(X_train, y_train)

# Right -- combine into one (48000, 5) matrix first
X_train_v2 = np.column_stack([X_train, prep_time_feature])
print(X_train_v2.shape)   # (48000, 5)  <- now the model actually sees it

Everything on this page reduces to four operations — dot product, matrix multiply, transpose, and broadcasting — and NumPy or PyTorch executes every one of them for you. What separates a productive ML engineer from a stuck one is not the ability to multiply matrices by hand; it is the habit of knowing what a shape like (1000, 4) represents and noticing immediately when a shape stops making sense. That is a reading skill, not a calculation skill, and it is entirely learnable without a math degree.

A tensor is just the general word for an array with any number of dimensions — a scalar is a 0-dimensional tensor, a vector is 1-dimensional, a matrix is 2-dimensional, and 'tensor' is simply what people call it once you go past two. PyTorch and TensorFlow named their core data type 'Tensor' because it is the one word that covers every case above without needing a different name for each dimension count — not because the underlying object is mathematically exotic.

In raw Python, yes — a 4D array would mean four nested loops to touch every element. NumPy and PyTorch exist specifically so you almost never write that loop. Broadcasting, the colon-slicing shown throughout this page, and vectorised operations let you write one line — X.mean(axis=0), X @ W — that operates across every dimension at once, at C or GPU speed instead of Python speed. Reaching for a manual loop over array elements is usually the sign something is being done the slow, wrong way.

Every row in a training table is a vector. Every dataset fed to model.fit() is a matrix. Every batch of images or token sequence a transformer processes is a tensor. There is no layer of the ML stack — from a scikit-learn script to a trillion-parameter LLM — where this structure is optional or theoretical. The vocabulary might disappear behind a library call, but the data underneath it never stops being exactly this.

High-level APIs hide the matrix multiplication, not the shape requirements. model.fit(X, y) will still raise an error the moment X and y disagree on the number of rows, or a Dense layer will still fail the moment its input dimension does not match the previous layer's output. This page's own closing rule holds regardless of which library sits on top: the large majority of ML bugs people hit are shape mismatches, and the fix is always the same — print the shape and look.

They are the same idea extended by one dimension each time. A scalar is a single number with zero dimensions. A vector is a one-dimensional list of numbers — one training example, or one word embedding. A matrix is a two-dimensional grid — rows and columns, like an entire training dataset where each row is an example and each column is a feature. A tensor is the general term for any of these — people specifically use the word 'tensor' once you go beyond two dimensions, like a batch of images with shape (batch, height, width, channels).

A single colour image is not just a flat grid of pixels — it has a colour channel dimension on top of height and width, typically red, green, and blue, so one image alone is already 3-dimensional: (height, width, channels). Training almost never happens on one image at a time; a batch of, say, 32 images adds a fourth dimension in front, giving a shape like (32, 224, 224, 3). The extra dimensions are not decorative — they are exactly what lets a GPU process 32 images with one matrix operation instead of 32 separate ones.

Broadcasting lets NumPy or PyTorch apply an operation between arrays of different shapes by conceptually stretching the smaller one to match the larger one, without ever actually copying that data in memory. The classic example is normalising a dataset: subtracting a (n_features,) mean vector from an (n_samples, n_features) matrix in a single line, instead of writing a loop that repeats the subtraction once per row. It matters for performance because that single vectorised line runs as compiled, often parallelised code, while the equivalent Python loop would run orders of magnitude slower.

First, print both shapes directly — print(A.shape, B.shape) — because the error message alone rarely tells you which of the two operands is wrong. Then apply the shape rule: for A @ B to work, A's number of columns must equal B's number of rows, and the result takes A's row count and B's column count. If the inner dimensions don't match, the fix is almost always either transposing one operand with .T or checking further upstream for where that array's shape was built incorrectly in the first place — a reshape, a concatenation, or a feature that got appended as a separate array instead of joined into the matrix.

Representing an entity as a vector of, say, 768 numbers lets a model encode many aspects of meaning simultaneously — direction and magnitude in that 768-dimensional space becomes a stand-in for semantic similarity. Two words with similar meanings end up with vectors that point in similar directions, so distance metrics like cosine similarity or Euclidean distance between two embedding vectors give a numeric measure of how related two things are — which is exactly the mechanism behind recommendation systems suggesting similar products and search engines matching a query to semantically related documents, not just exact keyword matches.