Advanced

PyTorch Fundamentals — Tensors, Autograd and Modules

Build a practical mental model of PyTorch tensors, automatic differentiation, modules, and data loading before writing a complete training loop.

Est. 24–30 min Updated Suggest a fix
A framework that keeps the math visible

PyTorch gives you tensor operations, gradients, and reusable layers—not a black box that trains itself.

In the neural-network lessons, you computed a forward pass and used the chain rule to understand how parameters change. PyTorch does the bookkeeping for those derivatives. You describe operations on tensors, mark the values that should be learned, calculate a loss, and ask autograd for gradients. You still choose the model, objective, optimizer, data split, and checks that make training trustworthy.

This lesson lays the groundwork: what a tensor represents, how shape and dtype affect an operation, how data moves between CPU and accelerators, what autograd records, and how nn.Module organizes trainable components. We finish by building a Dataset and DataLoader for a small tabular problem. The next lesson assembles these pieces into a full training and validation loop.

🎯 Pro Tip
Prerequisite: be comfortable with Python functions, NumPy-style arrays, matrix multiplication, derivatives, and the forward/backward pass of a small neural network. You do not need to memorize every PyTorch method; learn to inspect shapes and read the API documentation when details matter.
01 — The tensor model

A tensor is an n-dimensional array with explicit shape, dtype, and device.

A scalar is a zero-dimensional tensor, a vector has one axis, a matrix has two, and images or batches add more axes. The shape tells you the size of each axis; it does not tell you what those axes mean. For a batch of tabular examples, a common convention is [batch, features]. For an image batch, a common channel-first layout is [batch, channels, height, width]. Read the operation's expected layout rather than assuming every library uses the same convention.

python
import torch

score = torch.tensor(0.7)                        # scalar: []
features = torch.tensor([2.0, 5.0, 1.0])        # [3]
rows = torch.tensor([[2.0, 5.0], [1.0, 4.0]])  # [2, 2]

print(rows.shape)       # torch.Size([2, 2])
print(rows.ndim)        # 2
print(rows.dtype)       # commonly torch.float32 when specified below

# Prefer explicit dtype for model inputs and targets.
x = torch.tensor([[2, 5], [1, 4]], dtype=torch.float32)
y = torch.tensor([0, 1], dtype=torch.long)      # class-index labels

Shape errors are meaning errors in disguise

Suppose there are 64 customers and each has 12 numeric features. The input has shape [64, 12]. A linear layer with 32 output units maps it to [64, 32]: the batch axis stays in place, while the feature axis changes. If you accidentally transpose the input to [12, 64], the model may fail loudly—or worse, produce a plausible but semantically wrong computation if dimensions happen to line up. Print shapes at boundaries while developing.

Read dimensions left to right

Write down what each axis means next to important tensors: batch × features, batch × classes, or batch × channels × height × width. This small habit catches many broadcasting and reshape mistakes before they become training mysteries.

Dtypes determine which values and operations are representable

Neural-network inputs and weights are usually floating-point tensors such as float32. Integer class labels for losses such as cross entropy are usually int64 (shown as torch.long). A mismatch can trigger a useful error, but implicit conversions can also hide bugs. Keep labels in the dtype expected by the objective: class indices are integer targets; regression targets are floating point.

python
logits = torch.randn(4, 3, dtype=torch.float32)  # 4 rows, 3 class scores
class_ids = torch.tensor([0, 2, 1, 0], dtype=torch.long)
regression_targets = torch.tensor([1.2, 0.3, 0.8, 2.1], dtype=torch.float32)

assert logits.shape[0] == class_ids.shape[0]
assert logits.dtype.is_floating_point
assert class_ids.dtype == torch.long
02 — Devices and movement

CPU, CUDA GPUs, and Apple MPS are different execution devices; all tensors in one operation must be compatible.

A tensor's device tells PyTorch where its storage and operations live. CPU is the reliable baseline. CUDA enables supported NVIDIA GPUs, while MPS enables supported Apple silicon GPU operations through Metal. Availability depends on the installed PyTorch build, hardware, operating system, and operation support. Detect a usable device at runtime; do not assume a GPU exists because the code runs on a developer machine.

python
if torch.cuda.is_available():
    device = torch.device("cuda")
elif torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")

model = model.to(device)
x_batch = x_batch.to(device)
print(device, x_batch.device)

Moving a model does not automatically move separately created input tensors. Move the model and each batch to the same device before the forward pass. Moving data between CPU and accelerator has a cost, so load and transfer batches deliberately. Keep data loading, metric collection, and checkpoints mindful of CPU/GPU memory.

Device mismatch

If a tensor on cpu is combined with one on cuda or mps, PyTorch will report a device mismatch. Inspect both tensor.device and model's parameter devices; move the missing object explicitly rather than copying all tensors repeatedly inside individual layers.

Views, reshapes, and in-place operations

Operations such as indexing and reshaping can return views that share storage with the original tensor when the layout permits; reshape may return a copy when needed. In-place operations mutate existing storage and can invalidate values autograd needs to compute gradients. Prefer ordinary out-of-place expressions while learning, and use in-place updates only when you understand their memory and gradient implications.

python
matrix = torch.arange(12).reshape(3, 4)
column = matrix[:, 1]
print(matrix.shape, column.shape)  # [3, 4] and [3]

# A safe, explicit transfer for a batch:
x_batch = x_batch.to(device=device, dtype=torch.float32)
# Avoid .item() or .cpu() on every inner-loop step; they can synchronize devices.
03 — Autograd follows the computation

Autograd records operations involving gradient-tracked tensors, then applies the chain rule backward.

A trainable parameter has requires_grad=True. When operations combine it with inputs, PyTorch builds a dynamic computation graph. Calling backward() on a scalar objective computes gradients of that objective with respect to the participating leaf tensors. The gradients are accumulated in each leaf's .grad field. Accumulation is intentional—for example, gradient accumulation can simulate a larger batch—but a normal training iteration clears gradients before computing the next update.

python
w = torch.tensor(3.0, requires_grad=True)
x = torch.tensor(2.0)
loss = (w * x - 5.0) ** 2
loss.backward()

print(loss.item())  # 1.0
print(w.grad)       # derivative: 2 * (3*2 - 5) * 2 = 4

In a model, autograd handles a much larger expression graph: matrix multiplications, nonlinearities, and loss calculation. It does not choose the update rule. An optimizer reads the gradients and updates parameters; this distinction is why backward() and optimizer.step() are separate operations in the next lesson.

Clear gradients and distinguish training from inference

python
optimizer.zero_grad(set_to_none=True)  # clear previous accumulated gradients
logits = model(x_batch)
loss = loss_fn(logits, y_batch)
loss.backward()                      # populate parameter .grad values
optimizer.step()                     # update parameters using those gradients

Gradient tracking can be disabled for validation and prediction to reduce memory use and computation. Use torch.no_grad() or inference mode for computations where gradients are not needed. Do not wrap a training forward pass in a no-gradient context: that prevents autograd from recording the graph needed by backpropagation.

⚠️ Important
A common bug is forgetting to clear gradients. Then gradients from earlier batches add to the current batch's gradients. This is correct only when accumulation is intentional and the loss is scaled appropriately; otherwise, call optimizer.zero_grad() once per update.
04 — Organising a model with nn.Module

An nn.Module registers layers and parameters so PyTorch can find, move, save, and optimize them.

A module's __init__ defines submodules and parameters; forward describes how inputs flow through them. Assigning a layer to an attribute registers it with the module. Registered parameters appear in model.parameters(), move with model.to(device), and are included in a state dictionary. A plain Python list of layers is not automatically registered—use nn.Sequential, nn.ModuleList, or module attributes for components that should be tracked.

python
from torch import nn

class SalesRegressor(nn.Module):
    def __init__(self, n_features: int):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(n_features, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 1),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.network(x)

model = SalesRegressor(n_features=12)
print(sum(p.numel() for p in model.parameters()))

The forward method returns raw model outputs. For multi-class classification, the final layer typically returns logits—unnormalized class scores—and a suitable loss function combines those logits with class-index labels. Do not apply softmax before a loss that expects logits. For a one-output regressor, the output shape is often [batch, 1]; align the target shape intentionally rather than relying on broadcasting.

Parameters versus buffers

Learnable weights are parameters. Some modules also maintain state that should be saved but not optimized, such as BatchNorm's running statistics; these are buffers. Register a persistent non-learned tensor with register_buffer rather than assigning a tensor and hoping the optimizer treats it correctly. Inspect a module's named parameters and buffers when debugging model state.

python
for name, parameter in model.named_parameters():
    print(name, tuple(parameter.shape), parameter.requires_grad)

state = model.state_dict()  # tensors for parameters and persistent buffers
# Save/load state_dicts in a controlled training checkpoint (next lesson).
05 — Dataset and DataLoader

Separate how one example is represented from how batches are sampled and delivered.

A Dataset defines the mapping from an index to an example. A DataLoader groups those examples into batches and can shuffle training examples, use worker processes, and collate individual samples. This separation lets the model consume batches without knowing where the records came from. For a small in-memory dataset, a TensorDataset may be enough; custom datasets are useful for files, images, and on-demand transforms.

python
from torch.utils.data import DataLoader, TensorDataset

# X_train and y_train must be derived after a train/validation split.
train_dataset = TensorDataset(
    torch.as_tensor(X_train, dtype=torch.float32),
    torch.as_tensor(y_train, dtype=torch.float32).reshape(-1, 1),
)
train_loader = DataLoader(
    train_dataset,
    batch_size=64,
    shuffle=True,
    drop_last=False,
)

for features, target in train_loader:
    print(features.shape, target.shape)
    break

The example uses a numeric sales target, so the labels are floating point and shaped [batch, 1]. A classification dataset may instead return an integer class index per example. Shuffle training batches to reduce dependence on record order; keep validation and test evaluation deterministic. Fit preprocessing—imputation values, category encoders, and scalers—on training data only, then apply those fitted transformations to validation and test data.

Reproducibility is more than a seed

Random seeds help compare experiments, but exact reproducibility can still depend on device kernels, workers, library versions, and nondeterministic operations. Record data splits, preprocessing, model configuration, and the environment. The goal is to make the experiment traceable, not to promise identical results across every machine.

06 — Put the pieces together

A small model sanity check catches shape and gradient bugs before a long run.

Before full training, run one batch through the model, calculate the intended loss, backpropagate, and confirm that trainable parameters receive finite gradients. This is not a quality evaluation. It confirms the plumbing: inputs and targets have expected shapes and dtypes, the model output matches the objective, and the graph reaches the parameters.

python
features, target = next(iter(train_loader))
features = features.to(device)
target = target.to(device)
model = model.to(device)

prediction = model(features)
loss = nn.functional.mse_loss(prediction, target)
loss.backward()

assert prediction.shape == target.shape
assert torch.isfinite(loss)
assert any(p.grad is not None for p in model.parameters())

A finite loss on one batch does not mean the model is learning well. It only rules out a class of wiring errors. The next lesson adds epochs, optimizer updates, evaluation mode, validation metrics, early stopping, checkpointing, and a complete tabular example.

🎯 Pro Tip
When an operation fails, reduce the example to one batch and inspect shape, dtype, device, and whether gradients are enabled. Small executable checks are faster to reason about than a full training run.
What comes next

Turn the tensor and module foundations into a disciplined training loop.

Next, train a neural network end to end: compute batch losses, update parameters, switch correctly between training and evaluation, measure validation performance, stop when generalisation stalls, and save the best checkpoint.

Next — Deep Learning
Training Neural Networks in PyTorch — The Training Loop

Build and validate a reproducible tabular training workflow.

Start →

🎯 Key Takeaways

  • ✓A tensor is an array with a shape, dtype, and device. Name the meaning of each axis and check shape compatibility at model boundaries.
  • ✓Use floating-point tensors for model inputs and regression targets, and integer class-index labels where the loss expects class IDs. Make conversions explicit.
  • ✓CPU is the baseline; CUDA and Apple MPS are optional accelerators whose availability and supported operations depend on the environment. Put model and batch tensors on compatible devices.
  • ✓Autograd records operations involving gradient-tracked tensors. backward() computes gradients and accumulates them; optimizers use those gradients to update parameters.
  • ✓nn.Module registers layers, parameters, and buffers so they can be inspected, moved, saved, and optimized. Define the data flow in forward().
  • ✓Dataset describes individual examples; DataLoader batches and samples them. Fit preprocessing on training data only and reuse the fitted transformation elsewhere.
  • ✓Before training, run one batch through the full path and verify output-target shapes, finite loss, and parameter gradients.

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...