PyTorch Fundamentals — Tensors, Autograd and Modules
Build a practical mental model of PyTorch tensors, automatic differentiation, modules, and data loading before writing a complete training loop.
PyTorch gives you tensor operations, gradients, and reusable layers—not a black box that trains itself.
In the neural-network lessons, you computed a forward pass and used the chain rule to understand how parameters change. PyTorch does the bookkeeping for those derivatives. You describe operations on tensors, mark the values that should be learned, calculate a loss, and ask autograd for gradients. You still choose the model, objective, optimizer, data split, and checks that make training trustworthy.
This lesson lays the groundwork: what a tensor represents, how shape and dtype affect an operation, how data moves between CPU and accelerators, what autograd records, and how nn.Module organizes trainable components. We finish by building a Dataset and DataLoader for a small tabular problem. The next lesson assembles these pieces into a full training and validation loop.
A tensor is an n-dimensional array with explicit shape, dtype, and device.
A scalar is a zero-dimensional tensor, a vector has one axis, a matrix has two, and images or batches add more axes. The shape tells you the size of each axis; it does not tell you what those axes mean. For a batch of tabular examples, a common convention is [batch, features]. For an image batch, a common channel-first layout is [batch, channels, height, width]. Read the operation's expected layout rather than assuming every library uses the same convention.
Shape errors are meaning errors in disguise
Suppose there are 64 customers and each has 12 numeric features. The input has shape [64, 12]. A linear layer with 32 output units maps it to [64, 32]: the batch axis stays in place, while the feature axis changes. If you accidentally transpose the input to [12, 64], the model may fail loudly—or worse, produce a plausible but semantically wrong computation if dimensions happen to line up. Print shapes at boundaries while developing.
Write down what each axis means next to important tensors: batch × features, batch × classes, or batch × channels × height × width. This small habit catches many broadcasting and reshape mistakes before they become training mysteries.
Dtypes determine which values and operations are representable
Neural-network inputs and weights are usually floating-point tensors such as float32. Integer class labels for losses such as cross entropy are usually int64 (shown as torch.long). A mismatch can trigger a useful error, but implicit conversions can also hide bugs. Keep labels in the dtype expected by the objective: class indices are integer targets; regression targets are floating point.
CPU, CUDA GPUs, and Apple MPS are different execution devices; all tensors in one operation must be compatible.
A tensor's device tells PyTorch where its storage and operations live. CPU is the reliable baseline. CUDA enables supported NVIDIA GPUs, while MPS enables supported Apple silicon GPU operations through Metal. Availability depends on the installed PyTorch build, hardware, operating system, and operation support. Detect a usable device at runtime; do not assume a GPU exists because the code runs on a developer machine.
Moving a model does not automatically move separately created input tensors. Move the model and each batch to the same device before the forward pass. Moving data between CPU and accelerator has a cost, so load and transfer batches deliberately. Keep data loading, metric collection, and checkpoints mindful of CPU/GPU memory.
If a tensor on cpu is combined with one on cuda or mps, PyTorch will report a device mismatch. Inspect both tensor.device and model's parameter devices; move the missing object explicitly rather than copying all tensors repeatedly inside individual layers.
Views, reshapes, and in-place operations
Operations such as indexing and reshaping can return views that share storage with the original tensor when the layout permits; reshape may return a copy when needed. In-place operations mutate existing storage and can invalidate values autograd needs to compute gradients. Prefer ordinary out-of-place expressions while learning, and use in-place updates only when you understand their memory and gradient implications.
Autograd records operations involving gradient-tracked tensors, then applies the chain rule backward.
A trainable parameter has requires_grad=True. When operations combine it with inputs, PyTorch builds a dynamic computation graph. Calling backward() on a scalar objective computes gradients of that objective with respect to the participating leaf tensors. The gradients are accumulated in each leaf's .grad field. Accumulation is intentional—for example, gradient accumulation can simulate a larger batch—but a normal training iteration clears gradients before computing the next update.
In a model, autograd handles a much larger expression graph: matrix multiplications, nonlinearities, and loss calculation. It does not choose the update rule. An optimizer reads the gradients and updates parameters; this distinction is why backward() and optimizer.step() are separate operations in the next lesson.
Clear gradients and distinguish training from inference
Gradient tracking can be disabled for validation and prediction to reduce memory use and computation. Use torch.no_grad() or inference mode for computations where gradients are not needed. Do not wrap a training forward pass in a no-gradient context: that prevents autograd from recording the graph needed by backpropagation.
optimizer.zero_grad() once per update.An nn.Module registers layers and parameters so PyTorch can find, move, save, and optimize them.
A module's __init__ defines submodules and parameters; forward describes how inputs flow through them. Assigning a layer to an attribute registers it with the module. Registered parameters appear in model.parameters(), move with model.to(device), and are included in a state dictionary. A plain Python list of layers is not automatically registered—use nn.Sequential, nn.ModuleList, or module attributes for components that should be tracked.
The forward method returns raw model outputs. For multi-class classification, the final layer typically returns logits—unnormalized class scores—and a suitable loss function combines those logits with class-index labels. Do not apply softmax before a loss that expects logits. For a one-output regressor, the output shape is often [batch, 1]; align the target shape intentionally rather than relying on broadcasting.
Parameters versus buffers
Learnable weights are parameters. Some modules also maintain state that should be saved but not optimized, such as BatchNorm's running statistics; these are buffers. Register a persistent non-learned tensor with register_buffer rather than assigning a tensor and hoping the optimizer treats it correctly. Inspect a module's named parameters and buffers when debugging model state.
Separate how one example is represented from how batches are sampled and delivered.
A Dataset defines the mapping from an index to an example. A DataLoader groups those examples into batches and can shuffle training examples, use worker processes, and collate individual samples. This separation lets the model consume batches without knowing where the records came from. For a small in-memory dataset, a TensorDataset may be enough; custom datasets are useful for files, images, and on-demand transforms.
The example uses a numeric sales target, so the labels are floating point and shaped [batch, 1]. A classification dataset may instead return an integer class index per example. Shuffle training batches to reduce dependence on record order; keep validation and test evaluation deterministic. Fit preprocessing—imputation values, category encoders, and scalers—on training data only, then apply those fitted transformations to validation and test data.
Random seeds help compare experiments, but exact reproducibility can still depend on device kernels, workers, library versions, and nondeterministic operations. Record data splits, preprocessing, model configuration, and the environment. The goal is to make the experiment traceable, not to promise identical results across every machine.
A small model sanity check catches shape and gradient bugs before a long run.
Before full training, run one batch through the model, calculate the intended loss, backpropagate, and confirm that trainable parameters receive finite gradients. This is not a quality evaluation. It confirms the plumbing: inputs and targets have expected shapes and dtypes, the model output matches the objective, and the graph reaches the parameters.
A finite loss on one batch does not mean the model is learning well. It only rules out a class of wiring errors. The next lesson adds epochs, optimizer updates, evaluation mode, validation metrics, early stopping, checkpointing, and a complete tabular example.
shape, dtype, device, and whether gradients are enabled. Small executable checks are faster to reason about than a full training run.Turn the tensor and module foundations into a disciplined training loop.
Next, train a neural network end to end: compute batch losses, update parameters, switch correctly between training and evaluation, measure validation performance, stop when generalisation stalls, and save the best checkpoint.
Build and validate a reproducible tabular training workflow.
🎯 Key Takeaways
- ✓A tensor is an array with a shape, dtype, and device. Name the meaning of each axis and check shape compatibility at model boundaries.
- ✓Use floating-point tensors for model inputs and regression targets, and integer class-index labels where the loss expects class IDs. Make conversions explicit.
- ✓CPU is the baseline; CUDA and Apple MPS are optional accelerators whose availability and supported operations depend on the environment. Put model and batch tensors on compatible devices.
- ✓Autograd records operations involving gradient-tracked tensors. backward() computes gradients and accumulates them; optimizers use those gradients to update parameters.
- ✓nn.Module registers layers, parameters, and buffers so they can be inspected, moved, saved, and optimized. Define the data flow in forward().
- ✓Dataset describes individual examples; DataLoader batches and samples them. Fit preprocessing on training data only and reuse the fitted transformation elsewhere.
- ✓Before training, run one batch through the full path and verify output-target shapes, finite loss, and parameter gradients.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
