Training Neural Networks in PyTorch — The Training Loop
Train a tabular neural network with correct gradient updates, train/eval modes, validation, early stopping, checkpointing, and an introduction to mixed precision.
Training is a repeated experiment: predict, measure error, compute gradients, update, then test on data the model did not train on.
A neural network's forward method only produces outputs. A complete training workflow must define how examples are split, how batches are loaded, which loss matches the task, which optimizer updates parameters, and how performance is checked on held-out data. A loop that lowers training loss is not automatically a useful model: it may be overfitting, leaking information from validation, or optimizing the wrong objective.
We will use a fictional Indian retail demand dataset: each row contains store and product attributes plus calendar features, and the target is next-period units sold. Assume the data has already been split chronologically or by the intended deployment boundary, categorical data is encoded using training-fitted transformations, and numeric values are scaled without looking at validation outcomes. The example focuses on the mechanics that apply to other tabular regression tasks too.
Match the final output, target shape, and loss to the prediction problem.
For one continuous demand target, a regression network can emit one value per example. Mean squared error penalizes large residuals more strongly; mean absolute error is less sensitive to extreme residuals and is often useful as a reporting metric. It is common to optimize one loss and report several business-relevant metrics. The choice should reflect the cost of errors, not habit.
The final layer is linear: negative predictions are possible. Whether to constrain predictions depends on how the target is defined and how the model will be used. Do not add a clamp simply because demand cannot be negative without considering what it does to gradients and evaluation; model the target and constraints deliberately. For classification, change the output dimension and loss/target convention rather than reusing regression code unchanged.
Loss is not a business metric by itself
MSE is measured in squared target units, so its scale is not directly interpretable as units sold. Track metrics in the original target scale when that is meaningful. If targets or inputs were transformed, invert the target transformation before calculating human-readable metrics, and fit any transformation using training data only.
The core sequence is clear because each operation has one job.
zero_grad clears accumulated gradients; the model computes predictions; the loss compares predictions with targets; backward fills parameter gradients; and step changes parameter values. Each batch produces an update. One pass over all training batches is an epoch, not necessarily a meaningful unit of progress if the data pipeline repeats examples or drops records.
Aggregate batch metrics with the correct weighting
Averaging batch means equally can be slightly wrong when the final batch is smaller. Weight each batch loss by its example count, or accumulate a sum-reduced loss. For reporting MAE, accumulate the sum of absolute errors and divide by the number of examples. Convert values to Python scalars for periodic logging rather than synchronizing the accelerator on every operation.
torch.isfinite; do not hide invalid values by replacing them blindly.Evaluation changes both the module behavior and the gradient context.
Some layers behave differently during learning and prediction. Dropout randomly masks activations in training mode but not evaluation mode. Batch normalization uses batch statistics during training and stored running statistics during evaluation. Call model.train() before training batches and model.eval() before validation or inference.
Evaluation does not update weights. Pair model.eval() with torch.no_grad() (or inference mode) so PyTorch avoids building a gradient graph. Then calculate validation metrics over the complete validation set. Validation guides model and hyperparameter choices; reserve a separate test set for a final, limited-use estimate after those choices are settled.
When training resumes after validation, explicitly call model.train(). A metric averaged over batches should be weighted by the number of examples, as above. Make sure a validation dataset does not use a training-only random augmentation and that its preprocessing is identical to inference preprocessing.
Choose splits that mirror the way predictions will be made
For independent rows, a random split may be suitable if it respects groups and duplicate records. For repeat observations by customer, store, or product, random row splitting can put near-identical entities on both sides. For future demand forecasts, use a chronological split so validation occurs after training time. Split design is part of the evaluation claim: it defines what the reported metric can reasonably predict.
Use validation to decide when to stop, and save the best model—not simply the final epoch.
Training loss usually keeps improving as a model fits the training examples. Validation loss may improve, then flatten or worsen as generalisation declines. Early stopping tracks the best validation score and stops after a chosen patience window with no improvement. The patience and minimum improvement threshold are hyperparameters, not universal constants.
Cloning the state dictionary matters: a state dictionary references tensors that continue changing during training. For an in-memory best-weight snapshot, clone them. For a resumable checkpoint, save model state, optimizer state, epoch, best validation score, and any scheduler or gradient-scaler state. Store preprocessing metadata and the feature order alongside the model in a controlled, versioned artifact.
Extract the update and measurement logic so experiments stay readable.
The following function keeps the training-only operations together. In a real project, annotate inputs and return types, add structured logging, and define how non-finite batches are handled. Avoid hiding data-dependent behavior inside an opaque helper: training code should make it easy to see when gradients are enabled and when parameter updates occur.
A learning-rate scheduler may adjust optimizer learning rates over epochs or batches. Follow the scheduler's documented stepping convention: some schedulers step after each epoch, others after an optimizer update, and plateau-based schedulers take a validation metric. Log the actual learning rate so a schedule that is not advancing does not go unnoticed.
Use mixed precision as a measured optimization, not as a substitute for a correct baseline.
Mixed precision performs selected operations using lower-precision floating-point formats while retaining precision where needed. On supported hardware it can reduce memory use and improve throughput. The appropriate dtype and APIs depend on device, operation, and PyTorch release. CUDA training commonly combines autocasting with gradient scaling for float16; bfloat16 often has a wider exponent range and may not need scaling, depending on the workflow. Benchmark before keeping the added complexity.
The snippet is illustrative and assumes a current PyTorch API that accepts a device type for autocast and a CUDA GradScaler. Check the installed version's official documentation before adapting it. Keep a full-precision baseline, compare task metrics, and test numerical stability. Do not enable a CUDA-specific scaler for CPU or MPS simply because it appears in an example.
Other performance checks
Profile the input pipeline and model before optimizing. Small batches, repeated CPU-to-device copies, synchronous logging, and expensive preprocessing can dominate training. A faster kernel does not help if the accelerator waits for data. Reproducible benchmarks use the same data, batch size, warm-up, and measurement procedure.
When training looks wrong, investigate the data path and evaluation assumptions first.
A falling training loss with flat validation performance may indicate overfitting, a mismatched split, distribution shift, or an uninformative feature set. A constant loss can mean the learning rate is too small, gradients are missing, targets are misaligned, or parameters are not in the optimizer. Compare a tiny-batch overfit test: a sufficiently expressive model should often fit a very small clean sample. If it cannot, verify the labels, loss, output shape, and gradient flow.
For retail demand, inspect seasonality, stockouts, promotions, and zero-heavy targets before interpreting a model metric. Randomly splitting future observations into training rows can leak information into the evaluation. A low average error may still hide poor performance on important stores or products; report slices that reflect operational consequences.
With the training mechanics in place, explore architectures that use image structure.
A fully connected network sees an image as a flat feature vector. Convolutional networks preserve local spatial patterns and share filters across image locations. The CNN lesson applies PyTorch training to a visual classification problem and introduces the architectural ideas behind image models.
Learn how convolutions turn pixels into useful spatial features.
🎯 Key Takeaways
- ✓A training update clears gradients, runs the forward pass, computes a task-appropriate loss, calls backward(), and steps the optimizer.
- ✓Use model.train() for optimization and model.eval() plus a no-gradient context for validation and inference.
- ✓Weight batch-level metrics by example count, especially when the final batch is smaller.
- ✓Validation informs model choices; the test set is for a final limited-use estimate. Choose splits that match deployment, including chronological splits for forecasting.
- ✓Early stopping monitors validation performance and restores the best checkpoint, not automatically the last epoch.
- ✓A resumable checkpoint includes model and optimizer state plus relevant scheduler/scaler state, epoch, preprocessing, and provenance.
- ✓Mixed precision can improve throughput on supported hardware. Benchmark it against a stable full-precision baseline and verify the installed API.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
