Deep Learning · Practical Issues
An interactive guide to the five core problems in training deep networks — and what researchers and engineers do about them.
Problem 1
When a model lacks the capacity to capture the data's underlying pattern, it fails even on the training examples it was shown. Error stays stubbornly high on both training and test sets — the hallmark of high bias.
MODEL COMPLEXITY · DRAG SLIDER
Problem 2
The model learns the training set too well — including its noise and idiosyncrasies. Training error keeps falling but validation error starts rising. The widening gap between the two is the diagnostic signature of high variance.
Overfitting has four standard defenses: regularization, parameter sharing, early stopping, and rethinking architecture depth. We cover each below.
EPOCH SLIDER — WATCH THE GAP OPEN
Remedy 1
Add a penalty to the loss function that discourages large weights — forcing the model to find simpler solutions that generalize better.
L2 (weight decay) pushes all weights toward zero smoothly. L1 drives many weights to exactly zero, acting as feature selection. Dropout randomly disables neurons each step, forcing redundancy.
WEIGHT DISTRIBUTION — SELECT METHOD
Remedy 2
Force different parts of the network to use the same weights. This reduces the number of free parameters dramatically — and the sharing itself encodes what we already know about the data's structure.
CNNs slide the same filter across an entire image — the same weights detect an edge anywhere, encoding translation invariance. RNNs reuse weights across time steps, encoding temporal stationarity.
CONV FILTER SLIDING — WATCH IT SCAN
Remedy 3
Monitor a held-out validation set throughout training. Stop when validation error stops improving — and restore the checkpoint from the best epoch, not the last one.
This acts as implicit L2 regularization: it limits how far weights travel from their small initial values. Nearly free to implement, consistently effective.
CLICK ANY EPOCH TO SEE THE STATE
← Click an epoch on the bar above
Remedy 4
The universal approximation theorem says one wide layer can represent any function — but it may need exponentially many neurons. A deeper network builds hierarchical abstractions with far fewer total parameters.
Fewer parameters means less capacity to overfit. Depth is a form of inductive bias — you're encoding the belief that the problem has layered structure.
⚠ The cost: depth makes training harder. Gradients must travel further — which leads directly to our next problem.
COMPARE NETWORK ARCHITECTURES
1 hidden layer · 64 neurons · 64 params/layer
Problem 3
Backpropagation multiplies many Jacobians layer by layer. With sigmoid activations, the maximum derivative is 0.25 — so 10 layers give a gradient of at most 0.25¹⁰ ≈ 10⁻⁶. Early layers receive essentially no update and stop learning.
The opposite also occurs: if multiplied values exceed 1, gradients explode, updates destabilize training, and the loss becomes NaN.
GRADIENT MAGNITUDE PER LAYER — SELECT ACTIVATION
Problem 4
Even with healthy gradients, SGD can crawl, oscillate, or stall. The learning rate is the single most important hyperparameter: too high and updates overshoot; too low and training barely moves.
Ill-conditioning creates ravines — steep walls in some directions, nearly flat in others. SGD zig-zags between the walls instead of heading toward the minimum. Momentum and adaptive methods like Adam smooth this out.
LOSS LANDSCAPE — CLICK TO DROP OPTIMIZER
Problem 5
Neural network loss surfaces are highly non-convex. Gradient descent can settle into points that are not globally optimal.
Local minima — every direction goes up locally, but a better solution exists elsewhere. Saddle points — flat in some directions, curved in others; far more common than local minima in high dimensions, and they stall plain gradient descent. Spurious optima — low training loss but poor generalization.
Modern insight: in large networks, most local minima are nearly as good as the global minimum. Saddle points are the real obstacle, and SGD's inherent noise helps escape them.
1D LOSS SURFACE — CLICK TO RELEASE BALL
Summary
Look at your loss curves. Identify the pattern. Reach for the matching tool.
| Problem | Telltale sign | Main remedies |
|---|---|---|
| Underfitting | High training error | More capacity · longer training · less regularization |
| Overfitting | Train–val gap widens | Regularization · parameter sharing · early stopping · go deeper |
| Unstable gradients | Early layers frozen or loss → NaN | ReLU · He/Xavier init · batch norm · residual connections · clipping |
| Slow convergence | Loss oscillates or crawls | Tuned learning rate · momentum · Adam optimizer |
| Local / spurious optima | Training stalls at mediocre loss | SGD noise · momentum · good init · random restarts |
Further Reading
Overfitting · Dropout
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
JMLR 15, 1929–1958
Vanishing Gradients · Init
Understanding the Difficulty of Training Deep Feedforward Neural Networks
AISTATS 2010
Batch Normalization
Batch Normalization: Accelerating Deep Network Training
ICML 2015
Residual Connections
Deep Residual Learning for Image Recognition
CVPR 2016
Saddle Points
Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization
NeurIPS 2014
Adam Optimizer
Adam: A Method for Stochastic Optimization
ICLR 2015
Weight Initialization
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet
ICCV 2015
Gradient Clipping
On the Difficulty of Training Recurrent Neural Networks
ICML 2013