Generalization is measured on data the model did not fit
Increasing polynomial degree gives the model more capacity. Too little capacity produces bias; too much capacity allows the model to follow sampling noise and increases variance.
L2 regularization penalizes large coefficients. The objective is not the smallest training error, but a stable model with low error on unseen observations.
Run, inspect, and compare
Follow the three guided moves above. Change one variable at a time so every visual change has a clear cause.
By the end of this lesson, you will be able to:
- Distinguish underfitting, appropriate model capacity, and overfitting from training and test behaviour.
- Explain the bias-variance trade-off and how polynomial degree changes model flexibility.
- Use L2 regularization to control model complexity and reduce the generalization gap.
In the early stages of training a neural network, everything may appear to be going well: the training loss decreases steadily, and the training accuracy improves epoch by epoch. However, when new, unseen data is introduced, the model's performance can suddenly collapse. This phenomenon is not a bug - it is a fundamental challenge in deep learning, known as overfitting. Overfitting occurs when the model fits the training data too closely, capturing not only the underlying patterns but also noise, anomalies, and incidental correlations. As a result, the model performs well on the data it has seen, but fails to generalize to new examples. In contrast, underfitting describes a scenario where the model fails to capture even the patterns in the training data, often due to insufficient capacity or inadequate training. Both overfitting and underfitting degrade the model's generalization ability - the fundamental goal.