Modern Deep Learning Foundations

Labs: 6

Lab 2 of 6 Overfitting and Regularization Foundation / 45 min / Deliverable: Model selection brief

Screen 1 of 3 - Read before you experiment

Generalization is measured on data the model did not fit

Increasing polynomial degree gives the model more capacity. Too little capacity produces bias; too much capacity allows the model to follow sampling noise and increases variance.

L2 regularization penalizes large coefficients. The objective is not the smallest training error, but a stable model with low error on unseen observations.

Evidence levelExact polynomial fitting on a reproducible synthetic experiment

What is real, computed, or illustrative

Evidence and provenance

Derived Experiment 24 independently resampled training sets

Bias squared and variance are estimated from repeated fits on a fixed evaluation grid.

Measured Model fit Polynomial ridge regression solved in the browser

The current blue curve and train/held-out MSE use the selected degree and L2 value.

Derived Decomposition Expected MSE = Bias squared + Variance + noise

Noise variance is fixed by the known synthetic data-generating process.

Scope Claim boundary The green truth is available only for teaching

In real projects the noise-free function is unknown and model selection relies on validation evidence.

Screen 2 of 3 - Experiment

Run, inspect, and compare

Follow the three guided moves above. Change one variable at a time so every visual change has a clear cause.

Guided mode Predict first, advance one stage at a time, and explain the displayed values before changing another control.

Reproducible experiment record

Compare runs instead of trusting one result

Run the laboratory, then capture the controls, metrics, evidence status, seed, and timestamp.

The accessible visual summary will update when the laboratory renders its first result.

Evidence checkpoint

Explain what happened, then transfer it

0 / 3responses complete

Explain the evidence

Use the displayed Bias squared, Variance, noise, train MSE, and held-out MSE to explain the observed fit.

Use at least one value or visible change from the experiment.

Transfer the idea

Describe how you would choose model capacity and regularization when the true data-generating function is unknown.

Name the new context and the design choice you would make.

Defines a validation or resampling procedure
Separates bias, variance, and noise
Chooses capacity and L2 using unseen data
Explains how stability will be monitored

Continue to the bias-variance explanation

Screen 3 of 3 - Consolidate

By the end of this lesson, you will be able to:

Distinguish underfitting, appropriate model capacity, and overfitting from training and test behaviour.
Explain the bias-variance trade-off and how polynomial degree changes model flexibility.
Use L2 regularization to control model complexity and reduce the generalization gap.

In the early stages of training a neural network, everything may appear to be going well: the training loss decreases steadily, and the training accuracy improves epoch by epoch. However, when new, unseen data is introduced, the model's performance can suddenly collapse. This phenomenon is not a bug - it is a fundamental challenge in deep learning, known as overfitting. Overfitting occurs when the model fits the training data too closely, capturing not only the underlying patterns but also noise, anomalies, and incidental correlations. As a result, the model performs well on the data it has seen, but fails to generalize to new examples. In contrast, underfitting describes a scenario where the model fails to capture even the patterns in the training data, often due to insufficient capacity or inadequate training. Both overfitting and underfitting degrade the model's generalization ability - the fundamental goal.

Mastery artifact

Leave with inspectable evidence, not a completion click

Your artifact combines the prerequisite check, prediction, experiment configuration, displayed evidence, explanation, transfer rubric, and confidence change.

Complete the evidence cycle to unlock the artifact.

Import progress JSON

Retention plan

Retrieve the concept after time has passed

Complete the artifact to schedule a 24-hour retrieval prompt and a seven-day transfer revisit.

Learning-study instrument

Help evaluate whether the laboratory teaches the concept

Clarity 3 / 5 Mental effort 3 / 5 Transfer confidence 3 / 5

No name, email address, free text, or IP address is retained in the learning record.

INTERACTIVE DEEP-LEARNING GUIDE

Generalization and Regularization

Use the experiment to see how model capacity and regularization change training fit, held-out error, bias, and variance together.

What to change

Move polynomial degree and L2 regularization separately. Resample the data to check whether the same conclusion survives a different training set.

What to compare

Compare training MSE with held-out MSE. A widening gap is evidence of overfitting; high values for both can indicate underfitting.

What to conclude

Prefer the simplest configuration that performs consistently on unseen data, not the configuration with the lowest training error.

GENERALIZATION FAQ

What is generalization?

Generalization is a model’s ability to perform well on new observations that were not used to fit it.

How can I recognize overfitting?

Training error becomes very small while held-out error stays high or rises, and results change substantially after resampling.

How can I recognize underfitting?

Both training and held-out errors remain high because the model is too restricted to capture the useful pattern.

What does L2 regularization do?

It penalizes large coefficients, reducing model sensitivity and often improving stability on unseen data.

Is the lowest held-out error always best?

Not by itself. Confirm that the result is stable across multiple samples or validation folds and is operationally meaningful.

Why resample the observations?

Resampling reveals whether a result reflects a durable pattern or one fortunate split of the data.

What is the bias–variance trade-off?

More flexibility can reduce bias but increase sensitivity to the sample; regularization can reduce variance at the cost of some bias.

How should the final model be selected?

Use repeated validation, uncertainty, simplicity, and deployment constraints—not training performance alone.

Explore all interactive deep-learning laboratories View the Modern Deep Learning Foundations course