Modern Deep Learning Foundations

Labs: 6

Lab 1 of 6 Understanding Evaluation with AUC-ROC Curve Foundation / 35 min / Deliverable: Threshold policy brief

Screen 1 of 3 - Read before you experiment

A score becomes a decision only after you choose a threshold

ROC-AUC evaluates how well a classifier ranks positive cases above negative cases across every possible threshold. It does not select the operating threshold for you.

The selected threshold creates the confusion matrix. Moving it changes false positives and false negatives, so the right operating point depends on the cost of each error in the real decision context.

Exact book map

Lesson 5 - Performance Evaluation Metrics

Printed pages 33-40; PDF pages 35-42.

Open the exact pages

Confusion Matrix printed pp. 34-36 / PDF pp. 36-38 paragraphs 1-6; Equations 5.2-5.8
ROC Curve and AUC printed pp. 37-38 / PDF pp. 39-40 paragraphs 1-5; Equation 5.9
Balanced Accuracy and MCC printed pp. 39-40 / PDF pp. 41-42 closing metric-comparison paragraphs

Evidence levelReal holdout predictions + controlled teaching example

What is real, computed, or illustrative

Evidence and provenance

Measured Dataset UCI Wisconsin Diagnostic Breast Cancer holdout

113 held-out cases; the operating threshold never changes the underlying scores.

Derived Computation Exact confusion matrix, ROC, and trapezoidal AUC

Every displayed metric is recomputed in the browser from the selected profile.

Illustrative Teaching profile A deliberately readable AUC 0.80 example

Use it to learn threshold trade-offs, then verify the same ideas on the real holdout.

Scope Claim boundary Ranking quality is not a deployment policy

AUC compares rankings; an operating threshold must reflect the cost of false positives and false negatives.

60-second prerequisite check

Choose the support level from evidence

A classifier threshold increases from 0.40 to 0.75. Which statement is generally correct? Recall usually falls while specificity usually rises. ROC-AUC must increase because the threshold is higher. The underlying prediction scores are retrained.

A threshold changes the operating point, not the ranking of scores. Review the confusion-matrix and ROC sections before experimenting.

Guided task

Complete these three moves

1
Predict
Before moving the threshold, decide which error matters more in the selected scenario.
2
Experiment
Move the threshold and trace the resulting change in the confusion matrix, recall, specificity, and F1.
3
Compare
Switch between the teaching example and the real UCI holdout. Explain why AUC stays fixed while threshold metrics change.

Record your prediction before you run the lab

Before running the lab, which error is more costly in your chosen scenario: a false positive or a false negative? State why.

Confidence before experimenting 50% A useful prediction names the expected change and why it should happen.

Start the threshold experiment

Interactive model evaluation laboratory

AUC ROC Curve Simulator

Move the decision threshold and see sensitivity, specificity, TPR, FPR, the confusion matrix, the ROC curve, and the AUC score update in real time.

Reproducible experiment record

Compare runs instead of trusting one result

Run the laboratory, then capture the controls, metrics, evidence status, seed, and timestamp.

The accessible visual summary will update when the laboratory renders its first result.

Evidence checkpoint

Explain what happened, then transfer it

0 / 3responses complete

Explain the evidence

After moving the threshold, explain what changed in TP, FP, TN, FN, recall, and specificity, and why AUC did not change.

Use at least one value or visible change from the experiment.

Transfer the idea

Choose a real screening, fraud, or quality-control decision and justify a threshold policy for it.

Name the new context and the design choice you would make.

Names the stakeholder and decision context
Specifies the relative cost of FP and FN
Chooses and justifies an operating threshold
States a limitation or monitoring condition

Continue to the metric interpretation

Screen 3 of 3 - Consolidate

By the end of this lesson, you will be able to:

Calculate and interpret accuracy, precision, recall, specificity, and F1 from a confusion matrix.
Explain how changing the decision threshold alters classification outcomes and operating trade-offs.
Interpret a ROC curve and distinguish threshold-independent AUC from one selected operating point.

In the previous lessons, we built a neural network, trained it on real data, updated weights using gradients, and selected an optimizer. Now it is time to ask a critical question: how do we know if our model is actually good? The goal of machine learning is not to memorize training data, but to generalize - that is, to perform well on new, unseen data. This ability is known as the model's generalization ability, and it is the central measure of any predictive system.

Mastery artifact

Leave with inspectable evidence, not a completion click

Your artifact combines the prerequisite check, prediction, experiment configuration, displayed evidence, explanation, transfer rubric, and confidence change.

Complete the evidence cycle to unlock the artifact.

Import progress JSON

Retention plan

Retrieve the concept after time has passed

Complete the artifact to schedule a 24-hour retrieval prompt and a seven-day transfer revisit.

Learning-study instrument

Help evaluate whether the laboratory teaches the concept

Clarity 3 / 5 Mental effort 3 / 5 Transfer confidence 3 / 5

No name, email address, free text, or IP address is retained in the learning record.

AUC ROC explained through interaction

Learn the AUC ROC Curve by Moving the Threshold

This AUC-ROC Simulator turns model-evaluation metrics into a live experiment. Move the decision threshold and watch sensitivity, specificity, false-positive rate, the confusion matrix, the selected ROC point, and the AUC score respond together.

What the ROC curve shows

The receiver operating characteristic curve plots true-positive rate against false-positive rate across classification thresholds. Each point represents a different operating decision.

What the AUC score measures

Area under the ROC curve summarizes how well prediction scores rank positive cases above negative cases. It evaluates the score ordering across thresholds, not one confusion matrix.

Why the threshold matters

Changing the threshold changes false positives, false negatives, recall, and specificity. Use the simulator to choose an operating point that matches the real cost of each error.

AUC ROC FAQ

What is AUC ROC?

ROC is the curve of true-positive rate versus false-positive rate across thresholds. AUC is the area under that curve and summarizes ranking discrimination in a single value.

Does changing the decision threshold change AUC?

Not when the underlying prediction scores stay the same. The threshold changes the operating point and confusion matrix; AUC summarizes the ranking across all thresholds.

What is a good AUC score?

An AUC of 0.5 reflects random ordering and values closer to 1 indicate stronger discrimination. A useful score still depends on the dataset, error costs, calibration, and deployment context.

When should ROC AUC be used?

Use ROC AUC to compare binary classifiers across thresholds. For heavily imbalanced problems, also inspect precision-recall behaviour and select a threshold using the real costs of false positives and false negatives.

Is there a minimum sample size for AUC ROC?

There is no universal minimum. AUC can be calculated when the test set contains at least one positive and one negative case, but estimates from small samples are unstable. Use an independent test set with enough cases in each class and report a confidence interval.

What does an AUC below 0.5 mean?

It means the scores tend to rank negative cases above positive cases. Check whether labels or score direction were reversed, then investigate data leakage, distribution shift, or a pipeline error. Reversing the score direction changes AUC to approximately 1 minus the original value.

How is ROC AUC used with more than two classes?

Compute class-level AUC with one-vs-rest or pairwise AUC with one-vs-one, then aggregate it. Macro averaging weights classes equally, weighted averaging uses class support, and micro averaging pools decisions. Always report the strategy and averaging method.

Does ROC AUC measure probability calibration?

No. A model can rank cases well and still produce poorly calibrated probabilities. Evaluate calibration separately with a reliability diagram, calibration error, log loss, or Brier score.

How should uncertainty in AUC be reported?

Report a confidence interval from a suitable method such as bootstrap resampling or DeLong's method on an independent test set. When comparing models on the same cases, use a paired comparison rather than treating the AUC estimates as independent.

Is ROC AUC enough for imbalanced classes?

Not always. ROC AUC is relatively insensitive to class prevalence and can look strong while precision is poor for a rare positive class. Also inspect the precision-recall curve, PR AUC, class-specific errors, and the operating threshold required in deployment.