Modern Deep Learning Foundations
- Labs
- 6
Attention constructs a new representation for every token
Each token is projected into Query, Key, and Value vectors. Query-Key dot products create compatibility scores, scaling controls their magnitude, and softmax converts them into attention weights.
A causal mask removes future positions before softmax. The resulting weights combine Value vectors into a context vector that represents the selected token in its current sequence.
Lesson 11 - Self-Attention and the Transformer Principle
Printed pages 72-76; PDF pages 74-78.
- Query, Key, and Value Projections printed pp. 72-73 / PDF pp. 74-75 projection paragraphs; Equations 11.1-11.2
- Scaled Scores, Masking, and Softmax printed pp. 73-74 / PDF pp. 75-76 attention normalization; Equations 11.3-11.5
- Context Vector and Transformer Block printed pp. 74-76 / PDF pp. 76-78 weighted sum and block composition; Equation 11.6
Evidence and provenance
Token IDs, embeddings, Q/K/V projections, scores, softmax, and context vectors are computed in the browser.
Masked future positions are removed before softmax and therefore receive zero probability.
The compact dimensions make every number inspectable; they are not claims about a production language model.
A high weight shows contribution in this forward pass, not human-like reasoning or causal importance.
Run, inspect, and compare
Follow the three guided moves above. Change one variable at a time so every visual change has a clear cause.
Compare runs instead of trusting one result
Explain what happened, then transfer it
Follow the selected token through Q/K/V, scaled scores, masking, softmax, and the context vector using at least one displayed number.
Use at least one value or visible change from the experiment.Explain how the computation would differ for bidirectional attention or for a longer sequence with several heads.
Name the new context and the design choice you would make.- Distinguishes causal from bidirectional attention
- Explains Q-K scoring and scaling
- Explains softmax and the weighted Value sum
- Addresses sequence length or multiple heads
By the end of this lesson, you will be able to:
- Follow tokens through embedding and Query, Key, and Value projections.
- Explain scaled dot-product attention and why a causal mask blocks future tokens.
- Read attention weights and construct the weighted context vector for a selected token.
Explains shift from RNN to self-attention. Introduces Query, Key, Value, and weighted sums. Outlines Transformer encoder-decoder and multi-head attention.
Leave with inspectable evidence, not a completion click
Your artifact combines the prerequisite check, prediction, experiment configuration, displayed evidence, explanation, transfer rubric, and confidence change.
Retrieve the concept after time has passed
Complete the artifact to schedule a 24-hour retrieval prompt and a seven-day transfer revisit.
Help evaluate whether the laboratory teaches the concept
No name, email address, free text, or IP address is retained in the learning record.
Transformer Forward Pass
Use the experiment to follow one token through tokenization, embeddings, Q/K/V projections, scaled scores, softmax, context construction, and output probabilities.
What to change
Edit the input, select a query token, and toggle masking while keeping the inspection point explicit.
What to inspect
Trace the query, key, and value vectors, attention scores, normalized weights, context vector, and output distribution.
What to conclude
Attention weights are computational coefficients for this forward pass; they are not automatically causal explanations.
TRANSFORMER FAQ
What is self-attention?
Self-attention lets each token combine information from other tokens using learned query, key, and value projections.
Why are attention scores scaled?
Dividing by the square root of the key dimension keeps dot products from becoming too large before softmax.
What does softmax do here?
Softmax converts permitted attention scores into non-negative weights that sum to one for each query.
What is causal masking?
Causal masking prevents a token from attending to later positions, which preserves left-to-right generation.
Do high attention weights prove importance?
No. They show contribution within this computation, not necessarily causal or human-interpretable importance.
Why are positional signals needed?
Attention alone does not encode sequence order, so positional information is added or represented separately.
What is multi-head attention?
Multiple heads learn different projection spaces and can capture different relationships in parallel.
How should a transformer be evaluated?
Evaluate task quality, calibration, robustness, latency, memory use, and failure behavior—not attention maps alone.