Transformers: Build with Attention and Position
Attention gives us a way to connect positions directly.
Transformers: Build with Attention and Position explained visually
Attention gives us a way to connect positions directly. English captions are optional and start off. The complete transcript follows the written lesson.
What you will learn
Attention gives us a way to connect positions directly.
Combine reusable blocks
Attention gives us a way to connect positions directly. A transformer combines attention with feed-forward transformations, residual connections, and normalization. These blocks are repeated to build increasingly useful representations. The architecture can process relationships across a sequence without using the same step-by-step recurrence as a basic recurrent network.
Represent order explicitly
Because ordinary attention alone does not encode sequence order in the required way, transformers include positional information. Different architectures use different positional schemes. Multiple attention heads provide several learned comparison spaces, but we should not assume each head has one simple human-readable role. Encoder-style, decoder-style, and encoder-decoder designs support different tasks and masking rules.
Match the architecture to the task
Standard full self-attention compares pairs of positions, so its attention matrix grows quadratically with sequence length. Longer contexts therefore have important computation and memory costs, although alternative designs can change those tradeoffs. For image tasks, patches can become sequence elements, but patch size and resolution influence the available detail. Evaluate the architecture with realistic input lengths and resource limits rather than assuming a transformer is always the best choice.
Worked example
If a sequence grows from 100 positions to 200, a full attention score matrix grows from 10,000 entries to 40,000 entries. That is four times as many pairwise scores, even though the number of input positions only doubled.
Quick check: test your understanding
Try it yourself: What happens to the number of full pairwise attention scores when sequence length triples?
Check the reasoning
It grows by a factor of nine, because the matrix has one dimension for each sequence position on both axes.
Do all transformer architectures permit every position to attend to future positions?
No. Causal decoders use masks to prevent access to future positions.
Further reading
Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola.
Deep Learning — Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Original ToolsInVision teaching examples. Editorial approach · Suggest a correction
Accessible referenceLesson 77 transcript233 words · Video transcript · Matches the published narration and captionsOpen transcript
We now have attention. How do we turn it into a complete network? A Transformer builds on attention by combining it with several other components.
A typical block includes attention, a feed-forward transformation, residual connections, and normalization. Stacking these blocks lets the model repeatedly refine its representations.
Self-attention allows positions within the input sequence to interact. Multiple attention heads learn different comparison spaces. However, we should not assume that each head has one simple role we can describe in a sentence.
There is another essential ingredient: position. The same words in a different order can mean something different. Transformers therefore include positional information so that the model can represent those relationships.
Different Transformer designs use different attention rules. A causal decoder must hide future positions when predicting the next token. An encoder can often use context from both directions. The right rule depends on the task.
There is also a cost to comparing positions. With standard full attention, one hundred positions produce ten thousand pairwise scores per attention matrix. Two hundred positions produce forty thousand. Doubling the sequence length quadruples the number of scores.
Images can also become sequences by dividing them into patches. Smaller patches preserve finer spatial detail, but usually create more positions to process.
So a Transformer is more than attention alone. Its positional information, masking, surrounding layers, and resource requirements all affect how well it fits a real problem.