🤖AI Learn Hub

Transformer Architecture, Visual Diagrams & Original Paper Deep Dive

Master the foundational backbone of all modern LLMs: decode Transformer structure with visualized diagrams and original paper core insights.

BeginnerLesson 1232 min read
← Back to course

Introduction: Why Transformer Is The Core Of All Modern LLMs

Released in 2017 via the paper Attention Is All You Need, the Transformer completely replaced traditional sequential neural networks like RNN and LSTM, becoming the universal underlying architecture for ChatGPT, Claude, Gemini and all mainstream large language models. Unlike prior sequence models limited by serial computation, Transformer relies fully on self-attention to achieve full parallel training and powerful long-distance context capture capability.

This lesson is split into three core modules: complete architecture decomposition with visual logic, visualized step-by-step workflow of every internal module, and systematic interpretation of the original research paper’s innovations, experimental results and future research directions.

Core takeaway: Self-attention is the core innovation of Transformer; encoder handles input comprehension while decoder handles sequential text generation.

1. Full Transformer Architecture Deep Dive

1.1 Predecessor Limitations: RNN / LSTM vs Transformer

  1. RNN/LSTM Defect 1: Serial time-step calculation, cannot parallelize training, extremely slow for long text.
  2. RNN/LSTM Defect 2: Severe long-distance dependency decay; distant words lose correlation signals.
  3. CNN Defect: Requires multi-layer stacking to connect far tokens, high computational overhead.
  4. Transformer Advantages: Fully parallelizable computation, direct global token correlation via self-attention, linear cost for long-range context capture.

1.2 Global Overall Structure Overview

The standard Transformer architecture consists of two symmetrical stacked modules: Encoder Stack and Decoder Stack, plus shared preprocessing layers and output projection layers.

Global Structure Hierarchy

  • Input Side: Tokenization → Input Embedding → Positional Encoding
  • Encoder Stack (6 identical layers): Multi-Head Self-Attention + Feed Forward + Add & Norm Residual
  • Cross Connection: Encoder output serves Key/Value for decoder cross attention
  • Decoder Stack (6 identical layers): Masked Self-Attention + Cross Attention + Feed Forward + Add & Norm
  • Output Side: Linear Projection → Softmax Token Probability Distribution

1.3 Module 1: Input Preprocessing Pipeline

Before entering encoder layers, raw text must be converted into model-processable numerical vectors via three sequential steps.

  1. 1. Tokenization: Split raw text into smallest semantic units (tokens); each token maps to a unique integer ID.
  2. 2. Input Embedding: Map each token ID to a fixed-dimension learnable vector (d_model=512 in original paper); semantically similar tokens generate similar vectors.
  3. 3. Positional Encoding: Add sinusoidal position vectors to embedding vectors to inject sequence order information, since pure attention has no inherent timing awareness.

Positional Encoding Core Logic

Sinusoidal function formula generates fixed position vectors without training: Even index: PE(pos,2i) = sin(pos / 10000^(2i/d_model)) Odd index: PE(pos,2i+1) = cos(pos / 10000^(2i/d_model)) Advantage: Can extrapolate to sequences longer than training data.

1.4 Module 2: Encoder Layer Internal Composition

Each encoder layer contains two sublayers wrapped by residual shortcut connection and layer normalization (Add & Norm). 6 identical layers are stacked in the official paper setting.

  • Sublayer A: Multi-Head Self-Attention — every token calculates correlation weights with all other tokens in the input sequence.
  • Sublayer B: Position-wise Feed Forward Network — independent two-layer nonlinear transformation for each single token vector.
  • Residual Add & Norm: Add original input vector to sublayer output, then normalize feature distribution to stabilize deep network training.

1.5 Module 3: Decoder Layer Internal Composition

Decoder adds one unique cross-attention sublayer on top of encoder’s two sublayers, designed for sequential autoregressive text generation.

  • Sublayer 1: Masked Multi-Head Self-Attention — apply upper triangular mask to block access to future ungenerated tokens, prevent information leakage.
  • Sublayer 2: Encoder-Decoder Cross Attention — decoder token vectors act as Query, encoder global context vectors act as Key & Value to align generated text with input semantics.
  • Sublayer 3: Same Feed Forward + Add & Norm residual block as encoder.

1.6 Core Module: Multi-Head Attention Mechanism

Single self-attention only captures one single dimension of semantic correlation; multi-head splits Q/K/V into multiple independent projection subspaces to simultaneously model syntax, reference, logic and other multi-dimensional relationships.

Single Self-Attention Calculation Process

1. Each token vector projects to Query(Q), Key(K), Value(V) via independent weight matrices 2. Compute similarity score: Q · Kᵀ 3. Scale scores by 1/√d_k to avoid gradient vanishing under high dimensions 4. Apply softmax to convert scores into normalized attention weights summing to 1 5. Multiply weights with Value vectors and sum to get contextualized token output

Multi-head workflow: Split Q/K/V into h parallel heads (h=8 original paper), run independent attention computation for each head, concatenate all head outputs and apply final linear projection to fuse multi-subspace features.

2. Transformer Visualized Diagram Step-by-Step Explanation

2.1 Global Architecture Visual Logic

The official Transformer structural diagram divides the model into left encoder stack and right decoder stack, with clear data flow arrows marking information transmission paths between layers.

  • Visual Rule 1: Left encoder only receives complete input sequence, no masking applied; all tokens can mutually attend each other.
  • Visual Rule 2: Right decoder input is shifted right generated sequence, masked self-attention hides all later positions in the sequence.
  • Visual Rule 3: Horizontal data line from encoder stack top to every decoder cross attention block represents global input context delivery.
  • Visual Rule 4: Every sublayer box has an Add & Norm shortcut line looping back to layer input, representing residual skip connection.

2.2 Token Embedding & Positional Encoding Visual Demo

Visual analogy: Treat each token vector as a coordinate point in high-dimensional space. Words with similar semantics cluster close together after embedding training; positional encoding adds unique offset coordinates to each position to distinguish word order.

Intuitive Visual Example

Sentence: The rain falls in Changsha Token list: [The, rain, falls, in, Changsha] Embedding vectors group weather-related tokens (rain, falls) into adjacent coordinate regions; positional encoding adds separate offset values for position 0 to 4 so the model distinguishes 'rain falls' from 'falls rain'.

2.3 Self-Attention Weight Visualization

Attention weight matrix visualization uses heatmap brightness to represent correlation strength between two tokens; brighter cells mean higher attention weight, the model focuses more heavily on that corresponding word when calculating the current token’s context vector.

  • Example sentence: The cat sat on the mat because it was tired Heatmap rule: Token 'it' shows maximum weight on 'cat', the model automatically links pronoun to its antecedent noun via attention calculation.
  • Multi-head visualization difference: Different attention heads form distinct heatmap patterns; one head tracks subject-object relations, another tracks time and location matching.

2.4 Masked Attention Visual Mask Logic

Masked self-attention visualizes the upper triangular region as fully zeroed gray blocks, which set similarity scores of future tokens to negative infinity before softmax, forcing attention weights to zero to block access to unreleased text during generation.

2.5 End-to-End Generation Visual Workflow

Stepwise Generation Visual Flow

1. User input text → tokenize + embedding + positional encoding → full encoder stack processing 2. Decoder initial input = start special token <START> 3. Masked self-attention only sees <START>, cross attention reads full encoder context, feed forward computes token probability 4. Select highest-probability token as first output word 5. Append generated token to decoder input sequence, repeat masking and attention calculation loop until end token <END> is produced

3. Original Paper Attention Is All You Need Full Interpretation

3.1 Paper Basic Information & Research Background

  1. Release Time: June 2017, arXiv:1706.03762
  2. Author Team: Google Brain & Google Research researchers
  3. Research Motivation: Eliminate serial computation bottleneck of RNN/CNN sequence models, improve translation training speed and long-text modeling performance.
  4. Core Proposition: Attention mechanism alone can construct state-of-the-art sequence transduction models without recurrent or convolutional layers.

3.2 Four Core Innovations Proposed In The Paper

  • 1. Pure Attention Architecture: Fully discard recurrence and convolution, build entire seq2seq model based only on multi-head self-attention.
  • 2. Multi-Head Attention: Split attention calculation into multiple independent subspaces to capture diverse semantic features simultaneously.
  • 3. Scaled Dot-Product Attention: Introduce dimension scaling factor to resolve large dot-product gradient vanishing problem under high embedding dimensions.
  • 4. Sinusoidal Positional Encoding: Fixed mathematical position vectors without extra trainable positional parameters.

3.3 Official Model Hyperparameter Settings From Paper

Standard Base Model Hyperparameters

  • Embedding dimension d_model = 512
  • Number of attention heads h = 8, single head dimension d_k = d_v = 64
  • Feed forward hidden dimension d_ff = 2048
  • Encoder stack layers N = 6, Decoder stack layers N = 6
  • Dropout rate = 0.1, sinusoidal positional encoding base constant = 10000

3.4 Key Experimental Results & Performance Breakthroughs

All experiments focus on machine translation benchmark datasets WMT 2014 English-German and WMT 2014 English-French.

  • English-German Translation: Base Transformer achieves BLEU score 28.4, surpassing all prior single models and ensemble models.
  • English-French Translation: Large Transformer hits BLEU 41.0, training cost less than 1/4 of previous state-of-the-art systems.
  • Training Speed Advantage: Transformer completes training in days while comparable RNN models require weeks on identical hardware.
  • Generalization Test: Strong performance on English constituency parsing task even with limited training data.

3.5 Paper Limitations & Follow-Up Research Directions

The paper clearly lists unresolved problems and expansion directions for attention-based models, which later spawned BERT, GPT series and multimodal Transformer variants.

  • 1. Extend Transformer beyond text input to image, audio, video multimodal tasks.
  • 2. Design sparse/localized attention variants to cut computation cost for ultra-long sequences.
  • 3. Optimize autoregressive generation process to reduce sequential decoding overhead.
  • 4. Explore lightweight small Transformer architectures for edge device deployment.

4. Real-World Model Classification Based On Transformer

All mainstream LLMs derive from the original Transformer architecture, split into three major branches based on encoder/decoder usage design.

  • Encoder-only (BERT): Only retain encoder stack, bidirectional full attention, optimized for classification, embedding and understanding tasks.
  • Decoder-only (GPT series): Remove encoder stack, only stacked masked decoder layers, autoregressive text generation architecture for chat and writing.
  • Encoder-Decoder full (T5, original Transformer): Complete dual stack design, suited for translation, summarization and seq2seq conversion tasks.

5. Core Summary & Developer Practical Implications

  • 1. Self-attention parallel computation and global context capture are the fundamental competitive advantages of Transformer over older neural networks.
  • 2. Multi-head attention, positional encoding and residual normalization are three indispensable core auxiliary components for stable Transformer training.
  • 3. The original Attention Is All You Need paper provides a universal baseline architecture; all modern LLMs only adjust layer count, dimension and attention variants on this foundation.
  • 4. Distinguish encoder-only, decoder-only and full encoder-decoder architectures when selecting base models for different AI application scenarios.
  • 5. Masked attention is mandatory for all autoregressive generation models to eliminate future token information leakage during inference.

Beginner Core Mental Model

Treat Transformer as a two-stage system: Encoder = full-text comprehension engine, Decoder = sequential word-by-word writing engine; attention weights act as dynamic semantic links connecting every word in the text.

Key Takeaway: Transformer, originating from the 2017 paper Attention Is All You Need, is the foundational architecture powering all modern large language models. This lesson systematically breaks down its full internal structure, visualizes every module’s data flow and attention logic, and interprets the paper’s core innovations, experimental data and follow-up research directions. Mastering Transformer theory is a mandatory prerequisite for understanding prompt engineering, LLM fine-tuning and AI agent development.