The transformer is the architecture behind nearly every modern LLM (GPT, Claude, Llama). Its breakthrough was processing a whole sequence in parallel using attention, instead of word-by-word like older RNNs — which is what made training on internet-scale data possible.
The pipeline
text → tokens → embeddings → [ attention + feed-forward ] ×N layers → next-token probabilities- Tokenize the text into pieces.
- Turn each token into an embedding (a vector), plus a position signal.
- Pass it through a stack of identical layers, each with:
- Self-attention — every token looks at every other token to gather context.
- Feed-forward network — processes each position's enriched representation.
- The final layer outputs a probability for every possible next token; sampling picks one (Temperature and Sampling Explained).
Why it won
| Transformer | Older RNN/LSTM |
|---|---|
| Whole sequence in parallel | One step at a time |
| Long-range context via attention | Forgets distant tokens |
| Scales to billions of params | Hard to scale |
An LLM is just this stack, trained to predict the next token, scaled up massively. Everything else — chat, reasoning, code — emerges from that one objective.
Related: The Attention Mechanism · What Are Embeddings · Tokenization Explained