Attention is the core trick of the transformer: for each token, the model decides which other tokens matter and blends their information accordingly. It's how "it" in a sentence figures out what it refers to.
Query, Key, Value
Every token produces three vectors:
- Query — what am I looking for?
- Key — what do I offer?
- Value — the information I carry.
Each token's query is compared against every token's key; high matches get high weight, and the output is a weighted blend of the values. So each word's representation becomes a context-aware mix of the whole sentence.
attention(Q,K,V) = softmax(Q·Kᵀ / √d) · VMulti-head attention
The model runs several attention "heads" in parallel, each learning a different kind of relationship — one tracks syntax, another long-range references, another tone. Their outputs are combined.
The catch: every token attends to every other token, so cost grows with the square of the sequence length — the reason long context windows are expensive.
Related: How Transformers Work · Context Windows Explained · What Are Embeddings