Concept
Multi-Head Attention
A way of running self-attention multiple times in parallel with different learned perspectives, so the model can pick up on several types of relationships in the same sentence at once.
Added May 18, 2026
Self-attention is powerful, but a single attention computation can only focus on one type of relationship at a time. In any given sentence, there are many simultaneous relationships worth tracking: subject-verb agreement, pronoun reference, spatial relationships, temporal ordering, semantic similarity. Multi-head attention is the solution: run several self-attention operations in parallel, each with its own learned parameters, then combine the results.
Each "head" in multi-head attention independently learns to attend to different aspects of the input. One head might learn to track syntactic relationships - identifying which noun a verb applies to. Another might specialise in tracking long-range dependencies - connecting a pronoun back to its referent many words earlier. A third might focus on semantic similarity - clustering words with related meanings. These specialisations emerge from training; they are not programmed in.
The mechanics are straightforward: the input is transformed into several sets of queries, keys, and values using different learned weight matrices. Each set runs its own self-attention computation. The outputs from all heads are then concatenated and projected back into the model's working dimension. The total computation cost is similar to running one larger attention, but the multi-head structure allows the model to capture more varied patterns simultaneously.
In practice, different heads in trained models visibly specialise. Some heads attend primarily to adjacent tokens, picking up local grammatical structure. Others attend across very long distances. Some seem to track named entities, linking every mention of a person's name back to its first occurrence in a long document. This specialisation is what gives transformer models their richly contextual representations.
Modern large language models use dozens of attention heads per layer, and many layers stacked on top of each other. By the time text has passed through all these layers, every token has been processed through thousands of independent attention computations, each contributing its particular perspective on the input.
Analogy
A panel of specialists reviewing the same document simultaneously. The legal specialist reads for liability. The financial specialist reads for risk. The communications specialist reads for tone. Each produces their own analysis, then their findings are combined into a single briefing. Multi-head attention is the same principle - multiple perspectives on the same input, combined into one richer understanding.
Real-world example
Research visualising what individual attention heads learn in trained language models has found striking specialisations. Some heads track syntactic dependency structure, essentially learning to parse sentences. Others track coreferential mentions - every time a person is referred to by name or pronoun, the head links those mentions together. These emergent structures were not designed; they arose because tracking them helped with next-word prediction.
Why it matters
Multi-head attention is why transformers are so much more powerful than models with single-attention mechanisms. The ability to simultaneously track multiple types of linguistic relationships is what gives these models their deep contextual understanding. Every frontier language model in use today relies on this mechanism.
In the news
No recent coverage - check back later.
Related concepts