A_ made by arjun

2026.08.25 · WRITING

How Transformers Build Context with Self-Attention

Starting with a single service call, follow one token through self-attention: see how it forms Query, Key, and Value vectors, computes attention weights, and gathers context with multi-head attention.

Self-attention lets each token update its representation using the context it is allowed to see.

That still sounds abstract. So instead of starting with an equation, we will follow one token through the entire process.

Before self-attention, a token has no sentence-specific context

Suppose the input is:

Service A called configuration service B because it needed the latest configuration.

The text is first split into tokens and converted into vectors. How position enters the model depends on the architecture. The original Transformer adds positional encoding to the input embeddings. Models that use RoPE encode position later, when producing the Query and Key vectors. Either way, the embedding for “it” alone cannot tell the model whether the pronoun refers to Service A or configuration service B.

The sentence itself is slightly ambiguous. If the previous sentence said, “Service A started without a local configuration,” the relationship between “it” and Service A would usually become stronger. Change the context and the representation of “it” changes with it.

The token “it” examines visible context and gathers related information
Encoder-style self-attention. Line strength illustrates the calculation; it is not a measurement from a specific model.

One way to think about self-attention is this: before a token moves to the next layer, it gathers information from the positions it is allowed to see. In an encoder, that may be the whole input. In a causal decoder, it is normally the current token and the tokens that came before it.

For “it,” the model compares relationships such as:

Those relationships become different attention weights. A stronger relationship causes more information to be retrieved from the corresponding position. After the operation, “it” is no longer represented as an isolated token; its new representation contains information gathered from the current context.

These attention weights are the coefficients one layer and one head use to combine Value vectors. They should not be treated as a causal explanation of the model’s final prediction.

Every other token in the sequence goes through the same process.

Query, Key, and Value: three roles for the same token

To find relevant positions and retrieve their information, a Transformer maps each token representation into three vectors: Query, Key, and Value.

One token representation is projected into Query, Key, and Value
All three vectors come from the same token representation, but each has a different role in the attention calculation.

A search system offers a useful analogy:

When the model updates “it,” it compares that token’s Query with the Keys of every visible position. This comparison answers only “which positions should matter?” The information that enters the new representation comes from the corresponding Values.

Key and Value can therefore come from the same token while doing different jobs: the Key is used for matching; the Value carries the content.

What happens in one self-attention calculation

The calculation can be split into four steps.

The four stages of scaled dot-product attention
Q and K determine relevance first; V contributes the information only in the final step.

Step 1: calculate relevance scores

The current token’s Query is multiplied by every Key using a dot product:

QKᵀ

A larger score means the two vectors are a closer match in the current representation space. These are raw scores, not the final weights.

Step 2: scale the scores

Each score is divided by √dₖ, where dₖ is the Key dimension.

As vector dimensionality grows, dot-product magnitudes tend to grow as well. Sending those values directly into Softmax can push the distribution toward saturation and produce very small gradients. Scaling keeps the values in a more useful range.

Step 3: produce attention weights

Softmax converts the scores into positive weights that sum to one. The result says how much attention the current token should allocate to every visible position.

Step 4: combine the Values

The model computes a weighted sum of the Values. Positions with larger weights contribute more information, while positions with smaller weights contribute less.

The complete calculation is:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V

The equation contains matrix operations, but the underlying job is simple: decide which positions are relevant, then retrieve information in proportion to that relevance.

What multi-head attention actually means

A Head is not a separate model or another “brain.” It is one set of Q, K, and V projection parameters followed by one scaled dot-product attention calculation.

Several attention heads examine the same sequence in different representation spaces
Each head uses independent parameters to examine the same sequence in a different representation space.

Multi-head attention sends the same input through several heads with different learned parameters. Each head works in a smaller representation space and can therefore look for relationships from a different angle. In the original Transformer Base model, d_model is 512, there are eight heads, and each head uses dₖ = dᵥ = 64.

Some heads may emphasize coreference, while others may respond more strongly to local phrases, semantic relationships, or positional patterns. These roles are not assigned by hand, and not every head maps to a clean, nameable linguistic function.

The outputs of all heads are concatenated and passed through a final linear projection to return to the model’s required output dimension. Multi-head attention is not valuable simply because it repeats the calculation. Its purpose is to preserve several views of the relationships in the sequence at once.

Why Transformers model long-range dependencies more easily

An RNN processes a sequence in time order. Step two depends on step one, and step three depends on step two. Information from an early position must travel through a long chain of intermediate steps to reach a much later position.

Fully connected self-attention lets any two mutually visible positions connect directly. Within one layer, information does not need to pass through intermediate tokens, so the maximum path length can be treated as O(1). In a causal decoder, later tokens can read earlier tokens, while the reverse direction is blocked by the mask.

RNN serial paths compared with direct self-attention connections
Shorter paths and parallel forward-pass computation are two reasons Transformers scale effectively during training.

The other difference is parallelism. Each RNN step depends on the previous step, so an entire sequence is difficult to compute at once. Self-attention is built around matrix multiplication and can calculate relationships among many tokens together, which maps well to GPUs.

Shorter paths and greater parallelism make Transformers better suited to long-range dependencies and large-scale training.

This parallelism applies to training and to a single forward pass. Autoregressive generation still produces one token before it can produce the next.

Self-attention also has a cost

Standard self-attention compares every token with every other visible token. For a sequence of length n, the attention matrix contains roughly n² positions. Computation and memory use therefore rise quickly as the context grows.

Sparse and linear attention methods reduce connections or change the form of the calculation. FlashAttention preserves the exact attention result and O(n²) arithmetic complexity, but improves speed and memory use by reducing GPU memory traffic and avoiding large intermediate tensors.

Without position information, self-attention cannot distinguish token order from the input representations alone. The original Transformer adds sinusoidal positional encoding to the input embeddings. RoPE rotates Query and Key vectors so that relative position affects the attention scores.

Putting the path together

The core process can be compressed into three statements:

  1. Each token produces its own Query, Key, and Value.
  2. Its Query is compared with all visible Keys to produce attention weights.
  3. Those weights combine the Values into a new, context-sensitive representation.

Multi-head attention runs this process in several representation spaces at once. Each Transformer layer then updates the result produced by the previous layer. The model does not form context in a single decision; it builds it gradually through layer-by-layer information exchange.

References