Transformer Attention Mechanisms

A token's representation should depend on surrounding tokens because its meaning or grammatical role can change drastically depending on its context within a sequence.

Sharing projection parameters (like query, key, and value matrices) across all token positions allows the exact same learned transformation to process tokens regardless of the sequence's overall length.

An attention mechanism allows a neural network to dynamically vary the strengths of connections (attention weights) between different token representations based on the actual input sequence.

Long-distance connections are crucial in language processing because relevant contextual words (like a pronoun and the noun it refers to) may occur many positions away from each other.

In an attention mechanism, a query vector determines what specific information a token is seeking by matching itself against the key vectors of other tokens.

A key vector provides a representation or "label" against which queries are matched, helping the model determine how relevant this specific token is to the querying token.

The value vector holds the actual token information that is mixed into the final weighted sum output, scaled by the attention weights computed between the queries and keys.

The basic compatibility score between a query and a key is computed using a dot product, which mathematically measures their similarity.

Query-key dot product scores are divided by the square root of the key dimension. This scaling limits score magnitudes to prevent the softmax function from saturating, which would otherwise cause vanishing gradients.

The softmax function converts unscaled query-key scores into positive attention weights that sum to 1, creating a valid probability distribution of where the token should "focus."

Without added positional information, standard self-attention is permutation equivariant: if you reorder the input sequence, the output sequence will reorder in the exact same way without changing the internal values.

Positional encodings are added to token embeddings so the Transformer model can distinguish the order of tokens, breaking permutation equivariance and allowing it to understand syntax and sequence.

A causal mask ensures that a token can only attend to itself and preceding positions (blocking future tokens) by setting future attention scores to negative infinity before applying the softmax function.

In self-attention, queries, keys, and values all come from the same sequence. In cross-attention, queries come from one sequence (e.g., a decoder) while keys and values are drawn from a separate sequence (e.g., an encoder).

Multi-head attention uses multiple different learned projections in parallel, allowing the model to simultaneously capture different types of relationships (e.g., one head for grammar, one for vocabulary).

When sequences are formatted as row vectors (shape N × D), the self-attention output is calculated as: Output = softmax_rows(Q * K^T / sqrt(dimension)) * V.

To compute a query vector for a token embedding x1 = [1, 0] with a weight matrix W_Q = [[1, 0], [0, 1]], you multiply the row vector by the matrix (x1 * W_Q) to get [1, 0]. Keys and values are computed using this same matrix multiplication step.

An unscaled attention score between a query and a key is their dot product. For example, if query q2 = [0, 1] and key k3 = [1, 2], the dot product is calculated as (0 * 1) + (1 * 2) = 2.

To scale scores, divide each by the square root of the key dimension (d_k). If an unscaled score row is [1, 0, 1] and d_k = 2, the scaled row is [1/sqrt(2), 0, 1/sqrt(2)], or approximately [0.7071, 0, 0.7071].

The overarching goal when evaluating a trained machine learning model is to ensure it generalizes well to unseen test data, rather than just memorizing the training set.

Parameters (like weights and biases) are learned autonomously from the data during training, whereas hyperparameters (like learning rate, batch size, and network architecture) are manually configured prior to training.

The validation set is strictly used to tune hyperparameters and guide architectural decisions during training, ensuring the final test set remains completely unseen until final evaluation.

The gap between perfect predictions and actual model performance is driven by three components: Bias, Variance, and inherent Noise in the dataset.

Bias is the source of test error that occurs when a model lacks the capacity or flexibility to capture the true underlying pattern of the data, resulting in underfitting.

Variance is the source of test error that occurs when a model is overly complex and highly sensitive to random noise in the training set, causing it to fit differently on different samples and resulting in overfitting.

As a model's capacity increases, its bias decreases (it fits the training data better), but its variance increases (it becomes more sensitive to noise). The optimal model balances these two forces.

Adding more training data generally reduces model variance by averaging out inherent noise and better representing the true input space, which improves overall generalization.

Precision measures the accuracy of positive predictions (True Positives / (True Positives + False Positives)). Recall measures how many actual positives the model successfully identified (True Positives / (True Positives + False Negatives)).

The F1 score is the harmonic mean of precision and recall. It provides a more balanced and reliable evaluation metric than standard accuracy, especially when dealing with highly imbalanced datasets.

Establishing a simple baseline model (like predicting the statistical mean for regression or the most frequent class for classification) provides a vital reference point to prove that a complex neural network is actually learning useful features.

L2 regularization adds a penalty to the loss function based on the sum of the squared parameter weights. This discourages large weights, pushing the network to learn a smoother, simpler output function that is less prone to overfitting.

Early stopping acts as implicit regularization by monitoring the validation loss during training and halting the process the moment the validation loss stops improving, keeping model weights from growing large enough to overfit the noise.

Dropout randomly zeroes out a percentage of hidden units during each training step. This acts as multiplicative noise, forcing the network to distribute information redundantly across many nodes rather than relying heavily on any single unit.

Because dropout is turned off during testing, the weights (or activations) must be mathematically scaled during inference to account for the fact that all hidden units are now active simultaneously.

Ensembling involves training multiple independent models (often from different random initializations) and averaging their predictions. This cancels out individual model errors and reduces prediction variance.

Data augmentation artificially expands the training set by applying random, label-preserving transformations (such as cropping, shifting, or adding noise) to existing examples, teaching the model to ignore irrelevant variations.

Label smoothing discourages a model from making overly confident predictions by assuming a small percentage of the training labels might be incorrect, smoothing the target probability distribution away from absolute 1s and 0s.

Transfer learning improves generalization on small datasets by initializing a model with weights pre-trained on a massive, related task. This places the parameters in a sensible region of space before adapting (fine-tuning) them to the specific new task.

When human labels are scarce, self-supervised learning creates its own training signals from raw data. Generative methods hide parts of the data (like masking a word) and ask the model to predict it, while contrastive methods train the model to distinguish between related and unrelated pairs of inputs.