Deep Learning: Backpropagation and Initialization

1) Allows us to calculate gradients (derivatives) efficiently
2) Helps us initialize parameters to avoid huge losses, and avoid extremely large or small gradients

Forward pass and Backwards pass

The network processes the input data and computes the activations of all hidden units layer by layer until the output is obtained. (Like betas and omegas)
The activations at each layer are saved to be used during the second phase of backpropagation to compute the gradients efficiently.

The network uses the stored activations from the backwards pass to compute the gradients of the loss with respect to each parameter efficiently.
This is done by propogating the error from the output layer back through the network to the input layer.

- Vanishing gradient: gradients and weights will get smaller and smaller (if the variance is very small)
- Exploding Gradient: If the variance is very large, then each weight or element (like beta or omega) will get very large, much larger magnitude than the inputs.
In either scenario we run into the problem of floating point numbers not being able to repersent the artithmetic.
Goal: $h_k \approx h_{k-1}$

• Variance scaling: He initialization compensates for ReLU's halving effect by doubling the variance of the random weights.
• Formula: The weights are initialized from a zero-mean distribution (Gaussian or uniform) scaled by the number of input connections ($(n_{in}\)$.

When deep learning frameworks like Pytorch calculate the derivatives automatically. Each functional component in the framework knows how to compute its own derivative. These frameworks use parallelization with modern GPUs do do these things quickly.

Generalization, we wnat our trained model to perform well not just on the training data but also in unseen test data.

1) Inherent uncertainty
2) Amount of training data
3) Choice of model

It allows the model to learn the task, used to fit the model parameters. Helps learn model parameters with things like stochastic gradient descent.

It helps choose which model is the best, helps tune hyperparameters and make decisions surrounding the model architecture. Like comparing the number of layers of nuerons.

Used to evaluate the final model's generalization performance.

The model performs well on the training data but fails to generalize to unseen data.
What can happen is the training loss decreases steadily toward zero, then the test loss decreases a first but then subsequently increases as the model becomes increasingly confident about its (wrong) predictions.

Noise, Bias, and Variance.

There are multiple possible valid outputs y for each input x
May arise when:
- There is a genuine stochastic element to the data generation process
- Because some of the data is mislabled
- Because there are further explanatory variables that were not observed

The model is not flexible enough to fit the true function perfectly, the error introduced by approximating a real-world problem, which may be complex, by a simplified model. Can cause underfitting, where the model is too simple to capture the underlying patterns in the data.
- Can be fixed by adding more layers or nuerons

When we have limited training examples, and there is no way to distinguish systematic changes in underlying function from noise in the underlying data.
- Can be fixed by adding more training data

For fixed-size training data, the variance typically increases as the model capacity increases. So by trying to solve the bias of the data by increasing the number of hidden layers, we increase the variance.

Usually chosen empirically, we train the models with different parameters and keep the one that performs the best. Use validation set to test trained model on then modify hyperparameters.

Accuracy is the proportion of correct predictions. It is the number of correct predictions divided by the total number of predictions.
$Accuracy = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}}$
- it is a good metric when classes are balanced, a bad metric when not.
- ex: a model gets 95% accuracy when predicting the same class for a dataset that is 95% that class

True Positive (TP): Correctly predicted the positive class
False Positive (FP): Incorrectly predicted the negative class as positive
True Negative (TN): Correctly predicted the negative class
False Negative (FN): Incorrectly predicted the positive class as negative
$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$

The proportion of positive predictions that are correct.
$Precision = \frac{TP}{TP + FP}$

The proportion of positive instances that are correctly predicted
$Recall = \frac{TP}{TP + FN}$

Predicts positives only when it's absolutley certain, so it never makes a false positive, but misses most true positives.

Predicts many more positives than truly exist, catching almost all real positives but generating many false alarms.

A harmonic mean of the precision and recall?
$F1-Score = 2 \times \frac{Precision \times Recall}{Precision + Recall}$

The average of the absolute differences between the predicted and the actual values
$\frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$

The average of the squared differences between the predicted and actual values
$\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

A technique used to asses how the results of a statistical analysis will generalize to an independent dataset. Common example method: k-fold cross-validation.

A family of methods that reduce the generalization gap between training and test performance.
Strictly speaking, regularization involves adding explicit terms to the loss function that favor certain parameter choices, however this term is commonly used to refer to any strategy that improves generalization.

L2 punishes the sum of the squares of the parameter values, favoring smaller parameter values. Is usually applied to the weights but not the biases and is hence referred to as a weight decay term.
$g[\phi] = argmin \sum_{i=1}^{n} l_i[x_i,y_i] + \lambda \sum_{j}\phi^2_j$

Refers to stopping the training prodcedure of a model before it has fully convereged.
This can reduce overfitting if the model has already captured the coarse shape of the underlying function but has not yet had time to overfit to the noise.

Another approach to reducing the generalization gap. To build several models and average their predictions, a group of such models is known as an ensemble. Reliably improves test performance at the cost of training and storing multiple models and performing ingerence multiple times.
You can train different models through random initialization or re-sampling the training data.

Another approach to reducing generalization gap. Dropout clams a random subset (typically 50%) of hidden units to zero at each iteration of SGD. This makes the network less dependent on any given hidden unit and encourages the weights to have smaller magnitudes so that the change in the function due to the presence or absence of any specific hidden unit is reduced.

When training data is limited, other datasets can be expoited to improve performance. In transfer learning, the network is pretrained to perform a related secondary task for which data are more plentiful. The resulting model is then adapted to the original task.

A network is trained to solve several problems concurrently. Some things are learned better when done simulataneously.

Two Types: Generative and Constrastive
Generative: Part of each data example is masked, and the secondary task is to predict the missing part
Constrastive: Pairs of examples with commonalities are compared to unrelated pairs.

When we want to expand the dataset, but we are limited.
We can often transform each input data example in such a way that the label stays the same, like rotating or flipping and image horizontally.