IST 664 Week 2 Key Terms

Given some corpus of natural language utterances, the frequency of any word is inversely proportional to its rank in the frequency table

- As you move down the rank (from most common to least common), the frequency drops quickly

frequency(word at rank (r)) = Constantank of word^s

Contiguous sequence of itmes that can be extracted from a given sample of text or speech. A number saying how long a string of words we have considered in it's construcution.

- Data Bias: leads to incorrect decisions or unfair treatment
- Passive Learning: may not have the opportunity to clarify any ambiguities in the data leading to a lack of understanding and poor performance on unseen data.
- Issues of explainability: Black box algorithms exist therefore lack of trust and more
- Issues of robostness: are threatened by adversarial attacks

Ability of models to efficiently make predictions on unseen data drawn from the same distribution as that of the training data from which it is trained on

Refers to the presence of zero valued words in the corpus which are words that do not appear in a given text. Has significant impacts on performance.

Technique that involves generating additional training data by applying random transformations to the existing training data.

This helps the model learn more robust and generalizable representations of the data and improves the model's performance on new data.

Technique that involves the use of knowledge and representations learned by a pre-trained model on one task and applying them to a different but related task.

Another approach is to use multitask learning where a single model is trained to perform multiple related tasks simultaneously

Technique used in NLP generalization to adapt models trained on one type of data called the source domain to a different type of data under the target domain

This kind of cross-learning is useful because real-world data can be highly diverse and unpredictable

Preventing a language model (n-gram models) from assigning zero probability to unseen events and handling out-of-vocab words

All counts that used to be 0 will now have a count of 1 (+1 to all existing words). Guarantees there will be no 0s.

Prob of word = number of times it showed up/total number of words

Adding a fractional count. Requires that we have a method for choosing k, and that can be done by optimizing for k by trying different values on a holdout set. Useful for text classification