Lecture 3 + Lecture 3 (IDK) INFOMAIR

Using feature vectors (e.g., [2,110,2,1])

Representing categorical values with 0/1 (true or false) vectors

Sentence length, number of likes

Whether text is capitalized (yes/no)

Country, genre of text

Classify new examples based on the most similar training examples

Stores all training instances and predicts using nearby points

Distance metric, number of neighbors, weighting function (optional), fitting method

Euclidean distance

It is highly sensitive to outliers and can overfit

Model always predicts the majority label (underfitting)

Labels of points should be similar to nearby points, all features equally important, sensitive to feature scaling

By predicting the mean of the neighbors’ values

Random choice, fallback to 1-NN, or use majority class from training data

Training is fast and flexible, but predictions are slow and require storing all data

Decision trees learn axis-aligned models; KNN stores data and compares new points, allowing complex boundaries but sensitivity to irrelevant features

Automatic processing and analysis of natural language

Machine translation, IBM Watson Q&A

Polysemy, syntactic ambiguity, and continuous language change

Segmenting text into tokens (often words)

No spaces (Japanese), multiword expressions, contractions

Stopword removal, lemmatization, stemming, lowercasing, removing infrequent words

Reduce vocabulary size while retaining useful signals

Intersection over union of token sets; ignores frequency and order

Numerical vectors representing words, documents, or sentences

Euclidean distance (L2) and cosine similarity

The angle between vectors, ranging from -1 (opposite) to 1 (same direction)

Each word is mapped to a unique ID with a vector of all zeros except one position

Words with similar contexts tend to have similar meanings

They are sparse (many zeros) and frequency-based

Dense, learned vector representations of words where dimensions are less interpretable

They capture semantic similarity between words in a compact form

Word vectors that depend on surrounding context (e.g., “bank” in money vs. river)

It looks at the K nearest neighbors and assigns the majority label (classification) or average value (regression).

It defines what counts as “near” (e.g., Euclidean, Manhattan, cosine distance).

Without scaling, features with larger numeric ranges dominate the distance metric.

Model becomes very flexible, but prone to overfitting.

Model smooths boundaries, reducing overfitting, but may underfit by ignoring local patterns.

The prediction is the most common label among the K neighbors.

Neighbors closer to the query point are given higher weight in the prediction.

A method (like KNN) that defers generalization until prediction time.

Learning a model (like decision trees, SVMs) during training time, so predictions are fast.

Distances become less meaningful due to the curse of dimensionality.

As features increase, all points appear similarly distant, reducing effectiveness.

Mean, weighted mean, or sometimes median of neighbor values.

When a sentence can be parsed in multiple ways (e.g., "I saw the man with a telescope").

When a word or phrase has multiple meanings (e.g., “bank”).

To reduce duplicate forms of the same word (e.g., “Dog” vs. “dog”).

They add noise and rarely improve performance in models.

Reducing words to their base form using vocabularyrammar rules (e.g., “running” → “run”).

A crude way of chopping words to root form by removing suffixes (e.g., “running” → “runn”).

Common words (e.g., “the”, “is”) often removed because they carry little meaning.

Stopwords can carry syntactic or semantic meaning in certain tasks (e.g., “not happy”).

Documents as vectors of word counts, ignoring order.

It ignores word order, context, and semantics.

Term frequency–inverse document frequency; emphasizes important but not too common words.

Sparse vectors have many zeros (e.g., one-hot), dense vectors have few or no zeros (e.g., embeddings).

It ignores vector length and focuses on orientation, making it robust to differences in text length.

Large corpora of text, optimizing word vectors to capture distributional similarity.

“king – man + woman ≈ queen”

Static embeddings (Word2Vec, GloVe) assign one vector per word, while contextual embeddings (BERT, GPT) adjust vectors based on surrounding words.

No, it increases the risk of overfitting.

We may not want to use certain features (e.g., demographics), and fewer features improve interpretability.

Manhattan uses absolute value, while Euclidian uses squared differences.

A metric that combines or generalizes Manhattan and Euclidian distances.

The label of a data point should be similar to the labels of nearby points.

All features are equally important.

Removing common words (e.g., a, an, the, it) using a list or by filtering very frequent words.

Reducing different forms of a word (like sang, sung, sings) to its base or dictionary form (sing).

Stripping the endings off a word (e.g., 'running' becomes 'run').

Converting all text to lowercase (e.g., 'Running' to 'running').

They are filtered out because they occur too rarely (e.g., in less than 10 documents) to be useful.

The minimum number of editing operations (insertion, deletion, substitution) needed to transform one string into another.

Intersection / Union.

It ignores the frequency and order of words.

Words that occur in similar contexts tend to have similar meanings.