1What is the Book Corpus?
A famous database full of books used to train language models
2What is Wikipedia used for in NLP training?
As a large-scale knowledge and language dataset
3What is Common Crawl?
An internet-scale dataset containing massive web text
4What is Reddit used for in AI training?
A major text source, providing over 40% of AI factual knowledge
5What is Stack Exchange mainly used for in training data?
For coding-related Q&A text
6What is a Bag of Words model?
A text representation method that counts words without context
7What is the main weakness of Bag of Words?
It loses word order and semantic meaning
8What is Word2vec?
A neural network method that learns word embeddings to capture relationships between words
9What do word embeddings capture?
Semantic closeness between words
10What is NLP?
Natural Language Processing, the field of teaching machines to understand and generate human language
11Why did GANs struggle with language generation?
Because text is sequential and GANs are not effective at long sequence modeling
12What were RNNs used for in early NLP?
Generating text on datasets like Shakespeare and Wikipedia
13What replaced RNNs in NLP?
Transformers
14What are the main steps in a Transformer?
Embeddings → Positional encodings → Attention → Output probabilities
15What does attention do in Transformers?
It models context and relationships between tokens
16What are n-gram models used for?
Specific task helpers in statistical language modeling
17What role does Word2vec play in NLP tasks?
Task-agnostic feature learner through embeddings
18What kind of task solver are BERT and GPT?
Transferable NLP task solvers
19What does BERT focus on?
Encoding, making it good for classification tasks
20What does GPT focus on?
Decoding, making it good for generation tasks
21What are scaling laws in language models?
Performance improves smoothly with model size, dataset size, and compute
22For optimal performance under scaling laws, what must be scaled together?
Model size, dataset size, and compute power
23What is the first step in LLM data preprocessing?
Collecting a raw corpus
24What does filtering and selection in preprocessing do?
Removes irrelevant data by language, metric, or statistic rules
25What is de-duplication in preprocessing?
Removing duplicate sentences or documents
26What is privacy reduction in preprocessing?
Detecting and removing personal information
27What is tokenization in NLP preprocessing?
Breaking text into tokens (smaller units) for model input
28What is the goal of prompt engineering?
To design prompts that guide an LLM’s responses effectively
29What are the components of prompt engineering?
Goal, return format, warning, context dump
30What is hallucination in LLMs?
When a model generates false or made-up information
31What is the problem of knowledge recency in LLMs?
They lack the most recent facts not present in training data
32Why are LLMs weak at numerical computation?
They are not optimized for exact arithmetic
33What does reasoning inconsistency mean in LLMs?
The model may produce invalid or illogical reasoning paths
34What is Thought Preference Optimization (TPO)?
A training method where LLM outputs are judged, and best/worst pairs are used for optimization
35What does DPO stand for in LLM optimization?
Direct Preference Optimization
36What is the purpose of embeddings in NLP?
To map words into vectors in a semantic space
37What are positional encodings in Transformers?
Signals that provide sequence order information to the model
38What is the formula for an n-gram probability estimate?
P(w_n | w_{n-1}, …, w_{n-k+1})
39What is transfer learning in NLP?
Using pre-trained models like BERT or GPT for new downstream tasks
40What does pre-training mean in LLMs?
Training on massive unlabeled text before fine-tuning
41What does fine-tuning mean in LLMs?
Adapting a pre-trained model to a specific task with additional data
42What is an embedding space?
A mathematical space where semantically similar tokens are close together
43What is the main advantage of Transformers over RNNs?
They process sequences in parallel instead of step-by-step
44What role does a tokenizer play in LLMs?
It converts text into tokens that the model can understand