IST 664 Week 1 Key Terms

The field of artificial intelligence that helps a computer to understand and interpret human language

Involves the machine's ability to comprehend and interpret human language. It deals with tasks like parsing grammar, understanding context, resolving ambiguity, sentiment analysis, and extracting meaning from text or speech

A way of representing words, sentences, or other data by automatic processing of text to extract a numerical representation of the meaning of that text, which takes the form of a vector. The key idea is that the position of a vector in that space captures semantic meaning — so similar concepts end up close together.

Used by search engines to understand what your search query means and then find you web pages that contain information about that topic, or used in email services to classify email importance

Process of reducing a word to its base or root form by stripping off suffixes (and sometimes prefixes). The goal is to treat different variations of a word as the same underlying term, to obtain a shorter form, even if that form doesn’t have any meaning.

a software process that turns data converted by NLU, into a natural language output that humans can understand, usually in text format

The process of reducing a group of words into their lemma

A large collection of text used to train or evaluate AI models

Refers to the total number of units of text in each piece of writing. Each word, punctuation mark, or number is counted

| V | = k x N^b

Any word or a token is made up of two things: The base form (morpheme) and an inflectional form like the prefix or the suffix. Reduces the word to its base form by removing the inflectional part (by using stemming or lemmatization).

every word in a text corpus is assigned a unique vector that consists of 0 and 1. Allows the machine learning model to recognize each word uniquely through its vector

Converts a whole piece of text into fixed-length vectors. Done by counting the number of times a word has appeared in the document

Library for filtering the raw, unstructured text data and build a corpus for training different machine learning models

Words can have multiple meanings depending on context.

For example, “bank” can mean a financial institution or the side of a river.

Understanding a sarcastic remark requires grasping the speaker’s intended opposite meaning.

Language is not static; it evolves over time with new words, slang, and cultural references.

Training NLP models requires large amounts of diverse data. Obtaining and annotating such data can be expensive and time-consuming