Lecture 5 + Lecture 5 (IDK)

It is a foundation model for image and video generation that turns text prompts into images or new image–prompt combinations.

It was trained on about 2.3 billion image–text pairs.

It uses convolutional layers, self-attention layers, and a U-Net architecture.

It encodes the image by down-scaling to detect features and decodes by up-scaling to reconstruct details during denoising.

Noise is gradually added during training and then removed step-by-step during generation to form the final image.

It reverses the noising process to clean the noisy latent image step by step until it becomes a clear picture.

The prompt is tokenized and converted into vectors (tensors) by a text encoder such as CLIP.

It’s a compressed lower-dimensional space (e.g., 64×64×4) where images are represented for faster processing.

It starts from a Gaussian-sampled noisy latent instead of pixels to speed up generation and let the U-Net sculpt it into the target image.

It converts the final clean latent back into full-resolution RGB pixels (e.g., 512×512).

Contrastive Language-Image Pre-training.

CLIP converts text prompts into vector embeddings that guide the U-Net’s denoising to match the prompt’s meaning.

On over 400 million image-caption pairs using contrastive learning to bring matching text-image vectors closer and push mismatches apart.

A text encoder (often a Transformer) and an image encoder (e.g., Vision Transformer or ResNet).

It combines conditioned and unconditioned predictions with a guidance scale to make outputs more prompt-faithful.

It adds conditional control such as pose, depth, or reference to ensure the input condition appears in the output image or video.

It keeps a locked copy of the original diffusion model’s weights and trains a separate lightweight copy for the new condition.

After many training iterations the trainable branch eventually fits the specific input condition.

Low-Rank Adaptation, a parameter-efficient fine-tuning method that adds small trainable matrices to attention layers.

It lets models learn new styles or characters without retraining or overwriting the entire pretrained model.

Weight-Decomposed LoRA, a newer variant designed to increase training efficiency and adaptability.

An image-prompt conditioning adapter that freezes most U-Net components to let users create image variants efficiently.

A text-to-image/video adapter that improves on IP-Adapter to give extra control in generation.

A workflow where the prompt transitions smoothly from a starting prompt to an ending prompt during a generated video.

Reference sheets with multiple views or poses of a character to maintain character consistency across video frames.

CogVideo (2022) and Imagen Video (2022).

Phenaki (2022).

Sora.

A 2025 large-scale generative video model that uses a mixture-of-experts architecture.

A generative video tool whose training data has not been disclosed.

To keep characters, people, and objects stable and coherent across frames.

A tool or process that detects the main figure in an image to aid consistency.

The potential misuse of generative AI, such as for deepfakes or harmful content.

To create video sequences from various inputs, most commonly text prompts or images.

Phenaki

Maintaining temporal consistency (ensuring characters, objects, and styles remain coherent).

Prompt Travel

ControlNet

A lightweight neural network adapter that enables image prompting by guiding generation with a reference image.

By injecting image features into the diffusion model's cross-attention layers.

An efficient fine-tuning method to adapt large pre-trained models to specific characters or styles without full retraining.

As training data for creating a character-specific LoRA.

Animating a character from a reference image based on a driving pose video.

Video Autoencoder (VAE)

Diffusion Transformer (DiT) blocks

The convolutional U-Net structure.

Using different expert models or components specialized for different stages (e.g., high vs. low noise levels).

Disinformation (or deepfake commodification, scams, harassment).

By modifying the text embedding used for conditioning the diffusion model at different specified frame numbers.