VIST 375 - Final Exam Review

Cornea - the camera lens (focus incoming light)
Iris - the shutter/aperture control (controls the amount of light)
Pupil - the aperture (the opening that light enters from)
Retina - the film/image sensor (receive light)
Optic Nerve - data cable/SD card (transmits visual info to the brain for processing)

rods; cones

Red (longer wavelengths) to Violet (shorter wavelengths)

- highly sensitive to low levels of light, responsible for scotopic (night) vision
- mainly involved in detecting motion and vision in dimly lit conditions
- do not enable color vision
- more abundant in the peripheral regions of the retina

- sensitive to color and detail, providing photopic (daytime) vision
- primarily responsible for color vision, capturing detail
- function best in well-lit conditions
- most abundant in the central part of the retina, particularly in the fovea
- three types of cones:
- S-cones: sensitive to short wavelengths (blue light)
- M-cones: sensitive to medium wavelengths (green light)
- L-cones: sensitive to long wavelengths (red light)

Info on computers is digital and needs to be converted from continuous (analog) to discrete signals
- needs sampling and quantization

Sampling: sample the signla at regular intervals
Sampling resolution: how frequently/densely the signal is sampled

Quantization: round each continuous value to a discrete level
Quantization resolution: how many discrete levels are used to approximate the value

Spatial resolution: the number of pixels used to sample the given image space, typically measured in pixels per inch (PPI) or dots per inch (DPI)

- grayscale image = an intensity function defined on a rectangular region
- or discretely, a 2D matrix
- Coordinates: H rows and W columns, upper left corner (0,0)
- each pixel's intensity value normalized to [0,1], or quantized to [0,255] on a 8-bit computer

- color image = 3 grayscale images (R, G, B channels)
- stronger channel -> brighter intensity

- RGB (Light Based): additive color model
- colors are created by adding light of the three primary colors
- R = (1,0,0); G = (0,1,0); B = (0,0,1); White = (1,1,1); Black (0,0,0)
- best suited for devices that emit light (computer monitors, TVs, etc.)

- CMY (Pigment Based): subtractive color model
- created by subtracting volors from white light
- start with default white (0,0,0)
- adding C (cyan) ink absorbs R (red) light
- adding M (magenta) ink absorbs G (green) light
- adding Y (yellow) ink absorbs B (blue) light
- c = m = y = 0 -> white
- c = m = y = 1 -> black
- (c,m,y) = (1,0,0) -> cyan; (0,1,0) -> magenta; (0,0,1) -> yellow; (1,1,0) -> blue; etc.
- best suitable for printing (inkjet printers, laser printers, color photocopies, etc.)

- an extension of CMY
- added a black (Key) component
- most printers use CMYK:
- black not generated by mixing C+M+Y inks, but by K directly
- cheaper
- better quality in black-related regions
- avoiding over-saturation

- H: Hue [0, 360°], specify the type of color
- S: Saturation [0,1], indicates the amount of vividnessray/colorfulness of the color
- V: Value/Brightness [0,100%]
- L, I: Lightness/Intensity [0 (black), 1 (white)]

- HSI: often used in illumination correction, color-based object tracking, image enhancement, etc.
- HSL: preferred in graphic deisgn and user interface
- HSV: usefule when color differentiation and separation are important (image processing, computer graphics, computer vision)

- Lossless compression: PNG, TIFF, BMP, GIF
- identify and replace repeated patterns or sequences with simpler or shorter symbols
- offers perfect reconstruction

- Lossy compression: JPEG, MPEG
- approximates data by disgarding some details (high-frequency info)
- sacrifices some quality for greater size reduction

- aggregated image intensities
- variation of pixel values

- they are:
- useful for quantitative measures of image quality
- basis for automated processing and analysis about images

- the mean value of an image
- the median is the pixel value such that one half of the image pixels have values greater than the median and the other half have values less than the median

- dynamic range (range) of an image is the difference between its maximum and minimum pixel values
image variance: measures the distribution of pixel values
- if pixel values are close to the mean value = small variance
- a pure-color image has 0 variance

- a histogram that plots the number of pixels for each pixel value
- visualizes an image's intensity distribution
- x-axis: pixel values = 0 - 255
- y-axis: the # of occurrence of that corresponding pixel value

- a scheme that "normalizes" the histogram
- Main idea:
- Map intensity values to their corresponding "percentiles"
- Ex. if a pixel value (v) > 75% of the others, then the transfer function (f) should map v -> f(v) = 75% of the brightest value, namely 255*75% = 191
- original brightest pixel value = 100% (white)
- -original darkest pixel value = 0% (black)

- local operations such as correlation and convolution operators modify the pixel value of eacch pixel locally, typically using values from its neighbors

- correlation (similar to convolution, only with flips)
- can be considered as pattern matching that detects the patteren (mask/kernal) in a given image
- in deep neural networks, kernals are learned to detect various interesting pattern features for recognition/classification

- how to calculate:
- multiply mask value to corresponding input image value and altogether

Local Pass filters:
- allow low-frequency info to pass, filter out high-frequency info
- keep global structures; remove details
- Box, averaging filters, Gaussian filters

High Pass filters:
- allow high-frequency info to pass, filter out low-frequency info
- capture detail or local change; remove global structures
- Derivatives, Laplacian, edge detectors, image sharpening operators

- must add "extra" border pixels if wanting to mask an edge pixel
- 0-padding: each new virtual cell value = 0
- replicate padding: each new virtual cell value = boundary pixel value

Upsampling: increase spatial resolution usually through interpolation (nearest neighbors, linear interpolation, etc.)

Downsampling: decrease spatial resolution
- to avoid aliasing, use smoothing then downsampling

- to combine visual elements from separate sources into a single image
- usually use a mask or matte image to control how/where to show/hidelend regions from both images
- mass/matte can be a binary image or grayscale image

- AI: programs created to mimic human intelligence; aims to build programs to learn functions from observed data
- Machine Learning (ML): subset of AI that focuses on development of algorithms and models that can learn from data and make decisions or predicitions on that data
- Deep Learning (DL): subset of ML that builds model using artificial neural netowrks, which aim to simulate the human brain, allowing it to learn from large amount of data
- Neural Networks: network composed of many nodes; each node (neuron) connects to others and has an associated activation mechanism; inspired by human brain, mimicking biological neurons
- Multi-layer Perceptron (MLP): type of Neural Network where nodes are arranged by layers; consist of an input layer, output layer, and hidden layers; linear combination, nonlinear activation
- Convolutional Neural Networks (CNNs):
- Convolutional Layer: a neuron learns a convolutional filter; learn multiple filters to get multiple channels
- Activation: each neuron receives signals from previous layers, and produces a response through a nonlinear activation function
- Pooling Layer: to detext the precense of patterns, ignoring possitional info
- Flattening: output of previous layers are learned 2D feature maps, which are flattened into an I D long vector, then passed onto a fully connected network (MLP) for classification

- points: cartesian coordinates; polar coordinates
- lines/curves: piecewise linear interpolation
- piecewise: curves = consecutive line segments
- piecewise linear, not smooth

- simple analytic shapes: circles, rectangles, spheres, tori conic senctions
- may be directly described using closed-form representations
- more complicated shapes:
- can be composed via Boolean operations (addition or union, intersection, or subtraction (difference)

- polygonal mesh -> machine oriented representation
- CSG -> user-oriented representation
- store the "logic of the shape"
- CSG modeling system: building blocks, Boolean operations
- widely used in 3DMax, Maya, etc. as their modeling scheme:
- support user-intervention
- good for simple shapes

- hierarchical spacial representations
- quadtrees for 2D; octrees for 3D
- adaptive and can save storage cost (compared with directly using grids)

- using computer programs to createender images and visual effects

- 3 main techniques currently utilized in CG rendering:
- Rasterization: forward approach
- start from 3D objects/scenes (meshes), project triangles onto 2D pixels, then apply shading/textures to pixels
- map 3D objects -> 2D pixels
- standard approach for realtime graphics
- Ray tracing: high-quality visual output, especially for better effects on reflection, refraction, shadows, etc.
- Neural rendering: a more recent addition; leverages ML for rendering
- based on Image-to-Image, 3D object manipulation, rendering from sparse, semantic input, etc.
- rasterization and ray tracing take 3D mesh models as input and output a pixelized image suitable for display on screens

- 3D-to-2D projection: the pipeline to assemble the transformations that maps 3D points to 2D screen coordinates
- viewing transformations: place the camera
- model transformations: move objects to position within the view volume
- projection transformations: project 3D points onto 2D image plane
- viewport transformations: map image plane to actual coordinates on the screen's viewport

- surfaces in 3D have normals defined on each point
- normal: a vector that is perpendicular to the tangent plane at a point on the surface
- for triangular mesh:
- face normal: computed as cross-product of two edge vectors of the triangle
- vertex normal: typically approximated as a weighted average of the normal of the faces adjacent to the vertex

- shading models: methods used in CG to simulate how light interects with surfaces; determine how light and color are calculated on the rendered 3D surfaces
- common shading models:
- flat shading: no interpolation; each face has one color, faceted-look
- gouraud shading: color calculated at face vertices, then linear interpolation inside face, smoother appearance
- phong shading: normal interpolated inside face, color computed pixel-by-pixel, further enhanced smoothness and accuracy (especially in areas with specular highlights)

- support 3 types of reflections:
- ambient reflection: global environment illumination
- diffuse reflection: when light hits a rough surface and is scattered evenly in all directions
- specular reflection: when light hits a shiny surface and is reflected in a concentrated beam, yielding bright highlights

- reflection intensity determined by the angle between incoming light direction and the surface normal
- surface appears brighter when angle is smaller and is brightest when the light is perpendicular to the surface
- reflection scattered evenly in all directions
- influenced by the light source position, but independent of the observer's or camera's position

- on shiny surfaces, incoming light reflected in a concentrated direction, preserving the angle of incidence (law of reflection)
- intensity influenced by both the light source position and the observer's or camera's position

- Ray tracing/casting: backward approach; shoot rays from the eye through each image pixels
- ray tracing casted secondary rays to simulate extra reflection, refraction, etc.
- ray casting: phone shading + shadow rays to calculate intensity
- if in shadow: only ambient component
- if not in shadow: apply the complete shading model
- ray tracing = ray casting + recursively tracing secondary rays

- classic vs Neural rendering:
- classic: algoritms based on principles of light physics and geometry
- Neural: data-driven approaches; need to provide training data; convenient editing (changing the lighting or viewpoint, modifying objects/scenes from the semantic aspect, etc.)

- modeling -> texturing -> lighting:
- modeling: geometry no longer needs meshes but is implicitly encoded in neural networks (using NeRF or other Neural Representation schemes)
- texturing: no longer uses bitmap images + texture mapping; now directly generated by neural networks
- lighting: no longer uses shading models (Gouraud/Phong); now typically learned together with geometry from the data

- Neural Radiance Field:
- field: a quantity (value or vector) defined on each point in space
- Neural Field: using a neural network to represent the field
- radiance field encoding both geometry and light intensity (radiance)
- can be trained using a set of images with known camera poses
- can be used to synthesize the scene from any new view angles

- pick a reference photo, use its camera pose to perform a volume rendering and generate a rendered image
- compare each pixel of the reference and rendered image and use this as feedback to modify NeRF
- repeat process with the entire set of available photographs

- both have similar network structures (encoder and decoder)
- a fundamental tool for unsupervised learning
- deterministic vs probabilistic:
- AE is deterministic; each image is encoded into a low-dimensional latent code
- VAE is aprobabilistic; each image is mapped to a distribution
- latent space:
- AE: lacks structure; each point -> an image; space not continuously defined
- VAE: regularized, continuous and smooth; better suited for interpolation and generation of new samples

- Pros:
- simple architecture, simple training
- efficient at dimensionality reduction and feature learning
- Cons:
- latent space may not be well-structured; may not generalize well beyond the training data

- Pros:
- structured latent space allows for interpolation and new data generation
- more robust to overfitting
- Cons:
- more complex training
- generation quality (compared with GANs, DMs)

- involves two networks, a generator (G) and a discriminator (D), that are trained simultaneously
- G aims to fool D in generating fake data
- D aims to classify real data from fake data generated from G
- G uses D's feedback to improve its generation capability
- D uses G's new generated data to also improve its classification capability
- evolve iteratively together
- generate higher-quality data than VAEs
- Pros:
- high-quality; flexibility in generating complex data
- Cons:
- more complex than VAEs; hard to train

- create data by simulating and reversing diffusion process
- inspired by the physical process of diffusion, where particles spread from areas of higher concentration (original image) to lower concentration (white noise) over time
- involves two processes: a forward process that adds noise to the data, and a reverse process that learns to remove the noise and reconstruct the original data or create new data
- Pros:
- high-quality; good diversity
- better stabiloty during training than GANs
- Cons:
- complex architecture; expensive to build and interpret
- slower inference: requires multiple steps to generate an image