Audio Lecture

To listen to a sound and tell you what it is (Audio Recognition).

A raw audio Waveform.

It analyzes the sound in two ways at once: the "Ears" Path (1D raw waveform/Wavegram) and the "Eyes" Path (2D "sound picture"/Logmel).

It mixes (concatenates) the clues from both the "Ears" and "Eyes" paths.

A Prediction of what sounds are in the clip.

To create new audio based on a "recipe" (Audio Generation).

A "recipe" or set of instructions (e.g., a mel spectrogram).

The MelGAN Generator (the "chef") takes the input "recipe" and generates audio from it.

A brand new, realistic Output Waveform.

To learn to organize music (in a latent space) and create new pieces.

The Encoder "squashes" the music into a simple code in the 2D Latent Space, and the Decoder learns to rebuild the music from that code.

You pick a new spot on the "sound map" (Latent Space) and give it to the Decoder, which turns it into music.

To create any sound you describe in text (Text-to-Audio Generation).

A Text prompt.

It starts with random noise, and a Latent Diffusion model (guided by text) slowly "denoises" the static, step-by-step, into a clear sound picture (spectrogram).

A Vocoder converts the generated sound picture (spectrogram) into an actual audio file.

To copy a person's voice and make it say new things (Speech Generation).

A short Speaker reference utterance (voice clip) and the new text.

It analyzes the reference voice's sound, then generates a "sound picture" (mel spectrogram) of the *new text* being spoken in *that voice*.

A Synthesized mel spectrogram.

To act as a high-quality Vocoder, turning "sound pictures" (spectrograms) into very realistic audio.

A spectrogram.

It generates the audio *one single tiny point (sample) at a time*, making sure each new point perfectly follows the previous ones.

An extremely high-quality and realistic-sounding Waveform.

To create highly emotional speech and control its timing (Expressive Speech Generation).

Text (what to say), a Speaker Prompt (whose voice), and a Style Prompt (what emotion).

A large Transformer model "disentangles" the voice from the emotion, allowing it to apply the chosen style (e.g., angry) to the chosen speaker's voice while saying the new text.

The most basic 1D graph of audio (amplitude over time); the "raw" data.

A 2D "picture" of sound plotting time (x-axis) vs. frequency (y-axis), with color showing amplitude.

A spectrogram where frequencies are converted to the "mel scale" to match human pitch perception.

The combination of letters that create a single sound (e.g., `s`, `p`, `oo`, `n`).

The smallest unit of sound in a language (e.g., the /s/, /p/, ː/, /n/ sounds).

A symbolic representation of music; instructions on what note to play, when, and how.

A 2D grid that visualizes MIDI data (pitch vs. time).

A stack of several pianorolls, one for each instrument.

An early, non-AI method that generates robotic speech by recreating formants (frequency peaks that distinguish vowels).

A method that builds speech by "stitching together" (concatenating) tiny snippets of pre-recorded audio from a large database.

A pre-neural method that generates speech based on statistical properties (acoustic features) paired with linguistic features (phonemes).