Universitat Politècnica de Catalunya, profile picture
Uploaded byUniversitat Politècnica de Catalunya
PDF, PPTX810 views

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

The document discusses deep generative learning, contrasting discriminative and generative models, and explores various architectures including GANs, VAEs, and diffusion models for tasks such as image and music generation. It outlines the technical elements involved in training these models and acknowledges contributions from various researchers. The content emphasizes the advancements and applications of generative models across different domains, providing references to recent noteworthy works.

Embed presentation

Download as PDF, PPTX
1 / 115
2 / 115
Most read
3 / 115
4 / 115
Most read
5 / 115
6 / 115
7 / 115
8 / 115
9 / 115
10 / 115
11 / 115
12 / 115
13 / 115
14 / 115
15 / 115
16 / 115
17 / 115
18 / 115
19 / 115
20 / 115
21 / 115
22 / 115
23 / 115
24 / 115
Most read
25 / 115
26 / 115
27 / 115
28 / 115
29 / 115
30 / 115
31 / 115
32 / 115
33 / 115
34 / 115
35 / 115
36 / 115
37 / 115
38 / 115
39 / 115
40 / 115
41 / 115
42 / 115
43 / 115
44 / 115
45 / 115
46 / 115
47 / 115
48 / 115
49 / 115
50 / 115
51 / 115
52 / 115
53 / 115
54 / 115
55 / 115
56 / 115
57 / 115
58 / 115
59 / 115
60 / 115
61 / 115
62 / 115
63 / 115
64 / 115
65 / 115
66 / 115
67 / 115
68 / 115
69 / 115
70 / 115
71 / 115
72 / 115
73 / 115
74 / 115
75 / 115
76 / 115
77 / 115
78 / 115
79 / 115
80 / 115
81 / 115
82 / 115
83 / 115
84 / 115
85 / 115
86 / 115
87 / 115
88 / 115
89 / 115
90 / 115
91 / 115
92 / 115
93 / 115
94 / 115
95 / 115
96 / 115
97 / 115
98 / 115
99 / 115
100 / 115
101 / 115
102 / 115
103 / 115
104 / 115
105 / 115
106 / 115
107 / 115
108 / 115
109 / 115
110 / 115
111 / 115
112 / 115
113 / 115
114 / 115
115 / 115
Deep Generative Learning for All (a.k.a. The GenAI Hype) Xavier Giro-i-Nieto @DocXavi xavigiro.upc@gmail.com Associate Professor (on leave) Universitat Politècnica de Catalunya Institut de Robòtica Industrial ELLIS Unit Barcelona
2 Acknowledgements Santiago Pascual @santty128 PhD 2019 Universitat Politecnica de Catalunya Technical University of Catalonia Albert Pumarola @AlbertPumarola PhD 2021 Institut de Robòtica Industrial (IRI) Universitat Politècnica de Catalunya (UPC) Kevin McGuinness Research Fellow Insight Centre for Data Analytics Dublin City University Gerard I. Gállego PhD Student Universitat Politècnica de Catalunya gerard.ion.gallego@upc.edu @geiongallego
3 Acknowledgements Eduard Ramon Applied Scientist Amazon Barcelona @eram1205 Wentong Liao Applied Scientist Amazon Barcelona Ciprian Corneanu Applied Scientist Amazon Seattle Laia Tarrés PhD Student Universita Pompeu Fabra laia.tarres@upf.edu
Outline 1. Motivation 2. Discriminative vs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Latent variable 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
Image generation 5 #StyleGAN3 (NVIDIA) Karras, Tero, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Alias-free generative adversarial networks." NeurIPS 2021. [code]
6 #DiT (UC Berkeley, NYU) Peebles, William, and Saining Xie. "Scalable Diffusion Models with Transformers." ICCV 2023. Image generation
7 #Imagen2 (Google Deepmind) Blog post Text-to-Image (T2I) generation “A shot of a 32-year old female, up and coming conservationist in a jungle; athletic with short, curly hair and a warm smile
8 #DALL-E-2 (OpenAI) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen "Hierarchical Text-Conditional Image Generation with CLIP Latents." 2022. [blog] #DALL·E-3 (OpenAI) James Betker, Gabriel Goh, et al, “Improving Image Generation with Better Captions” 2023 [blog] Text-to-Image (T2I) generation ““An expressive oil painting of a basketball player dunking, depicted as an explosion of a nebula.
9 Text-to-Music (T2M) generation (Stability AI) Evans, Zach, Julian D. Parker, C. J. Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. "Long-form music generation with latent diffusion." arXiv 2024.
10 Text-to-Video (T2V) generation #Sora (OpenAI) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, Aditya Ramesh. Video generation models as world simulators. OpenAI 2024.
Human Motion Transfer 11 #DreamMoving Mengyang Feng, Jinlin Liu, Kai Yu, et al., DreaMoving: A Human Video Generation Framework based on Diffusion Models. arXIv 2023 “A girl, smiling, dancing in a wooden house, wearing sweater, and long pants.” “A girl, smiling, dancing in the park with golden leaves in autumn, wearing light blue dress.” “A girl, smiling, dancing in Times Square, wearing dress-like white shirt, with long sleeves, long pants.”
12 Image Edition & Text-to-Video (T2V) generation #EMU (Meta) Girdhar, Rohit, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. "Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning." arXiv 2023. [blog]
13 #Imagine Flash (Meta) Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Vajda, and Ali Thabet. Imagine Flash: Accelerating Emu Diffusion Models with Backward Distillation. Meta 2024. Text-to-Image (T2I) generation Albert Pumarola @AlbertPumarola PhD 2021 Institut de Robòtica Industrial (IRI) Universitat Politècnica de Catalunya (UPC)
Outline 1. Motivation 2. Discriminative vs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Latent variable 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
15 Discriminative vs Generative Models Philip Isola, Generative Models of Images. MIT 2023.
Outline 1. Motivation 2. Discriminative vs Generative Models a. Pθ (Y|X): Discriminative Models b. Pθ (X): Generative Models c. Pθ (X|Y): Conditioned Generative Models 3. Latent variable 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
Pθ (Y|X): Discriminative Models 17 Slide credit: Albert Pumarola (UPC 2019) Classification Regression Text Prob. of being a Potential Customer Image Audio Speech Translation Jim Carrey What Language? X=Data Y=Labels θ = Model parameters Discriminative Modeling Pθ (Y|X)
18 0.01 0.09 0.9 input Network (θ) output class Figure credit: Javier Ruiz (UPC TelecomBCN) Discriminative model: Tell me the probability of some ‘Y’ responses given ‘X’ inputs. Pθ (Y | X = [pixel1 , pixel2 , …, pixel784 ]) Pθ (Y|X): Discriminative Models
Outline 1. Motivation 2. Discriminative vs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Sampling 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
20 Slide Concept: Albert Pumarola (UPC 2019) Pθ (X): Generative Models Classification Regression Generative Text Prob. of being a Potential Customer “What about Ron magic?” offered Ron. To Harry, Ron was loud, slow and soft bird. Harry did not like to think about birds. Image Audio Language Translation Music Composer and Interpreter MuseNet Sample Jim Carrey What Language? Discriminative Modeling Pθ (Y|X) Generative Modeling Pθ (X) X=Data Y=Labels θ = Model parameters
Each real sample xi comes from an M-dimensional probability distribution P(X). X = {x1 , x2 , …, xN } Pθ (X): Generative Models
22 1) We want our model with parameters θ to output samples with distribution Pθ (X), matching the distribution of our training data P(X). 2) We can sample points from Pθ (X) plausibly looking how P(X) distributed. P(X) Distribution of training data Pλ,μ,σ (X) Distribution of training data Example: Gaussian Mixture Models (GMM) Pθ (X): Generative Models
23 What are the parameters θ we need to estimate in deep neural networks ? θ = (weights & biases) output Network (θ) ? Pθ (X): Generative Models
Outline 1. Motivation 2. Discriminative vs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Sampling 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
Pθ (X|Y): Conditioned Generative Models Joint probabilities P(X|Y) to model conditioning variables on the generative process: X = {x1 , x2 , …, xN } Y = {y1 , y2 , …, yN } DOG CAT TRUCK PIZZA THRILLER SCI-FI HISTORY /aa/ /e/ /o/
Outline 1. Motivation 2. Discriminative vs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Sampling 4. Architectures a. Generative Adversarial Networks (GANs) b. Auto-regressive c. Variational Autoencoders (VAEs) d. Diffusion
Our learned model should be able to make up new samples from the distribution, not just copy and paste existing samples! 27 Figure from NIPS 2016 Tutorial: Generative Adversarial Networks (I. Goodfellow) Sampling
Philip Isola, Generative Models of Images. MIT 2023. Sampling
Slide concept: Albert Pumarola (UPC 2019) Learn Sample Out Training Dataset Generated Samples Feature space Manifold Pθ (X) “Model the data distribution so that we can sample new points out of the distribution” Sampling
Sampling z Generated Samples How could we generate diverse samples from a deterministic deep neural network ? Generator (θ)
Sampling Generated Samples How could we generate diverse samples from a deterministic deep neural network ? Generator (θ) Sample z from a known prior, for example, a multivariate normal distribution N(0, I). Example: dim(z)=2 x’ z
Slide concept: Albert Pumarola (UPC 2019) Learn Training Dataset Interpolated Samples Feature space Manifold Pθ (X) Traversing the learned manifold through interpolation. Interpolation
Disentanglement Philip Isola, Generative Models of Images. MIT 2023.
Disentanglement Philip Isola, Generative Models of Images. MIT 2023.
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ■ Generator & Discriminator Networks ■ Adversarial Training ○ Auto-regressive ○ Variational Autoencoders (VAEs) ○ Diffusion
36 Credit: Santiago Pascual [slides] [video]
37 Generator & Discriminator We have two modules: Generator (G) and Discriminator (D). ● They “fight” against each other during training→ Adversarial Learning D’s goal: Classify between real samples and those produced by G. G’s goal: Fool D to missclassify. Goodfellow, Ian J., Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative Adversarial Nets." NeurIPS 2014.
38 Discriminator Discriminator network D → binary classifier between real (x) and generated (x’). samples. Generated (1) Discriminator (θ) x’ Discriminator (θ) x Real (0)
39 Generator Real world samples Database Discriminator Real Loss Latent random variable Sample Sample Generated z Generator & Discriminator
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ■ Generator & Discriminator Networks ■ Adversarial Training ○ Auto-regressive ○ Variational Autoencoders (VAEs) ○ Diffusion
Imagine we have a counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. 100 100 FAKE: It’s not even green Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
Imagine we have a counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. 100 100 FAKE: There is no watermark Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
Imagine we have a counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. 100 100 FAKE: Watermark should be rounded Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
Imagine we have a counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. After enough iterations, and if the counterfeiter is good enough (in terms of G network it means “has enough parameters”), the police should be confused. REAL? FAKE? Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
Adversarial Training Generator Real world images Discriminator Real Loss Latent random variable Sample Sample Generated Alternate between training the discriminator and generator Neural Network Neural Network Figure: Kevin McGuinness (DCU)
Adversarial Training: Discriminator Generator Real world images Discriminator Real Loss Latent random variable Sample Sample Generated 1. Fix generator weights, draw samples from both real world and generated images 2. Train discriminator to distinguish between real world and generated images Backprop error to update discriminator weights Figure: Kevin McGuinness (DCU)
Adversarial Training: Discriminator Generator Discriminator Real Loss Latent random variable Sample Backprop error to update discriminator weights Figure: Kevin McGuinness (DCU) Generated Consider a binary encoding of “1” (Real) and “0” (Fake). Which of these two values should the discriminator predict when we train the discriminator with a generated image ?
Adversarial Training: Generator 1. Fix discriminator weights 2. Sample from generator by injecting noise. 3. Backprop error through discriminator to update generator weights Generator Real world images Discriminator Real Loss Latent random variable Sample Sample Backprop error to update generator weights Figure: Kevin McGuinness (DCU) Generated
Adversarial Training: Generator Generator Real world images Discriminator Real Loss Latent random variable Sample Sample Backprop error to update generator weights Figure: Kevin McGuinness (DCU) Generated Consider a binary encoding of “1” (Real) and “0” (Fake). Which of these two values should the discriminator predict when we train the generator with a generated image ?
Conditional GANs (cGAN) #StyleGAN-T Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, Timo Aila, "StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis". ICML 2023
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ■ Generator & Discriminator Networks ■ Adversarial Training ■ Conditional GANs ○ Variational Autoencoders (VAEs) ○ Diffusion ○ Auto-regressive
Non-Conditional GANs 52 Slide credit: Víctor Garcia Discriminator D(·) Generator G(·) Real World Random seed (z) Real/Generated
53 Conditional GANs (cGAN) Slide credit: Víctor Garcia Conditional Adversarial Networks Real World Real/Generated Condition Discriminator D(·) Generator G(·)
54 Learn more about GANs Ian Goodfellow. NeurIPS Barcelona 2016. Mihaela Rosca & Jeff Donahue. UCL x Deepmind 2020.
Soumith Chintala, “How to train a GAN ? Tips and tricks to make GAN work”. Github 2016. Learn more about GANs Ian Goodfellow @ Lex Friedman (2020)
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferenc ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
Encode Decode “Generate” 58 Auto-Encoder (AE) z Feature space ● Trained with a reconstruction loss. ● Proposed as a pre-training stage for the encoder (“self-supervised learning”). #AE Hinton, Geoffrey E., Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." Science 2006. x’ Learned manifold Loss term: Reconstruction loss
59 Auto-Encoder (AE) for generation ? Could we generate new samples by sampling from a normal distribution and feeding it into the encoder ? And into the decoder (as in GANs) ? Encode Decode “Generate” ? Learned manifold Feature space
60 Auto-Encoder (AE) for generation ? No, because the noise (or encoded noise) would be out of the learned manifold. Encode Decode “Generate” Could we generate new samples by sampling from a normal distribution and feeding it into the encoder, or the decoder (as in GANs) ? Learned manifold Feature space
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferen ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
62 Source: Wikipedia. Image by Bscan - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=25235145 Maths 101: Multivariate normal distribution If all dimensions are independent: ∑ = σ I, where σ ∈ Rk and I ∈ Rk x k is the identity matrix.
63 VAE - Training Encoder (training only): Instead of mapping the input x to a fixed vector, we map it into a multivariate normal distribution z ~ N( μ(X), ∑(X) ). Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. DKL [ N( μ(X), ∑(X) ) || N( μ, σ I ) ] z Encode Loss term #1: KL divergence Sample ~
64 VAE - Training Encode Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Decoder: Reconstruct the input data from z ~ N( μ(X), ∑(X) ). Decode Loss term #2: Reconstruction loss z Sample ~
65 VAE - Training Encode Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Decode Loss term #2: Reconstruction loss z Sample ~ Do you see any challenge in this scheme if use deep neural networks to estimate Q(z|X) and P(X|z) ?
Sample ~ 66 VAE - Training Encode Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Decode Loss term #2: Reconstruction loss z Challenge: We cannot backprop through “Sampling” because it is not a differentiable operation.
67 VAE - Training Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Sampling: The reparametrization trick uses an auxiliary variable ε~N(0,I) to obtain a sample z from the distribution N( μ, σI ). ε~N(0,I) Decode Loss term #2: Reconstruction loss z
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferen ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
VAE Inference 69 How can we generate new samples from a trained VAE ? ε~N(0,I) Decode z
We can sample from our prior ε~N(0,I), discarding the encoder path. 70 Generative behaviour ε~N(0,I) Decode z
71 Generative behaviour #NVAE Vahdat, Arash, and Jan Kautz. "NVAE: A deep hierarchical variational autoencoder." NeurIPS 2020. [code]
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferee ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
73 VQ-VAE #VQ-VAE (Deepmind) Van Den Oord, Aaron, Oriol Vinyals, Koray Kavukcuoglu. "Neural discrete representation learning." NeurIPS 2017. Z space is a discretized and learned embedding space (vector quantization), so every encoded point z(x) is mapped to nearest embedding e, which is the information given to decode the sample.
Learn more about VAEs 74 Andriy Mnih (UCL - Deepmind 2020) Max Welling - University of Amsterdam (2020)
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
#DDPM Jonathan Ho, Ajay Jain, Pieter Abbeel. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. Diffusion-based Data x0 x1 x2 x3 x4 … xT Forward diffusion process (fixed) 😊 Data Reverse diffusion process (must learn) 🤔 Noise Noise
#DDPM Jonathan Ho, Ajay Jain, Pieter Abbeel. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. Diffusion-based Data Manifold P(x0 ) x0 x1 xT … Reverse denoising Forward diffusion Noise In which stage are neural networks needed ?
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
Forward Diffusion Process Philip Isola, Generative Models of Images. MIT 2023.
Forward Diffusion Process Data Noise x0 x1 x2 x3 x4 … xT What is distribution q(xt | xt-1 ) ? It is a normal distribution of: ● mean: (1- βt )1/2 xt-1 ● covariance: βt I …with βt increasing from 0 to 1 with t. Forward diffusion process (fixed)
Forward Diffusion Process Data Noise x0 x1 x2 x3 x4 … xT Forward diffusion process (fixed) What is q(xT |x0 ), the approximate data distribution of xT when T is large (eg. 1000) ?
Forward Diffusion Process Data Noise x0 x1 x2 x3 x4 … xT Forward diffusion process (fixed) What is q(xT |x0 ), the approximate data distribution of xT when T is large (eg. 1000) ? Sohl-Dickstein, Jascha, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. "Deep unsupervised learning using nonequilibrium thermodynamics." ICML 2015.
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
Related work: Denoising Autoencoder (DAE) Encode Decode “Generate” #DAE Vincent, Pascal, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. "Extracting and composing robust features with denoising autoencoders." ICML 2008.
Philip Isola, Generative Models of Images. MIT 2023. Reverse Denoising process
Reverse Denoising process xt : sample at time t timestep t , to account for the schedule Data Noise x0 x1 x2 x3 x4 … xT What is the distribution of xt-1 , given xt ? Normal distribution of: ● mean: μӨ (xt ,t) ● covariance: ΣӨ (xt ,t)
Reverse Denoising process What is the dimension of the latent variable in diffusion models ? Same dimensionality as the diffused data. Data Manifold P(x0 ) x0 xT Noise Image Network learns to denoise step by step NN (eg.UNet, ViT) xt-1 xt
Relation with VAEs Diffusion models can be considered as a special form of VAEs. However in diffusion models: ● The encoder is fixed ● The latent variables have the same dimension as the data ● The decoder is run multiple times in an autoregressive fashion. Encoder qϴ (z|x) Decoder pϴ (x|z) z Forward “encoding” → fixed Reverse “decoding” → trainable VAE Diffusion
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
Latent Diffusion #LDM (Munich, Heidelberg) Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." CVPR 2022. [talk]
Latent Diffusion #LDM (Munich, Heidelberg) Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." CVPR 2022. [talk] VQ-VAE Decoder Sampler 4 4 Denoised latent Denoising UNet Text CLIP Encoder 768 “The girl with pearl earring by Johannes Vermeer.”
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) ■ PixelRNN ■ PixelCNN ■ Transformer-based solutions Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
Joint probability distribution Model the joint probability distribution of data x as a product of element-wise conditional distributions for each element xi conditioned on the previous elements x1 ,...xi-1 . Example: ○ An image x of size (n, n) is understood as a sequence of pixels x1 ,...xn*n ● Apply probability chain rule: xi is the i-th pixel in the image. 95 #PixelRNN Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. ICML 2016.
PixelRNN used a deep neural network (an RNN) to model 96 #PixelRNN Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. ICML 2016. Joint probability distribution Recurrent layer (RNN)
PixelRNN 97 #PixelRNN Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. ICML 2016. Why are not all completions identical ? (aka how can AR offer a generative behaviour ?)
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) ■ PixelRNN ■ PixelCNN ■ Transformer-based solutions Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
99 Auto-Regressive (AR) An auto-regressive approach, where the previous outputs become inputs in future steps, also allows modelling . x[t-L+2], …, x ̂ [t+2] x[t-L+1], …, x ̂ [t+1], x ̂ [t+1] x[t-L], …, x ̂ [t] x ̂ [t+2] x ̂ [t+3]
Wavenet 100 Wavenet used dilated convolutions to produce synthetic audio, sample by sample, conditioned over by receptive field of size T: #Wavenet Oord, Aaron van den, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. "Wavenet: A generative model for raw audio." arXiv 2016. [blog]
101 Training AR with Teacher Forcing #TeacherForcing Williams, Ronald J., and David Zipser. "A learning algorithm for continually running fully recurrent neural networks." Neural computation 1, no. 2 (1989): 270-280. AR (eg. CNN, MLP…) RNN x[t-L+1], …, x[t+1], x ̂ [t+1] x[t-L], …, x ̂ [t] x ̂ [t+2] X Known label
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) ■ PixelRNN ■ PixelCNN ■ Transformer-based solutions Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
The Transformer #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. The Transformer removed the recurrence mechanism thanks to self-attention...
The Transformer #Transformer Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. …which allows parallelizing across multiple machines using teacher forcing. AR
The Transformer Figure: Jay Alammar, “The illustrated Transformer” (2018)
Zero-shot learning #GPT-2 Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, Ilya Sutskever, “Better Language Models and Their Implications”. OpenAI Blog 2019. “GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text.” Zero-shot task performances (GPT-2 was never trained for these tasks)
#iGPT Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., & Sutskever, I. Generative Pretraining from Pixels. ICML 2020. [blog] iGPT - 1.5B params (similar to GPT-2) Training: Next-pixel prediction or masked pixel prediction. Inference: Autoregressive generation of zq ’s with a transformer (global attention). Input resolutions = {322 ,482 ,962 ,1922 } x 3 Model resolutions= {322 ,482 }
#iGPT Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., & Sutskever, I. Generative Pretraining from Pixels. ICML 2020. [blog] iGPT - 1.5B params (similar to GPT-2)
#DALL·E (OpenAI) Ramesh, Aditya, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. "Zero-shot text-to-image generation." ICML 2021. Two-Stage Approaches: DALL·E (12B params) Stage 1: Discrete variational autoencoder (dVAE). Stage 2: Autoregressive transformer
110 #Parti (Google) Yu, Jiahui, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan et al. "Scaling autoregressive models for content-rich text-to-image generation." TMLR 2023. [blog] Scaling up to 20B params Two-Stage Approaches: Parti (20B params)
Learn more about AR models Nal Kalchbrenner, Mediterranean Machine Learning Summer School 2022.
Outline 1. Motivation 2. Discriminative vs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
113 Source: David Foster
Recommended books Interview of David Foster for Machine Learning Street Talk (2023)
Deep Generative Learning for All (a.k.a. The GenAI Hype) Xavier Giro-i-Nieto @DocXavi xavigiro.upc@gmail.com Associate Professor (on leave) Universitat Politècnica de Catalunya Institut de Robòtica Industrial ELLIS Unit Barcelona

More Related Content

PPTX
Foundation Multimodels.pptx
bylauragutierrez90
 
PPTX
Introduction to Generative Models.pptx
byJOBANPREETSINGH62
 
PDF
Deep Generative Learning for All
byUniversitat Politècnica de Catalunya
 
PDF
Tutorial on Deep Generative Models
byMLReview
 
PPTX
Explaining Black-Box Machine Learning Predictions - Sameer Singh, Assistant P...
bySri Ambati
 
PDF
Dcgan
byBrian Kim
 
PDF
Machine Learning: Generative and Discriminative Models
bybutest
 
PDF
Introduction To Generative Adversarial Networks GANs
byHichem Felouat
 
Foundation Multimodels.pptx
bylauragutierrez90
 
Introduction to Generative Models.pptx
byJOBANPREETSINGH62
 
Deep Generative Learning for All
byUniversitat Politècnica de Catalunya
 
Tutorial on Deep Generative Models
byMLReview
 
Explaining Black-Box Machine Learning Predictions - Sameer Singh, Assistant P...
bySri Ambati
 
Dcgan
byBrian Kim
 
Machine Learning: Generative and Discriminative Models
bybutest
 
Introduction To Generative Adversarial Networks GANs
byHichem Felouat
 

What's hot

PDF
Generative adversarial networks
by남주 김
 
PDF
Capsule Networks
byCharles Martin
 
PPTX
Generative AI
byCarlos J. Costa
 
PDF
Introduction to Diffusion Models
bySangwoo Mo
 
PPTX
GPU and Deep learning best practices
byLior Sidi
 
PDF
Generative AI
byAll Things Open
 
PDF
Federated learning
byMindos Cheng
 
PPTX
Big Data Helsinki v 3 | "Federated Learning and Privacy-preserving AI" - Oguz...
byDataconomy Media
 
PDF
Deep Generative Models
byChia-Wen Cheng
 
PDF
An introduction to Machine Learning
bybutest
 
PDF
“Transformer Networks: How They Work and Why They Matter,” a Presentation fro...
byEdge AI and Vision Alliance
 
PPTX
Explainable AI in Healthcare
byvonaurum
 
PPTX
Regulating Generative AI - LLMOps pipelines with Transparency
byDebmalya Biswas
 
PDF
The Origin of Grad-CAM
byShintaro Yoshida
 
PDF
Neural networks and deep learning
byJörgen Sandig
 
PPTX
Machine learning ppt.
byASHOK KUMAR
 
PDF
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
byEdge AI and Vision Alliance
 
PPTX
RAG Patterns and Vector Search in Generative AI
byUdaiappa Ramachandran
 
PPTX
Resnet.pptx
byYanhuaSi
 
PPTX
Introduction to Deep learning
byleopauly
 
Generative adversarial networks
by남주 김
 
Capsule Networks
byCharles Martin
 
Generative AI
byCarlos J. Costa
 
Introduction to Diffusion Models
bySangwoo Mo
 
GPU and Deep learning best practices
byLior Sidi
 
Generative AI
byAll Things Open
 
Federated learning
byMindos Cheng
 
Big Data Helsinki v 3 | "Federated Learning and Privacy-preserving AI" - Oguz...
byDataconomy Media
 
Deep Generative Models
byChia-Wen Cheng
 
An introduction to Machine Learning
bybutest
 
“Transformer Networks: How They Work and Why They Matter,” a Presentation fro...
byEdge AI and Vision Alliance
 
Explainable AI in Healthcare
byvonaurum
 
Regulating Generative AI - LLMOps pipelines with Transparency
byDebmalya Biswas
 
The Origin of Grad-CAM
byShintaro Yoshida
 
Neural networks and deep learning
byJörgen Sandig
 
Machine learning ppt.
byASHOK KUMAR
 
"Semantic Segmentation for Scene Understanding: Algorithms and Implementation...
byEdge AI and Vision Alliance
 
RAG Patterns and Vector Search in Generative AI
byUdaiappa Ramachandran
 
Resnet.pptx
byYanhuaSi
 
Introduction to Deep learning
byleopauly
 

Similar to Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

PDF
Lec 1-2 ssdsdffffsssssfsdfsdfstGenAI.pdf
bytalhaimran7007
 
PDF
Deep Generative Modelling
byPetko Nikolov
 
PDF
Deep Generative Modelling (updated)
byPetko Nikolov
 
PPTX
GENERATIVE MODELLING, Generative and Explanable AI
byReenaChandra6
 
PPTX
Gnerative AI presidency Module1_L1_L2.pptx
byArunnaik63
 
PPTX
GDC2019 - SEED - Towards Deep Generative Models in Game Development
byElectronic Arts / DICE
 
PDF
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
byUniversitat Politècnica de Catalunya
 
PDF
GENAI GENAI GENAI GENAIGENAI GENAI GENAI GENAI
byfahima312
 
PPTX
GAN Deep Learning Approaches to Image Processing Applications (1).pptx
byRMDAcademicCoordinat
 
PPTX
100-Concepts-of-AI with Anupama Kate .pptx
byAnupama Kate
 
PDF
Generative Adversarial Networks
byMustafa Yagmur
 
PPTX
Generative AI Descriptive MOdel ai .pptx
byKanchanaRSVVV
 
PPTX
gan-190318135433 (1).pptx
bykiran814572
 
PPTX
Introduction to Generative AI refers to a subset of artificial intelligence
byKongu Engineering College, Perundurai, Erode
 
PDF
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
byCodemotion
 
PPTX
GAN for business value @ Data Science Milan
byAlex Honchar
 
PPTX
Generative Adversarial Networks (GAN)
byManohar Mukku
 
PDF
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
PDF
M4L19 Generative Models - Slides v 3.pdf
byyireme8491
 
PPTX
GAN Deep. Learning architecture and applications.pptx
byb74104635
 
Lec 1-2 ssdsdffffsssssfsdfsdfstGenAI.pdf
bytalhaimran7007
 
Deep Generative Modelling
byPetko Nikolov
 
Deep Generative Modelling (updated)
byPetko Nikolov
 
GENERATIVE MODELLING, Generative and Explanable AI
byReenaChandra6
 
Gnerative AI presidency Module1_L1_L2.pptx
byArunnaik63
 
GDC2019 - SEED - Towards Deep Generative Models in Game Development
byElectronic Arts / DICE
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
byUniversitat Politècnica de Catalunya
 
GENAI GENAI GENAI GENAIGENAI GENAI GENAI GENAI
byfahima312
 
GAN Deep Learning Approaches to Image Processing Applications (1).pptx
byRMDAcademicCoordinat
 
100-Concepts-of-AI with Anupama Kate .pptx
byAnupama Kate
 
Generative Adversarial Networks
byMustafa Yagmur
 
Generative AI Descriptive MOdel ai .pptx
byKanchanaRSVVV
 
gan-190318135433 (1).pptx
bykiran814572
 
Introduction to Generative AI refers to a subset of artificial intelligence
byKongu Engineering College, Perundurai, Erode
 
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
byCodemotion
 
GAN for business value @ Data Science Milan
byAlex Honchar
 
Generative Adversarial Networks (GAN)
byManohar Mukku
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
M4L19 Generative Models - Slides v 3.pdf
byyireme8491
 
GAN Deep. Learning architecture and applications.pptx
byb74104635
 

More from Universitat Politècnica de Catalunya

PDF
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
byUniversitat Politècnica de Catalunya
 
PDF
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
byUniversitat Politècnica de Catalunya
 
PDF
The Transformer - Xavier Giró - UPC Barcelona 2021
byUniversitat Politècnica de Catalunya
 
PDF
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
byUniversitat Politècnica de Catalunya
 
PDF
Open challenges in sign language translation and production
byUniversitat Politècnica de Catalunya
 
PPTX
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
byUniversitat Politècnica de Catalunya
 
PPTX
Discovery and Learning of Navigation Goals from Pixels in Minecraft
byUniversitat Politècnica de Catalunya
 
PDF
Learn2Sign : Sign language recognition and translation using human keypoint e...
byUniversitat Politècnica de Catalunya
 
PDF
Intepretability / Explainable AI for Deep Neural Networks
byUniversitat Politècnica de Catalunya
 
PDF
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
byUniversitat Politècnica de Catalunya
 
PDF
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
byUniversitat Politècnica de Catalunya
 
PDF
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
byUniversitat Politècnica de Catalunya
 
PDF
Curriculum Learning for Recurrent Video Object Segmentation
byUniversitat Politècnica de Catalunya
 
PDF
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
byUniversitat Politècnica de Catalunya
 
PDF
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
byUniversitat Politècnica de Catalunya
 
PDF
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
byUniversitat Politècnica de Catalunya
 
PDF
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
byUniversitat Politècnica de Catalunya
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
byUniversitat Politècnica de Catalunya
 
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
byUniversitat Politècnica de Catalunya
 
The Transformer - Xavier Giró - UPC Barcelona 2021
byUniversitat Politècnica de Catalunya
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
byUniversitat Politècnica de Catalunya
 
Open challenges in sign language translation and production
byUniversitat Politècnica de Catalunya
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
byUniversitat Politècnica de Catalunya
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
byUniversitat Politècnica de Catalunya
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
byUniversitat Politècnica de Catalunya
 
Intepretability / Explainable AI for Deep Neural Networks
byUniversitat Politècnica de Catalunya
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
byUniversitat Politècnica de Catalunya
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
byUniversitat Politècnica de Catalunya
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
byUniversitat Politècnica de Catalunya
 
Curriculum Learning for Recurrent Video Object Segmentation
byUniversitat Politècnica de Catalunya
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
byUniversitat Politècnica de Catalunya
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
byUniversitat Politècnica de Catalunya
 
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
byUniversitat Politècnica de Catalunya
 
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
byUniversitat Politècnica de Catalunya
 

Recently uploaded

PDF
"Master Excel with CA Suvidha Chaplot: 40 Must-Know Tips for Efficiency"
byCA Suvidha Chaplot
 
PDF
Material–Energy Nexus for EU Descarbonization Extending TIMES-EU to Capture C...
byIEA-ETSAP
 
PPTX
The Archive Fall 2025 Semester Overview by Daisy-Grace Hooper
bydh24372
 
PDF
Powering_AI-Ready_MySQL_MyVector_ProxySQL_v3.pptx.pdf
byAlkin Tezuysal
 
PPTX
Data_Gathering_Presentation.pptx Maed class
byhoneycooo14
 
PDF
Macro Energy Trust Shock - current developments in the Venezuelan energy sector
byAurélie Dupassieux
 
PPTX
Improving tools for working with high resolution geographical and temporal da...
byIEA-ETSAP
 
PPTX
TreatmentPlan of gingival diseases and options
byMohamadSaid29
 
PDF
AI + Data Science: How Artificial Intelligence is Transforming Modern Analytics
byalansanu8
 
PPTX
detector for xrays is for detecting with ai
bydishujeemain
 
PPTX
Energy Communities and Their Roles in Decarbonization
byIEA-ETSAP
 
PDF
dell-ai-factory-with-nvidia-ebook_with.pdf
bytarimocarlos2014
 
PDF
Introduction to R Programming for Data Analysis and Statistics
byNusrat Gulbarga
 
PDF
TOS in History.pdf ENGLISH macro.pdf mnbv
bykristinegorre7
 
PDF
제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계
byBOAZ Bigdata
 
PPTX
Hypothesis Test in Simple Linear Regression.pptx
byDebbieTonog
 
PPTX
Autism types, forms and everything we need to know
byfuzertamas
 
PPTX
715274620-Rws-11-Stem-Hypertext-and-Intertext-Ppt-1.pptx
bymelvinkylemroque24
 
PDF
Solution Sheet / Objectives management (2026)
byOptymyze
 
PDF
Let's build machine learning CNN in RUST + bonus_ ML in water meter with ESP...
byMarcin Bielak
 
"Master Excel with CA Suvidha Chaplot: 40 Must-Know Tips for Efficiency"
byCA Suvidha Chaplot
 
Material–Energy Nexus for EU Descarbonization Extending TIMES-EU to Capture C...
byIEA-ETSAP
 
The Archive Fall 2025 Semester Overview by Daisy-Grace Hooper
bydh24372
 
Powering_AI-Ready_MySQL_MyVector_ProxySQL_v3.pptx.pdf
byAlkin Tezuysal
 
Data_Gathering_Presentation.pptx Maed class
byhoneycooo14
 
Macro Energy Trust Shock - current developments in the Venezuelan energy sector
byAurélie Dupassieux
 
Improving tools for working with high resolution geographical and temporal da...
byIEA-ETSAP
 
TreatmentPlan of gingival diseases and options
byMohamadSaid29
 
AI + Data Science: How Artificial Intelligence is Transforming Modern Analytics
byalansanu8
 
detector for xrays is for detecting with ai
bydishujeemain
 
Energy Communities and Their Roles in Decarbonization
byIEA-ETSAP
 
dell-ai-factory-with-nvidia-ebook_with.pdf
bytarimocarlos2014
 
Introduction to R Programming for Data Analysis and Statistics
byNusrat Gulbarga
 
TOS in History.pdf ENGLISH macro.pdf mnbv
bykristinegorre7
 
제 23회 보아즈(BOAZ) 빅데이터 컨퍼런스 - [MBOAX] : ABSA를 활용한 소비자 반응 분석 기반 운영 효율화 대시보드 설계
byBOAZ Bigdata
 
Hypothesis Test in Simple Linear Regression.pptx
byDebbieTonog
 
Autism types, forms and everything we need to know
byfuzertamas
 
715274620-Rws-11-Stem-Hypertext-and-Intertext-Ppt-1.pptx
bymelvinkylemroque24
 
Solution Sheet / Objectives management (2026)
byOptymyze
 
Let's build machine learning CNN in RUST + bonus_ ML in water meter with ESP...
byMarcin Bielak
 

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

  • 1.
    Deep Generative Learning forAll (a.k.a. The GenAI Hype) Xavier Giro-i-Nieto @DocXavi xavigiro.upc@gmail.com Associate Professor (on leave) Universitat Politècnica de Catalunya Institut de Robòtica Industrial ELLIS Unit Barcelona
  • 2.
    2 Acknowledgements Santiago Pascual @santty128 PhD 2019 UniversitatPolitecnica de Catalunya Technical University of Catalonia Albert Pumarola @AlbertPumarola PhD 2021 Institut de Robòtica Industrial (IRI) Universitat Politècnica de Catalunya (UPC) Kevin McGuinness Research Fellow Insight Centre for Data Analytics Dublin City University Gerard I. Gállego PhD Student Universitat Politècnica de Catalunya gerard.ion.gallego@upc.edu @geiongallego
  • 3.
    3 Acknowledgements Eduard Ramon Applied Scientist AmazonBarcelona @eram1205 Wentong Liao Applied Scientist Amazon Barcelona Ciprian Corneanu Applied Scientist Amazon Seattle Laia Tarrés PhD Student Universita Pompeu Fabra laia.tarres@upf.edu
  • 4.
    Outline 1. Motivation 2. Discriminativevs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Latent variable 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
  • 5.
    Image generation 5 #StyleGAN3 (NVIDIA)Karras, Tero, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Alias-free generative adversarial networks." NeurIPS 2021. [code]
  • 6.
    6 #DiT (UC Berkeley,NYU) Peebles, William, and Saining Xie. "Scalable Diffusion Models with Transformers." ICCV 2023. Image generation
  • 7.
    7 #Imagen2 (Google Deepmind)Blog post Text-to-Image (T2I) generation “A shot of a 32-year old female, up and coming conservationist in a jungle; athletic with short, curly hair and a warm smile
  • 8.
    8 #DALL-E-2 (OpenAI) AdityaRamesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen "Hierarchical Text-Conditional Image Generation with CLIP Latents." 2022. [blog] #DALL·E-3 (OpenAI) James Betker, Gabriel Goh, et al, “Improving Image Generation with Better Captions” 2023 [blog] Text-to-Image (T2I) generation ““An expressive oil painting of a basketball player dunking, depicted as an explosion of a nebula.
  • 9.
    9 Text-to-Music (T2M) generation (StabilityAI) Evans, Zach, Julian D. Parker, C. J. Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. "Long-form music generation with latent diffusion." arXiv 2024.
  • 10.
    10 Text-to-Video (T2V) generation #Sora(OpenAI) Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Wing Yin Ng, Ricky Wang, Aditya Ramesh. Video generation models as world simulators. OpenAI 2024.
  • 11.
    Human Motion Transfer 11 #DreamMovingMengyang Feng, Jinlin Liu, Kai Yu, et al., DreaMoving: A Human Video Generation Framework based on Diffusion Models. arXIv 2023 “A girl, smiling, dancing in a wooden house, wearing sweater, and long pants.” “A girl, smiling, dancing in the park with golden leaves in autumn, wearing light blue dress.” “A girl, smiling, dancing in Times Square, wearing dress-like white shirt, with long sleeves, long pants.”
  • 12.
    12 Image Edition &Text-to-Video (T2V) generation #EMU (Meta) Girdhar, Rohit, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. "Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning." arXiv 2023. [blog]
  • 13.
    13 #Imagine Flash (Meta)Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Vajda, and Ali Thabet. Imagine Flash: Accelerating Emu Diffusion Models with Backward Distillation. Meta 2024. Text-to-Image (T2I) generation Albert Pumarola @AlbertPumarola PhD 2021 Institut de Robòtica Industrial (IRI) Universitat Politècnica de Catalunya (UPC)
  • 14.
    Outline 1. Motivation 2. Discriminativevs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Latent variable 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
  • 15.
    15 Discriminative vs GenerativeModels Philip Isola, Generative Models of Images. MIT 2023.
  • 16.
    Outline 1. Motivation 2. Discriminativevs Generative Models a. Pθ (Y|X): Discriminative Models b. Pθ (X): Generative Models c. Pθ (X|Y): Conditioned Generative Models 3. Latent variable 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
  • 17.
    Pθ (Y|X): Discriminative Models 17 Slidecredit: Albert Pumarola (UPC 2019) Classification Regression Text Prob. of being a Potential Customer Image Audio Speech Translation Jim Carrey What Language? X=Data Y=Labels θ = Model parameters Discriminative Modeling Pθ (Y|X)
  • 18.
    18 0.01 0.09 0.9 input Network (θ) output class Figurecredit: Javier Ruiz (UPC TelecomBCN) Discriminative model: Tell me the probability of some ‘Y’ responses given ‘X’ inputs. Pθ (Y | X = [pixel1 , pixel2 , …, pixel784 ]) Pθ (Y|X): Discriminative Models
  • 19.
    Outline 1. Motivation 2. Discriminativevs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Sampling 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
  • 20.
    20 Slide Concept: AlbertPumarola (UPC 2019) Pθ (X): Generative Models Classification Regression Generative Text Prob. of being a Potential Customer “What about Ron magic?” offered Ron. To Harry, Ron was loud, slow and soft bird. Harry did not like to think about birds. Image Audio Language Translation Music Composer and Interpreter MuseNet Sample Jim Carrey What Language? Discriminative Modeling Pθ (Y|X) Generative Modeling Pθ (X) X=Data Y=Labels θ = Model parameters
  • 21.
    Each real samplexi comes from an M-dimensional probability distribution P(X). X = {x1 , x2 , …, xN } Pθ (X): Generative Models
  • 22.
    22 1) We wantour model with parameters θ to output samples with distribution Pθ (X), matching the distribution of our training data P(X). 2) We can sample points from Pθ (X) plausibly looking how P(X) distributed. P(X) Distribution of training data Pλ,μ,σ (X) Distribution of training data Example: Gaussian Mixture Models (GMM) Pθ (X): Generative Models
  • 23.
    23 What are theparameters θ we need to estimate in deep neural networks ? θ = (weights & biases) output Network (θ) ? Pθ (X): Generative Models
  • 24.
    Outline 1. Motivation 2. Discriminativevs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Sampling 4. Architectures a. GAN b. Auto-regressive c. VAE d. Diffusion
  • 25.
    Pθ (X|Y): Conditioned GenerativeModels Joint probabilities P(X|Y) to model conditioning variables on the generative process: X = {x1 , x2 , …, xN } Y = {y1 , y2 , …, yN } DOG CAT TRUCK PIZZA THRILLER SCI-FI HISTORY /aa/ /e/ /o/
  • 26.
    Outline 1. Motivation 2. Discriminativevs Generative Models a. P(Y|X): Discriminative Models b. P(X): Generative Models c. P(X|Y): Conditioned Generative Models 3. Sampling 4. Architectures a. Generative Adversarial Networks (GANs) b. Auto-regressive c. Variational Autoencoders (VAEs) d. Diffusion
  • 27.
    Our learned modelshould be able to make up new samples from the distribution, not just copy and paste existing samples! 27 Figure from NIPS 2016 Tutorial: Generative Adversarial Networks (I. Goodfellow) Sampling
  • 28.
    Philip Isola, GenerativeModels of Images. MIT 2023. Sampling
  • 29.
    Slide concept: AlbertPumarola (UPC 2019) Learn Sample Out Training Dataset Generated Samples Feature space Manifold Pθ (X) “Model the data distribution so that we can sample new points out of the distribution” Sampling
  • 30.
    Sampling z Generated Samples How couldwe generate diverse samples from a deterministic deep neural network ? Generator (θ)
  • 31.
    Sampling Generated Samples How couldwe generate diverse samples from a deterministic deep neural network ? Generator (θ) Sample z from a known prior, for example, a multivariate normal distribution N(0, I). Example: dim(z)=2 x’ z
  • 32.
    Slide concept: AlbertPumarola (UPC 2019) Learn Training Dataset Interpolated Samples Feature space Manifold Pθ (X) Traversing the learned manifold through interpolation. Interpolation
  • 33.
    Disentanglement Philip Isola, GenerativeModels of Images. MIT 2023.
  • 34.
    Disentanglement Philip Isola, GenerativeModels of Images. MIT 2023.
  • 35.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ■ Generator & Discriminator Networks ■ Adversarial Training ○ Auto-regressive ○ Variational Autoencoders (VAEs) ○ Diffusion
  • 36.
  • 37.
    37 Generator & Discriminator Wehave two modules: Generator (G) and Discriminator (D). ● They “fight” against each other during training→ Adversarial Learning D’s goal: Classify between real samples and those produced by G. G’s goal: Fool D to missclassify. Goodfellow, Ian J., Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative Adversarial Nets." NeurIPS 2014.
  • 38.
    38 Discriminator Discriminator network D→ binary classifier between real (x) and generated (x’). samples. Generated (1) Discriminator (θ) x’ Discriminator (θ) x Real (0)
  • 39.
  • 40.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ■ Generator & Discriminator Networks ■ Adversarial Training ○ Auto-regressive ○ Variational Autoencoders (VAEs) ○ Diffusion
  • 41.
    Imagine we havea counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. 100 100 FAKE: It’s not even green Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
  • 42.
    Imagine we havea counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. 100 100 FAKE: There is no watermark Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
  • 43.
    Imagine we havea counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. 100 100 FAKE: Watermark should be rounded Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
  • 44.
    Imagine we havea counterfeiter (G) trying to make fake money, and the police (D) has to detect whether money is real or fake. After enough iterations, and if the counterfeiter is good enough (in terms of G network it means “has enough parameters”), the police should be confused. REAL? FAKE? Adversarial Training Analogy: is it fake money? Figure: Santiago Pascual (UPC)
  • 45.
    Adversarial Training Generator Real world images Discriminator Real Loss Latent random variable Sample Sample Generated Alternatebetween training the discriminator and generator Neural Network Neural Network Figure: Kevin McGuinness (DCU)
  • 46.
    Adversarial Training: Discriminator Generator Realworld images Discriminator Real Loss Latent random variable Sample Sample Generated 1. Fix generator weights, draw samples from both real world and generated images 2. Train discriminator to distinguish between real world and generated images Backprop error to update discriminator weights Figure: Kevin McGuinness (DCU)
  • 47.
    Adversarial Training: Discriminator Generator Discriminator Real Loss Latent random variable Sample Backproperror to update discriminator weights Figure: Kevin McGuinness (DCU) Generated Consider a binary encoding of “1” (Real) and “0” (Fake). Which of these two values should the discriminator predict when we train the discriminator with a generated image ?
  • 48.
    Adversarial Training: Generator 1.Fix discriminator weights 2. Sample from generator by injecting noise. 3. Backprop error through discriminator to update generator weights Generator Real world images Discriminator Real Loss Latent random variable Sample Sample Backprop error to update generator weights Figure: Kevin McGuinness (DCU) Generated
  • 49.
    Adversarial Training: Generator Generator Realworld images Discriminator Real Loss Latent random variable Sample Sample Backprop error to update generator weights Figure: Kevin McGuinness (DCU) Generated Consider a binary encoding of “1” (Real) and “0” (Fake). Which of these two values should the discriminator predict when we train the generator with a generated image ?
  • 50.
    Conditional GANs (cGAN) #StyleGAN-TAxel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, Timo Aila, "StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis". ICML 2023
  • 51.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ■ Generator & Discriminator Networks ■ Adversarial Training ■ Conditional GANs ○ Variational Autoencoders (VAEs) ○ Diffusion ○ Auto-regressive
  • 52.
    Non-Conditional GANs 52 Slide credit:Víctor Garcia Discriminator D(·) Generator G(·) Real World Random seed (z) Real/Generated
  • 53.
    53 Conditional GANs (cGAN) Slidecredit: Víctor Garcia Conditional Adversarial Networks Real World Real/Generated Condition Discriminator D(·) Generator G(·)
  • 54.
    54 Learn more aboutGANs Ian Goodfellow. NeurIPS Barcelona 2016. Mihaela Rosca & Jeff Donahue. UCL x Deepmind 2020.
  • 55.
    Soumith Chintala, “Howto train a GAN ? Tips and tricks to make GAN work”. Github 2016. Learn more about GANs Ian Goodfellow @ Lex Friedman (2020)
  • 56.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 57.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferenc ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 58.
    Encode Decode “Generate” 58 Auto-Encoder (AE) z Feature space ●Trained with a reconstruction loss. ● Proposed as a pre-training stage for the encoder (“self-supervised learning”). #AE Hinton, Geoffrey E., Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks." Science 2006. x’ Learned manifold Loss term: Reconstruction loss
  • 59.
    59 Auto-Encoder (AE) forgeneration ? Could we generate new samples by sampling from a normal distribution and feeding it into the encoder ? And into the decoder (as in GANs) ? Encode Decode “Generate” ? Learned manifold Feature space
  • 60.
    60 Auto-Encoder (AE) forgeneration ? No, because the noise (or encoded noise) would be out of the learned manifold. Encode Decode “Generate” Could we generate new samples by sampling from a normal distribution and feeding it into the encoder, or the decoder (as in GANs) ? Learned manifold Feature space
  • 61.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferen ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 62.
    62 Source: Wikipedia. Imageby Bscan - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=25235145 Maths 101: Multivariate normal distribution If all dimensions are independent: ∑ = σ I, where σ ∈ Rk and I ∈ Rk x k is the identity matrix.
  • 63.
    63 VAE - Training Encoder(training only): Instead of mapping the input x to a fixed vector, we map it into a multivariate normal distribution z ~ N( μ(X), ∑(X) ). Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. DKL [ N( μ(X), ∑(X) ) || N( μ, σ I ) ] z Encode Loss term #1: KL divergence Sample ~
  • 64.
    64 VAE - Training Encode Kingma,Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Decoder: Reconstruct the input data from z ~ N( μ(X), ∑(X) ). Decode Loss term #2: Reconstruction loss z Sample ~
  • 65.
    65 VAE - Training Encode Kingma,Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Decode Loss term #2: Reconstruction loss z Sample ~ Do you see any challenge in this scheme if use deep neural networks to estimate Q(z|X) and P(X|z) ?
  • 66.
    Sample ~ 66 VAE -Training Encode Kingma, Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Decode Loss term #2: Reconstruction loss z Challenge: We cannot backprop through “Sampling” because it is not a differentiable operation.
  • 67.
    67 VAE - Training Kingma,Diederik P., and Max Welling. "Auto-encoding variational bayes." ICLR 2014. Sampling: The reparametrization trick uses an auxiliary variable ε~N(0,I) to obtain a sample z from the distribution N( μ, σI ). ε~N(0,I) Decode Loss term #2: Reconstruction loss z
  • 68.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferen ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 69.
    VAE Inference 69 How canwe generate new samples from a trained VAE ? ε~N(0,I) Decode z
  • 70.
    We can samplefrom our prior ε~N(0,I), discarding the encoder path. 70 Generative behaviour ε~N(0,I) Decode z
  • 71.
    71 Generative behaviour #NVAE Vahdat,Arash, and Jan Kautz. "NVAE: A deep hierarchical variational autoencoder." NeurIPS 2020. [code]
  • 72.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ■ Limitations of the AE ■ VAE training ■ VAE inferee ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 73.
    73 VQ-VAE #VQ-VAE (Deepmind) VanDen Oord, Aaron, Oriol Vinyals, Koray Kavukcuoglu. "Neural discrete representation learning." NeurIPS 2017. Z space is a discretized and learned embedding space (vector quantization), so every encoded point z(x) is mapped to nearest embedding e, which is the information given to decode the sample.
  • 74.
    Learn more aboutVAEs 74 Andriy Mnih (UCL - Deepmind 2020) Max Welling - University of Amsterdam (2020)
  • 75.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Diffusion ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 76.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
  • 77.
    #DDPM Jonathan Ho,Ajay Jain, Pieter Abbeel. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. Diffusion-based Data x0 x1 x2 x3 x4 … xT Forward diffusion process (fixed) 😊 Data Reverse diffusion process (must learn) 🤔 Noise Noise
  • 78.
    #DDPM Jonathan Ho,Ajay Jain, Pieter Abbeel. "Denoising Diffusion Probabilistic Models." NeurIPS, 2020. Diffusion-based Data Manifold P(x0 ) x0 x1 xT … Reverse denoising Forward diffusion Noise In which stage are neural networks needed ?
  • 79.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
  • 80.
    Forward Diffusion Process PhilipIsola, Generative Models of Images. MIT 2023.
  • 81.
    Forward Diffusion Process Data Noise x0 x1 x2 x3 x4 …xT What is distribution q(xt | xt-1 ) ? It is a normal distribution of: ● mean: (1- βt )1/2 xt-1 ● covariance: βt I …with βt increasing from 0 to 1 with t. Forward diffusion process (fixed)
  • 82.
    Forward Diffusion Process Data Noise x0 x1 x2 x3 x4 …xT Forward diffusion process (fixed) What is q(xT |x0 ), the approximate data distribution of xT when T is large (eg. 1000) ?
  • 83.
    Forward Diffusion Process Data Noise x0 x1 x2 x3 x4 …xT Forward diffusion process (fixed) What is q(xT |x0 ), the approximate data distribution of xT when T is large (eg. 1000) ? Sohl-Dickstein, Jascha, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. "Deep unsupervised learning using nonequilibrium thermodynamics." ICML 2015.
  • 84.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
  • 85.
    Related work: DenoisingAutoencoder (DAE) Encode Decode “Generate” #DAE Vincent, Pascal, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. "Extracting and composing robust features with denoising autoencoders." ICML 2008.
  • 86.
    Philip Isola, GenerativeModels of Images. MIT 2023. Reverse Denoising process
  • 87.
    Reverse Denoising process xt :sample at time t timestep t , to account for the schedule Data Noise x0 x1 x2 x3 x4 … xT What is the distribution of xt-1 , given xt ? Normal distribution of: ● mean: μӨ (xt ,t) ● covariance: ΣӨ (xt ,t)
  • 88.
    Reverse Denoising process Whatis the dimension of the latent variable in diffusion models ? Same dimensionality as the diffused data. Data Manifold P(x0 ) x0 xT Noise Image Network learns to denoise step by step NN (eg.UNet, ViT) xt-1 xt
  • 89.
    Relation with VAEs Diffusionmodels can be considered as a special form of VAEs. However in diffusion models: ● The encoder is fixed ● The latent variables have the same dimension as the data ● The decoder is run multiple times in an autoregressive fashion. Encoder qϴ (z|x) Decoder pϴ (x|z) z Forward “encoding” → fixed Reverse “decoding” → trainable VAE Diffusion
  • 90.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ■ Forward diffusion process ■ Reverse denoising process ■ Latent diffusion models ○ Auto-regressive
  • 91.
    Latent Diffusion #LDM (Munich,Heidelberg) Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." CVPR 2022. [talk]
  • 92.
    Latent Diffusion #LDM (Munich,Heidelberg) Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. "High-resolution image synthesis with latent diffusion models." CVPR 2022. [talk] VQ-VAE Decoder Sampler 4 4 Denoised latent Denoising UNet Text CLIP Encoder 768 “The girl with pearl earring by Johannes Vermeer.”
  • 93.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 94.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) ■ PixelRNN ■ PixelCNN ■ Transformer-based solutions Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 95.
    Joint probability distribution Modelthe joint probability distribution of data x as a product of element-wise conditional distributions for each element xi conditioned on the previous elements x1 ,...xi-1 . Example: ○ An image x of size (n, n) is understood as a sequence of pixels x1 ,...xn*n ● Apply probability chain rule: xi is the i-th pixel in the image. 95 #PixelRNN Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. ICML 2016.
  • 96.
    PixelRNN used adeep neural network (an RNN) to model 96 #PixelRNN Van Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. ICML 2016. Joint probability distribution Recurrent layer (RNN)
  • 97.
    PixelRNN 97 #PixelRNN Van Oord,A., Kalchbrenner, N., & Kavukcuoglu, K. Pixel recurrent neural networks. ICML 2016. Why are not all completions identical ? (aka how can AR offer a generative behaviour ?)
  • 98.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) ■ PixelRNN ■ PixelCNN ■ Transformer-based solutions Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 99.
    99 Auto-Regressive (AR) An auto-regressiveapproach, where the previous outputs become inputs in future steps, also allows modelling . x[t-L+2], …, x ̂ [t+2] x[t-L+1], …, x ̂ [t+1], x ̂ [t+1] x[t-L], …, x ̂ [t] x ̂ [t+2] x ̂ [t+3]
  • 100.
    Wavenet 100 Wavenet used dilatedconvolutions to produce synthetic audio, sample by sample, conditioned over by receptive field of size T: #Wavenet Oord, Aaron van den, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. "Wavenet: A generative model for raw audio." arXiv 2016. [blog]
  • 101.
    101 Training AR withTeacher Forcing #TeacherForcing Williams, Ronald J., and David Zipser. "A learning algorithm for continually running fully recurrent neural networks." Neural computation 1, no. 2 (1989): 270-280. AR (eg. CNN, MLP…) RNN x[t-L+1], …, x[t+1], x ̂ [t+1] x[t-L], …, x ̂ [t] x ̂ [t+2] X Known label
  • 102.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) ■ PixelRNN ■ PixelCNN ■ Transformer-based solutions Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 103.
    The Transformer #Transformer Vaswani,A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. The Transformer removed the recurrence mechanism thanks to self-attention...
  • 104.
    The Transformer #Transformer Vaswani,A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I.. Attention is all you need. NeurIPS 2017. …which allows parallelizing across multiple machines using teacher forcing. AR
  • 105.
    The Transformer Figure: JayAlammar, “The illustrated Transformer” (2018)
  • 106.
    Zero-shot learning #GPT-2 AlecRadford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, Ilya Sutskever, “Better Language Models and Their Implications”. OpenAI Blog 2019. “GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text.” Zero-shot task performances (GPT-2 was never trained for these tasks)
  • 107.
    #iGPT Chen, M.,Radford, A., Child, R., Wu, J., Jun, H., Luan, D., & Sutskever, I. Generative Pretraining from Pixels. ICML 2020. [blog] iGPT - 1.5B params (similar to GPT-2) Training: Next-pixel prediction or masked pixel prediction. Inference: Autoregressive generation of zq ’s with a transformer (global attention). Input resolutions = {322 ,482 ,962 ,1922 } x 3 Model resolutions= {322 ,482 }
  • 108.
    #iGPT Chen, M.,Radford, A., Child, R., Wu, J., Jun, H., Luan, D., & Sutskever, I. Generative Pretraining from Pixels. ICML 2020. [blog] iGPT - 1.5B params (similar to GPT-2)
  • 109.
    #DALL·E (OpenAI) Ramesh,Aditya, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. "Zero-shot text-to-image generation." ICML 2021. Two-Stage Approaches: DALL·E (12B params) Stage 1: Discrete variational autoencoder (dVAE). Stage 2: Autoregressive transformer
  • 110.
    110 #Parti (Google) Yu,Jiahui, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan et al. "Scaling autoregressive models for content-rich text-to-image generation." TMLR 2023. [blog] Scaling up to 20B params Two-Stage Approaches: Parti (20B params)
  • 111.
    Learn more aboutAR models Nal Kalchbrenner, Mediterranean Machine Learning Summer School 2022.
  • 112.
    Outline 1. Motivation 2. Discriminativevs Generative Models 3. Sampling 4. Architectures ○ Generative Adversarial Networks (GANs) ○ Variational Autoencoders (VAEs) ○ Denoising Diffusion Models (DDM) ○ Auto-regressive Models (AR) Figure source: Lilian Weng, What are diffusion models ?, Lil’Log 2021.
  • 113.
  • 114.
    Recommended books Interview ofDavid Foster for Machine Learning Street Talk (2023)
  • 115.
    Deep Generative Learning forAll (a.k.a. The GenAI Hype) Xavier Giro-i-Nieto @DocXavi xavigiro.upc@gmail.com Associate Professor (on leave) Universitat Politècnica de Catalunya Institut de Robòtica Industrial ELLIS Unit Barcelona