auto encoder

1. Auto Encoders
In the name of God
Mehrnaz Faraz
Faculty of Electrical Engineering
K. N. Toosi University of Technology
Milad Abbasi
Faculty of Electrical Engineering
Sharif University of Technology
1

2. Auto Encoders
• An unsupervised deep learning algorithm
• Are artificial neural networks
• Useful for dimensionality reduction and clustering
Unlabeled data
𝑧 = 𝑠 𝑤𝑥 + 𝑏
𝑥
^= 𝑠 𝑤′z + 𝑏′
𝑥
^is 𝑥’s reconstruction
𝑧 is some latent representation or code and 𝑠 is a non-linearity such
as the sigmoid
𝑧 𝑥
^
𝑥
2
Encoder Decoder

3. Auto Encoders
• Simple structure:
𝒙𝟏
𝒙𝟑
𝒙𝟐
𝒙˜𝟏
𝒙˜𝟑
𝒙˜𝟐
Input
Reconstructed
Output
Hidden
Encoder
3
Decoder

4. Undercomplete AE
• Hidden layer is Undercomplete if smaller than the input
layer
– Compresses the input
– Hidden nodes will be Good features for the training
𝑥
^
𝑤′
𝑧
𝑤
𝑥
4

5. Overcomplete AE
• Hidden layer is Overcomplete if greater than the input layer
– No compression in hidden layer.
– Each hidden unit could copy a different input component.
𝑥
^
𝑤′
𝑧
𝑤
𝑥
5

6. Deep Auto Encoders
• Deep Auto Encoders (DAE)
• Stacked Auto Encoders (SAE)
6

7. Training Deep Auto Encoder
• First layer:
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒙^𝟏
𝒙^𝟒
𝒙^𝟑
𝒙^𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
Encoder Decoder
7

8. Training Deep Auto Encoder
• Features of first layer:
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝑎1
𝑎2
𝑎3
8

9. Training Deep Auto Encoder
• Second layer:
𝒂𝟑
𝒂𝟐
𝒂𝟏 𝒂^𝟏
𝒂^𝟑
𝒂^𝟐
𝒃𝟐
𝒃𝟏
9

10. Training Deep Auto Encoder
• Features of second layer:
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝒃𝟐
𝒃𝟏
𝑏1
𝑏2
10

11. Using Deep Auto Encoder
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
• Feature extraction
• Dimensionality reduction
• Classification
𝒙𝟏
𝒂𝟏
𝒃𝟐
𝒃𝟏
Inputs Features
𝒙𝟒
Encoder
11

12. Using Deep Auto Encoder
• Reconstruction
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝒃𝟐
𝒃𝟏
𝒂^𝟏
𝒂^𝟑
𝒂^𝟐
𝒙^𝟒
𝒙^𝟑
𝒙^𝟐
𝒙^𝟏
Encoder Decoder
12

13. Using AE
• Denoising
• Data compression
• Unsupervised learning
• Manifold learning
• Generative model
13

14. Types of Auto Encoder
• Stacked auto encoder (SAE)
• Denoising auto encoder (DAE)
• Sparse Auto Encoder (SAE)
• Contractive Auto Encoder (CAE)
• Convolutional Auto Encoder (CAE)
• Variational Auto Encoder (VAE)
14

15. Generative Models
• Given training data, generate new samples from same
distribution
– Variational Auto Encoder (VAE)
– Generative Adversarial Network (GAN)
15

16. Variational Auto Encoder
Encoder Decoder
Output
𝐱˜
∅
𝒒 𝒛|𝒙 𝒑𝜽 𝒙|𝒛
𝒙𝟏
16
𝒙𝟒
Input
x
𝒙𝟑
𝒙𝟐
𝒙˜𝟏
𝒙˜𝟒
𝒙˜𝟑
𝒙˜𝟐
𝒛𝟏
𝒛𝟐

17. Variational Auto Encoder
• Use probabilistic encoding and decoding
– Encoder:
– Decoder:
• x: Unknown probability distribution
• z: Gaussian probability distribution
𝑞∅ 𝑧|𝑥
𝑝𝜃 𝑥|𝑧
17

18. Training Variational Auto Encoder
• Latent space:
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒉𝟏
𝒉𝟐
𝒉𝟑
𝝈
𝝁
𝒛 𝑞∅ 𝑧|𝑥
18
Mean
Variance
1 dimensional
Gaussian probability distribution
If we have n neurons for 𝝈 and 𝝁 then
we have n dimensional distribution

19. Training Variational Auto Encoder
• Generating new data:
– Example: MNIST Database
𝐱
Encoder
Latent space
Decoder
19

20. Generative Adversarial Network
• VAE:
• GAN:
– Can generate samples
– Trained by competing each other
– Use neural network
– Z is some random noise (Gaussian/Uniform).
– Z can be thought as the latent representation of the
image.
x Decoder 𝐱
˜
z
Encoder
z Generator 𝐱
˜
x
Discriminator
Fake or real?
20
Loss

21. GAN’s Architecture
Discriminator
Generated
fake
samples
Fine tune training
Latent space
Noise
Is D
correct?
Generator
• Overview:
Real samples

22. Using GAN
• Image generation:
22

23. Using GAN
• Data manipulation:
23

24. Denoising Auto Encoder
• Add noise to its input, and train it to recover this original.
24

25. Denoising Auto Encoder
Input
Output
Hidden 3
Hidden 2
Hidden 1
+
Noise
Input
Output
Hidden 3
Hidden 2
Hidden 1
Dropout
25
Randomly switched input
Gaussian noise

26. Sparse Auto Encoder
• Reduce the number of active neurons in the coding layer.
– Add sparsity loss into the cost function.
• Sparsity loss:
– Kullback-Leibler(KL) divergence is commonly used.
26

27. Sparse Auto Encoder
j
j j

ˆ
KL  ̂  log
  1  log
1  
1  ̂
Jsparse w,b J w,b
KL ̂j 
j1
27