SlideShare a Scribd company logo
Auto Encoders
1
In the name of God
Mehrnaz Faraz
Faculty of Electrical Engineering
K. N. Toosi University of Technology
Milad Abbasi
Faculty of Electrical Engineering
Sharif University of Technology
Auto Encoders
2
• An unsupervised deep learning algorithm
• Are artificial neural networks
• Useful for dimensionality reduction and clustering
Unlabeled data
𝑧 = 𝑠 𝑤𝑥 + 𝑏
𝑥 = 𝑠 𝑤′
z + 𝑏′
𝑥 is 𝑥’s reconstruction
𝑧 is some latent representation or code and 𝑠 is a non-linearity such
as the sigmoid
𝑧 𝑥
𝑥 Encoder Decoder
Auto Encoders
• Simple structure:
3
𝒙𝟏
𝒙𝟑
𝒙𝟐
𝒙𝟏
𝒙𝟑
𝒙𝟐
Input
Reconstructed
Output
Hidden
Encoder Decoder
Undercomplete AE
• Hidden layer is Undercomplete if smaller than the input
layer
– Compresses the input
– Hidden nodes will be Good features for the training
4
𝑥
𝑥
𝑤
𝑤′
𝑧
Overcomplete AE
• Hidden layer is Overcomplete if greater than the input layer
– No compression in hidden layer.
– Each hidden unit could copy a different input component.
5
𝑥
𝑥
𝑤
𝑤′
𝑧
Deep Auto Encoders
• Deep Auto Encoders (DAE)
• Stacked Auto Encoders (SAE)
6
Training Deep Auto Encoder
• First layer:
7
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
Encoder Decoder
Training Deep Auto Encoder
• Features of first layer:
8
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝑎1
𝑎2
𝑎3
Training Deep Auto Encoder
• Second layer:
9
𝒂𝟑
𝒂𝟐
𝒂𝟏 𝒂𝟏
𝒂𝟑
𝒂𝟐
𝒃𝟐
𝒃𝟏
Training Deep Auto Encoder
• Features of second layer:
10
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝒃𝟐
𝒃𝟏
𝑏1
𝑏2
Using Deep Auto Encoder
• Feature extraction
• Dimensionality reduction
• Classification
11
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝒃𝟐
𝒃𝟏
Inputs Features
Encoder
Using Deep Auto Encoder
• Reconstruction
12
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒂𝟑
𝒂𝟐
𝒂𝟏
𝒃𝟐
𝒃𝟏
𝒂𝟏
𝒂𝟑
𝒂𝟐
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒙𝟏
Encoder Decoder
Using AE
• Denoising
• Data compression
• Unsupervised learning
• Manifold learning
• Generative model
13
Types of Auto Encoder
• Stacked auto encoder (SAE)
• Denoising auto encoder (DAE)
• Sparse Auto Encoder (SAE)
• Contractive Auto Encoder (CAE)
• Convolutional Auto Encoder (CAE)
• Variational Auto Encoder (VAE)
14
Generative Models
• Given training data, generate new samples from same
distribution
– Variational Auto Encoder (VAE)
– Generative Adversarial Network (GAN)
15
Variational Auto Encoder
16
Encoder Decoder
Input
x Output
𝐱
𝒒∅ 𝒛|𝒙 𝒑𝜽 𝒙|𝒛
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒛𝟏
𝒛𝟐
Variational Auto Encoder
• Use probabilistic encoding and decoding
– Encoder:
– Decoder:
• x: Unknown probability distribution
• z: Gaussian probability distribution
17
𝑞∅ 𝑧|𝑥
𝑝𝜃 𝑥|𝑧
Training Variational Auto Encoder
• Latent space:
18
𝒙𝟏
𝒙𝟒
𝒙𝟑
𝒙𝟐
𝒉𝟏
𝒉𝟐
𝒉𝟑
𝝈
𝝁
𝒛 𝑞∅ 𝑧|𝑥
Mean
Variance
1 dimensional
Gaussian probability distribution
If we have n neurons for 𝝈 and 𝝁 then
we have n dimensional distribution
Training Variational Auto Encoder
• Generating new data:
– Example: MNIST Database
19
𝐱
Encoder
Latent space
Decoder
Generative Adversarial Network
• VAE:
• GAN:
– Can generate samples
– Trained by competing each other
– Use neural network
– Z is some random noise (Gaussian/Uniform).
– Z can be thought as the latent representation of the
image.
20
x Decoder 𝐱
z
Encoder
z Generator 𝐱
x
Discriminator
Fake or real?
Loss
GAN’s Architecture
Real samples
Discriminator
Generated
fake
samples
Fine tune training
Latent space
Noise
Is D
correct?
Generator
• Overview:
Using GAN
• Image generation:
22
Using GAN
• Data manipulation:
23
Denoising Auto Encoder
• Add noise to its input, and train it to recover this original.
24
Denoising Auto Encoder
25
Input
Output
Hidden 3
Hidden 2
Hidden 1
+
Noise
Input
Output
Hidden 3
Hidden 2
Hidden 1
Dropout
Randomly switched input
Gaussian noise
Sparse Auto Encoder
• Reduce the number of active neurons in the coding layer.
– Add sparsity loss into the cost function.
• Sparsity loss:
– Kullback-Leibler(KL) divergence is commonly used.
26
Sparse Auto Encoder
27
   
1
ˆ log 1 log
ˆ ˆ
1
j
j j
KL
 
   
 

  

     
1
ˆ
, ,
sparse j
j
J w b J w b KL
  

  

More Related Content

Similar to autoencoder-190813145130.pdf

Foundations: Artificial Neural Networks
Foundations: Artificial Neural NetworksFoundations: Artificial Neural Networks
Foundations: Artificial Neural Networks
ananth
 
Lec16 - Autoencoders.pptx
Lec16 - Autoencoders.pptxLec16 - Autoencoders.pptx
Lec16 - Autoencoders.pptx
Sameer Gulshan
 
Autoencoder
AutoencoderAutoencoder
Autoencoder
HARISH R
 
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
Codemotion
 
Autoencoder
AutoencoderAutoencoder
Autoencoder
Wataru Hirota
 
Autoecoders.pptx
Autoecoders.pptxAutoecoders.pptx
Autoecoders.pptx
MirzaJahanzeb5
 
Variational Autoencoders For Image Generation
Variational Autoencoders For Image GenerationVariational Autoencoders For Image Generation
Variational Autoencoders For Image Generation
Jason Anderson
 
201907 AutoML and Neural Architecture Search
201907 AutoML and Neural Architecture Search201907 AutoML and Neural Architecture Search
201907 AutoML and Neural Architecture Search
DaeJin Kim
 
Deep learning from a novice perspective
Deep learning from a novice perspectiveDeep learning from a novice perspective
Deep learning from a novice perspective
Anirban Santara
 
Volodymyr Lyubinets “Generative models for images”
Volodymyr Lyubinets  “Generative models for images”Volodymyr Lyubinets  “Generative models for images”
Volodymyr Lyubinets “Generative models for images”
Lviv Startup Club
 
Hardware Acceleration for Machine Learning
Hardware Acceleration for Machine LearningHardware Acceleration for Machine Learning
Hardware Acceleration for Machine Learning
CastLabKAIST
 
Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14
Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14
Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14
Daniel Lewis
 
Smedberg niklas bringing_aaa_graphics
Smedberg niklas bringing_aaa_graphicsSmedberg niklas bringing_aaa_graphics
Smedberg niklas bringing_aaa_graphics
changehee lee
 
Denoising autoencoder by Harish.R
Denoising autoencoder by Harish.RDenoising autoencoder by Harish.R
Denoising autoencoder by Harish.R
HARISH R
 
Introduction to Neural Network
Introduction to Neural NetworkIntroduction to Neural Network
Introduction to Neural Network
Omer Korech
 
Structure learning with Deep Autoencoders
Structure learning with Deep AutoencodersStructure learning with Deep Autoencoders
Structure learning with Deep Autoencoders
Patrick Michl
 
Gpu Join Presentation
Gpu Join PresentationGpu Join Presentation
Gpu Join Presentation
Suman Karumuri
 
The Next Mainstream Programming Language: A Game Developer's Perspective
The Next Mainstream Programming Language: A Game Developer's PerspectiveThe Next Mainstream Programming Language: A Game Developer's Perspective
The Next Mainstream Programming Language: A Game Developer's Perspective
kfrdbs
 
OCR for Urdu translation
OCR for Urdu translation OCR for Urdu translation
OCR for Urdu translation
Yasar Hayat
 
Autoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesAutoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time Series
Yiqun Hu
 

Similar to autoencoder-190813145130.pdf (20)

Foundations: Artificial Neural Networks
Foundations: Artificial Neural NetworksFoundations: Artificial Neural Networks
Foundations: Artificial Neural Networks
 
Lec16 - Autoencoders.pptx
Lec16 - Autoencoders.pptxLec16 - Autoencoders.pptx
Lec16 - Autoencoders.pptx
 
Autoencoder
AutoencoderAutoencoder
Autoencoder
 
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
Alberto Massidda - Scenes from a memory - Codemotion Rome 2019
 
Autoencoder
AutoencoderAutoencoder
Autoencoder
 
Autoecoders.pptx
Autoecoders.pptxAutoecoders.pptx
Autoecoders.pptx
 
Variational Autoencoders For Image Generation
Variational Autoencoders For Image GenerationVariational Autoencoders For Image Generation
Variational Autoencoders For Image Generation
 
201907 AutoML and Neural Architecture Search
201907 AutoML and Neural Architecture Search201907 AutoML and Neural Architecture Search
201907 AutoML and Neural Architecture Search
 
Deep learning from a novice perspective
Deep learning from a novice perspectiveDeep learning from a novice perspective
Deep learning from a novice perspective
 
Volodymyr Lyubinets “Generative models for images”
Volodymyr Lyubinets  “Generative models for images”Volodymyr Lyubinets  “Generative models for images”
Volodymyr Lyubinets “Generative models for images”
 
Hardware Acceleration for Machine Learning
Hardware Acceleration for Machine LearningHardware Acceleration for Machine Learning
Hardware Acceleration for Machine Learning
 
Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14
Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14
Piotr Mirowski - Review Autoencoders (Deep Learning) - CIUUK14
 
Smedberg niklas bringing_aaa_graphics
Smedberg niklas bringing_aaa_graphicsSmedberg niklas bringing_aaa_graphics
Smedberg niklas bringing_aaa_graphics
 
Denoising autoencoder by Harish.R
Denoising autoencoder by Harish.RDenoising autoencoder by Harish.R
Denoising autoencoder by Harish.R
 
Introduction to Neural Network
Introduction to Neural NetworkIntroduction to Neural Network
Introduction to Neural Network
 
Structure learning with Deep Autoencoders
Structure learning with Deep AutoencodersStructure learning with Deep Autoencoders
Structure learning with Deep Autoencoders
 
Gpu Join Presentation
Gpu Join PresentationGpu Join Presentation
Gpu Join Presentation
 
The Next Mainstream Programming Language: A Game Developer's Perspective
The Next Mainstream Programming Language: A Game Developer's PerspectiveThe Next Mainstream Programming Language: A Game Developer's Perspective
The Next Mainstream Programming Language: A Game Developer's Perspective
 
OCR for Urdu translation
OCR for Urdu translation OCR for Urdu translation
OCR for Urdu translation
 
Autoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time SeriesAutoencoder Forest for Anomaly Detection from IoT Time Series
Autoencoder Forest for Anomaly Detection from IoT Time Series
 

Recently uploaded

一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
slg6lamcq
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Sm321
 
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
bopyb
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
nuttdpt
 
一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理
一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理
一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理
mzpolocfi
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
rwarrenll
 
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
ahzuo
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
v7oacc3l
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
nuttdpt
 
Natural Language Processing (NLP), RAG and its applications .pptx
Natural Language Processing (NLP), RAG and its applications .pptxNatural Language Processing (NLP), RAG and its applications .pptx
Natural Language Processing (NLP), RAG and its applications .pptx
fkyes25
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Walaa Eldin Moustafa
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
javier ramirez
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
sameer shah
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
u86oixdj
 
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
u86oixdj
 

Recently uploaded (20)

一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
一比一原版(Adelaide毕业证书)阿德莱德大学毕业证如何办理
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
 
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
 
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
一比一原版(UCSB文凭证书)圣芭芭拉分校毕业证如何办理
 
一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理
一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理
一比一原版(Dalhousie毕业证书)达尔豪斯大学毕业证如何办理
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
 
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
一比一原版(UIUC毕业证)伊利诺伊大学|厄巴纳-香槟分校毕业证如何办理
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
 
Natural Language Processing (NLP), RAG and its applications .pptx
Natural Language Processing (NLP), RAG and its applications .pptxNatural Language Processing (NLP), RAG and its applications .pptx
Natural Language Processing (NLP), RAG and its applications .pptx
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
原版制作(Deakin毕业证书)迪肯大学毕业证学位证一模一样
 
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
原版制作(swinburne毕业证书)斯威本科技大学毕业证毕业完成信一模一样
 

autoencoder-190813145130.pdf

  • 1. Auto Encoders 1 In the name of God Mehrnaz Faraz Faculty of Electrical Engineering K. N. Toosi University of Technology Milad Abbasi Faculty of Electrical Engineering Sharif University of Technology
  • 2. Auto Encoders 2 • An unsupervised deep learning algorithm • Are artificial neural networks • Useful for dimensionality reduction and clustering Unlabeled data 𝑧 = 𝑠 𝑤𝑥 + 𝑏 𝑥 = 𝑠 𝑤′ z + 𝑏′ 𝑥 is 𝑥’s reconstruction 𝑧 is some latent representation or code and 𝑠 is a non-linearity such as the sigmoid 𝑧 𝑥 𝑥 Encoder Decoder
  • 3. Auto Encoders • Simple structure: 3 𝒙𝟏 𝒙𝟑 𝒙𝟐 𝒙𝟏 𝒙𝟑 𝒙𝟐 Input Reconstructed Output Hidden Encoder Decoder
  • 4. Undercomplete AE • Hidden layer is Undercomplete if smaller than the input layer – Compresses the input – Hidden nodes will be Good features for the training 4 𝑥 𝑥 𝑤 𝑤′ 𝑧
  • 5. Overcomplete AE • Hidden layer is Overcomplete if greater than the input layer – No compression in hidden layer. – Each hidden unit could copy a different input component. 5 𝑥 𝑥 𝑤 𝑤′ 𝑧
  • 6. Deep Auto Encoders • Deep Auto Encoders (DAE) • Stacked Auto Encoders (SAE) 6
  • 7. Training Deep Auto Encoder • First layer: 7 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 Encoder Decoder
  • 8. Training Deep Auto Encoder • Features of first layer: 8 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝑎1 𝑎2 𝑎3
  • 9. Training Deep Auto Encoder • Second layer: 9 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒂𝟏 𝒂𝟑 𝒂𝟐 𝒃𝟐 𝒃𝟏
  • 10. Training Deep Auto Encoder • Features of second layer: 10 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒃𝟐 𝒃𝟏 𝑏1 𝑏2
  • 11. Using Deep Auto Encoder • Feature extraction • Dimensionality reduction • Classification 11 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒃𝟐 𝒃𝟏 Inputs Features Encoder
  • 12. Using Deep Auto Encoder • Reconstruction 12 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒂𝟑 𝒂𝟐 𝒂𝟏 𝒃𝟐 𝒃𝟏 𝒂𝟏 𝒂𝟑 𝒂𝟐 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒙𝟏 Encoder Decoder
  • 13. Using AE • Denoising • Data compression • Unsupervised learning • Manifold learning • Generative model 13
  • 14. Types of Auto Encoder • Stacked auto encoder (SAE) • Denoising auto encoder (DAE) • Sparse Auto Encoder (SAE) • Contractive Auto Encoder (CAE) • Convolutional Auto Encoder (CAE) • Variational Auto Encoder (VAE) 14
  • 15. Generative Models • Given training data, generate new samples from same distribution – Variational Auto Encoder (VAE) – Generative Adversarial Network (GAN) 15
  • 16. Variational Auto Encoder 16 Encoder Decoder Input x Output 𝐱 𝒒∅ 𝒛|𝒙 𝒑𝜽 𝒙|𝒛 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒛𝟏 𝒛𝟐
  • 17. Variational Auto Encoder • Use probabilistic encoding and decoding – Encoder: – Decoder: • x: Unknown probability distribution • z: Gaussian probability distribution 17 𝑞∅ 𝑧|𝑥 𝑝𝜃 𝑥|𝑧
  • 18. Training Variational Auto Encoder • Latent space: 18 𝒙𝟏 𝒙𝟒 𝒙𝟑 𝒙𝟐 𝒉𝟏 𝒉𝟐 𝒉𝟑 𝝈 𝝁 𝒛 𝑞∅ 𝑧|𝑥 Mean Variance 1 dimensional Gaussian probability distribution If we have n neurons for 𝝈 and 𝝁 then we have n dimensional distribution
  • 19. Training Variational Auto Encoder • Generating new data: – Example: MNIST Database 19 𝐱 Encoder Latent space Decoder
  • 20. Generative Adversarial Network • VAE: • GAN: – Can generate samples – Trained by competing each other – Use neural network – Z is some random noise (Gaussian/Uniform). – Z can be thought as the latent representation of the image. 20 x Decoder 𝐱 z Encoder z Generator 𝐱 x Discriminator Fake or real? Loss
  • 21. GAN’s Architecture Real samples Discriminator Generated fake samples Fine tune training Latent space Noise Is D correct? Generator • Overview:
  • 22. Using GAN • Image generation: 22
  • 23. Using GAN • Data manipulation: 23
  • 24. Denoising Auto Encoder • Add noise to its input, and train it to recover this original. 24
  • 25. Denoising Auto Encoder 25 Input Output Hidden 3 Hidden 2 Hidden 1 + Noise Input Output Hidden 3 Hidden 2 Hidden 1 Dropout Randomly switched input Gaussian noise
  • 26. Sparse Auto Encoder • Reduce the number of active neurons in the coding layer. – Add sparsity loss into the cost function. • Sparsity loss: – Kullback-Leibler(KL) divergence is commonly used. 26
  • 27. Sparse Auto Encoder 27     1 ˆ log 1 log ˆ ˆ 1 j j j KL                    1 ˆ , , sparse j j J w b J w b KL       