SlideShare a Scribd company logo
1 of 75
Download to read offline
Structure learning
with deep neuronal networks
7th
Network Modeling Workshop, 28/2/2014
Patrick Michl
Page 228.02.2014
Patrick Michl
Network Modeling
Agenda
AutoencodersAutoencoders
Biological ModelBiological Model
Validation & ImplementationValidation & Implementation
Page 328.02.2014
Patrick Michl
Network Modeling
Real world data usually is high dimensional …
x1
x2
Dataset Model
Autoencoders
Page 428.02.2014
Patrick Michl
Network Modeling
… which makes structural analysis and modeling complicated!
x1
x2
x1
x2
Dataset Model
? 
Autoencoders
Page 528.02.2014
Patrick Michl
Network Modeling
Dimensionality reduction techinques like PCA …
x1
x2
PCA
Dataset Model
Autoencoders
Page 628.02.2014
Patrick Michl
Network Modeling
… can not preserve complex structures!
x1
x2
PCA
Dataset Model
x1
x2
𝑥2=α 𝑥1+β 
Autoencoders
Page 728.02.2014
Patrick Michl
Network Modeling
Therefore the analysis of unknown structures …
x1
x2
Dataset Model
Autoencoders
Page 828.02.2014
Patrick Michl
Network Modeling
… needs more considerate nonlinear techniques!
x1
x2
Dataset Model
x1
x2
𝑥2=𝑓(𝑥1) 
Autoencoders
Page 928.02.2014
Patrick Michl
Network Modeling
Autoencoders are artificial neuronal networks …
Autoencoder
• Artificial Neuronal Network
Autoencoders
input data X
output data X‘
Perceptrons
Gaussian Units
Page 1028.02.2014
Patrick Michl
Network Modeling
Autoencoders are artificial neuronal networks …
Autoencoder
• Artificial Neuronal Network
Autoencoders
input data X
output data X‘
Perceptrons
Gaussian Units
Perceptron
1
0
Gauss Units
R
Page 1128.02.2014
Patrick Michl
Network Modeling
Autoencoders are artificial neuronal networks …
Autoencoder
• Artificial Neuronal Network
Autoencoders
input data X
output data X‘
Perceptrons
Gaussian Units
Page 1228.02.2014
Patrick Michl
Network Modeling
Autoencoder
• Artificial Neuronal Network
• Multiple hidden layers
Autoencoders
… with multiple hidden layers.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
Page 1328.02.2014
Patrick Michl
Network Modeling
Autoencoder
• Artificial Neuronal Network
• Multiple hidden layers
Autoencoders
Such networks are called deep networks.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
Page 1428.02.2014
Patrick Michl
Network Modeling
Autoencoder
• Artificial Neuronal Network
• Multiple hidden layers
Autoencoders
Such networks are called deep networks.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)Definition (deep network)
Deep networks are artificial neuronal
networks with multiple hidden layers
Page 1528.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
Such networks are called deep networks.
• Deep network
Page 1628.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
Autoencoders have a symmetric topology …
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
• Deep network
• Symmetric topology
Page 1728.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
… with an odd number of hidden layers.
Gaussian Units
input data X
output data X‘
Perceptrons
(Visible layers)
(Hidden layers)
• Deep network
• Symmetric topology
Page 1828.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
The small layer in the center works lika an information bottleneck
input data X
output data X‘
• Deep network
• Symmetric topology
• Information bottleneck
Bottleneck
Page 1928.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
... that creates a low dimensional code for each sample in the input data.
input data X
output data X‘
• Deep network
• Symmetric topology
• Information bottleneck
Bottleneck
Page 2028.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
The upper stack does the encoding …
input data X
output data X‘
• Deep network
• Symmetric topology
• Information bottleneck
• Encoder
Encoder
Page 2128.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
… and the lower stack does the decoding.
input data X
output data X‘
• Deep network
• Symmetric topology
• Information bottleneck
• Encoder
• Decoder
Encoder
Decoder
Page 2228.02.2014
Patrick Michl
Network Modeling
• Deep network
• Symmetric topology
• Information bottleneck
• Encoder
• Decoder
Autoencoder
Autoencoders
… and the lower stack does the decoding.
input data X
output data X‘
Encoder
Decoder
Definition (deep network)
Deep networks are artificial neuronal
networks with multiple hidden layers
Definition (autoencoder)
Autoencoders are deep networks with a symmetric
topology and an odd number of hiddern layers,
containing a encoder, a low dimensional
representation and a decoder.
Page 2328.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
Autoencoders can be used to reduce the dimension of data …
input data X
output data X‘
Problem: dimensionality of data
Idea:
1. Train autoencoder to minimize the distance
between input X and output X‘
2. Encode X to low dimensional code Y
3. Decode low dimensional code Y to output X‘
4. Output X‘ is low dimensional
Page 2428.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
… if we can train them!
input data X
output data X‘
Problem: dimensionality of data
Idea:
1. Train autoencoder to minimize the distance
between input X and output X‘
2. Encode X to low dimensional code Y
3. Decode low dimensional code Y to output X‘
4. Output X‘ is low dimensional
Page 2528.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
In feedforward ANNs backpropagation is a good approach.
input data X
output data X‘
Training
Backpropagation
Page 2628.02.2014
Patrick Michl
Network Modeling
Backpropagation
Autoencoder
Autoencoders
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is
computed. This gives a error function
error
 
In feedforward ANNs backpropagation is a good approach.
Page 2728.02.2014
Patrick Michl
Network Modeling
Backpropagation
Autoencoder
Autoencoders
In feedforward ANNs backpropagation is the choice
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is
computed. This gives a error function
Example (linear neuronal unit with two inputs)
Page 2828.02.2014
Patrick Michl
Network Modeling
Backpropagation
Autoencoder
Autoencoders
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is
computed. This gives a error function
(2) By calculating we get a vector that shows in a direction which decreases
the error
(3) We update the parameters to decrease the error
 
In feedforward ANNs backpropagation is a good approach.
Page 2928.02.2014
Patrick Michl
Network Modeling
Backpropagation
Autoencoder
Autoencoders
In feedforward ANNs backpropagation is the choice
input data X
output data X‘
Training
Definition (autoencoder)
Backpropagation
(1) The distance (error) between current output X‘ and wanted output Y is
computed. This gives a error function
(2) By calculating we get a vector that shows in a direction which decreases
the error
(3) We update the parameters to decrease the error
(4) We repeat that
 
Page 3028.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
… the problem are the multiple hidden layers!
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
Page 3128.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation is known to be slow far away from the output layer …
Backpropagation
Problem: Deep Network
• Very slow training
Page 3228.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
… and can converge to poor local minima.
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Page 3328.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
The task is to initialize the parameters close to a good solution!
Page 3428.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
• Pretraining
Therefore the training of autoencoders has a pretraining phase …
Page 3528.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
• Pretraining
• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Page 3628.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
• Pretraining
• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
• RBMs are Markov Random Fields
Page 3728.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
• Pretraining
• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
• RBMs are Markov Random Fields
Markov Random Field
Every unit influences every neighbor
The coupling is undirected
Motivation (Ising Model)
A set of magnetic dipoles (spins)
is arranged in a graph (lattice)
where neighbors are
coupled with a given strengt
Page 3828.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
• Pretraining
• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
• RBMs are Markov Random Fields
• Bipartite topology: visible (v), hidden (h)
• Use local energy to calculate the probabilities of values
Training:
contrastive divergency
(Gibbs Sampling)
h1
v1 v2
v3 v4
h2 h3
Page 3928.02.2014
Patrick Michl
Network Modeling
Autoencoder
Autoencoders
input data X
output data X‘
Training
Backpropagation
Problem: Deep Network
• Very slow training
• Maybe bad solution
Idea: Initialize close to a good solution
• Pretraining
• Restricted Boltzmann Machines
… which uses Restricted Boltzmann Machines (RBMs)
Restricted Boltzmann Machine
Gibbs Sampling
Page 4028.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
The top layer RBM transforms real value data into binary codes.
 
Top
Training
Page 4128.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Top
Therefore visible units are modeled with gaussians to encode data …
 
h2
v1 v2
v3 v4
h3 h4 h5h1
Training
Page 4228.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Top
… and many hidden units with simoids to encode dependencies
 
h2
v1 v2
v3 v4
h3 h4 h5h1
Training
Page 4328.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Top
The objective function is the sum of the local energies.
Local Energy
 
𝐸 𝑣 ≔ −∑
h
𝑤 𝑣h
𝑥 𝑣
𝜎 𝑣
𝑥
h
+
( 𝑥 𝑣 − 𝑏 𝑣 )2
2 𝜎 𝑣
2
 
h2
v1 v2
v3 v4
h3 h4 h5h1
Training
Page 4428.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Reduction
 
The next RBM layer maps the dependency encoding…
Training
Page 4528.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Reduction
… from the upper layer …
v
 
h1
v1 v2
v3 v4
h2 h3
Training
Page 4628.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Reduction
… to a smaller number of simoids …
h
 
h1
v1 v2
v3 v4
h2 h3
Training
Page 4728.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Reduction
… which can be trained faster than the top layer
Local Energy
𝐸𝑣 ≔−∑
h
𝑤 𝑣h 𝑥 𝑣 𝑥h+𝑥h 𝑏h
 
𝐸h ≔−∑
𝑣
𝑤 𝑣h 𝑥 𝑣 𝑥h+𝑥 𝑣 𝑏 𝑣
 
h1
v1 v2
v3 v4
h2 h3
Training
Page 4828.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Unrolling
The symmetric topology allows us to skip further training.
Training
Page 4928.02.2014
Patrick Michl
Network Modeling
Autoencoders
Autoencoder
Unrolling
The symmetric topology allows us to skip further training.
Training
Page 5028.02.2014
Patrick Michl
Network Modeling
After pretraining backpropagation usually finds good solutions
Autoencoders
Autoencoder
Training
• Pretraining
Top RBM (GRBM)
Reduction RBMs
Unrolling
• Finetuning
Backpropagation
Page 5128.02.2014
Patrick Michl
Network Modeling
The algorithmic complexity of RBM training depends on the network size
Autoencoders
Autoencoder
Training
• Complexity: O(inw)
i: number of iterations
n: number of nodes
w: number of weights
• Memory Complexity: O(w)
Page 5228.02.2014
Patrick Michl
Network Modeling
Agenda
Autoencoders
Biological Model
Validation & Implementation
Page 5328.02.2014
Patrick Michl
Network Modeling Network Modeling
Restricted Boltzmann Machines (RBM)
How to model the topological structure?
S
E
TF
Page 5428.02.2014
Patrick Michl
Network Modeling
We define S and E as visible data Layer …
S
E
TF
Network Modeling
Restricted Boltzmann Machines (RBM)
Page 5528.02.2014
Patrick Michl
Network Modeling
S E
TF
Network Modeling
Restricted Boltzmann Machines (RBM)
We identify S and E with the visible layer …
Page 5628.02.2014
Patrick Michl
Network Modeling
S E
… and the TFs with the hidden layer in a RBM
TF
Network Modeling
Restricted Boltzmann Machines (RBM)
Page 5728.02.2014
Patrick Michl
Network Modeling
S E
The training of the RBM gives us a model
TF
Network Modeling
Restricted Boltzmann Machines (RBM)
Page 5828.02.2014
Patrick Michl
Network Modeling
Agenda
Autoencoder
Biological Model
Implementation & Results
Page 5928.02.2014
Patrick Michl
Network Modeling
Results
Validation of the results
• Needs information about the true regulation
• Needs information about the descriptive power of the data
Page 6028.02.2014
Patrick Michl
Network Modeling
Results
Validation of the results
• Needs information about the true regulation
• Needs information about the descriptive power of the data
Without this infomation validation can only be done,
using artificial datasets!
Page 6128.02.2014
Patrick Michl
Network Modeling
Results
Artificial datasets
We simulate data in three steps:
Page 6228.02.2014
Patrick Michl
Network Modeling
Results
Artificial datasets
We simulate data in three steps
Step 1
Choose number of Genes (E+S) and create random bimodal distributed
data
Page 6328.02.2014
Patrick Michl
Network Modeling
Results
Artificial datasets
We simulate data in three steps
Step 1
Choose number of Genes (E+S) and create random bimodal distributed
data
Step 2
Manipulate data in a fixed order
Page 6428.02.2014
Patrick Michl
Network Modeling
Results
Artificial datasets
We simulate data in three steps
Step 1
Choose number of Genes (E+S) and create random bimodal distributed
data
Step 2
Manipulate data in a fixed order
Step 3
Add noise to manipulated data
and normalize data
Page 6528.02.2014
Patrick Michl
Network Modeling
Simulation
Results
Step 1
Number of visible nodes 8 (4E, 4S)
Create random data:
Random {-1, +1} + N(0,
 
Page 6628.02.2014
Patrick Michl
Network Modeling
Simulation
Results
Noise
 
Step 2
Manipulate data
Page 6728.02.2014
Patrick Michl
Network Modeling
Simulation
Results
Step 3
Add noise: N(0,
 
Page 6828.02.2014
Patrick Michl
Network Modeling
Results
We analyse the data X
with an RBM
Page 6928.02.2014
Patrick Michl
Network Modeling
Results
We train an autoencoder with 9 hidden layers
and 165 nodes:
Layer 1 & 9: 32 hidden units
Layer 2 & 8: 24 hidden units
Layer 3 & 7: 16 hidden units
Layer 4 & 6: 8 hidden units
Layer 5: 5 hidden units
input data X
output data X‘
Page 7028.02.2014
Patrick Michl
Network Modeling
Results
We transform the data from X to X‘
And reduce the dimensionality
Page 7128.02.2014
Patrick Michl
Network Modeling
Results
We analyse the
transformed data X‘
with an RBM
Page 7228.02.2014
Patrick Michl
Network Modeling
Results
Lets compare the models
Page 7328.02.2014
Patrick Michl
Network Modeling
Results
Another Example with more nodes and larger autoencoder
Page 7428.02.2014
Patrick Michl
Network Modeling
Conclusion
Conclusion
• Autoencoders can improve modeling significantly by reducing the
dimensionality of data
• Autoencoders preserve complex structures in their multilayer
perceptron network. Analysing those networks (for example with
knockout tests) could give more structural information
• The drawback are high computational costs
Since the field of deep learning is getting more popular (Face
recognition / Voice recognition, Image transformation). Many new
improvements in facing the computational costs have been made.
Page 7528.02.2014
Patrick Michl
Network Modeling
Acknowledgement
eilsLABS
Prof. Dr. Rainer König
Prof. Dr. Roland Eils
Network Modeling Group

More Related Content

Similar to Structure learning with Deep Neural Networks

[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf
[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf
[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf
DataScienceConferenc1
 
Deep Learning Made Easy with Deep Features
Deep Learning Made Easy with Deep FeaturesDeep Learning Made Easy with Deep Features
Deep Learning Made Easy with Deep Features
Turi, Inc.
 
“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...
“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...
“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...
Edge AI and Vision Alliance
 
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr TeterwakLearn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
PyData
 
Apache Spark for Cyber Security in an Enterprise Company
Apache Spark for Cyber Security in an Enterprise CompanyApache Spark for Cyber Security in an Enterprise Company
Apache Spark for Cyber Security in an Enterprise Company
Databricks
 

Similar to Structure learning with Deep Neural Networks (20)

ENCRYPTION KEY GENERATION FOR DIGITAL CIRCUITS USING ANALOG CIRCUITS
ENCRYPTION KEY GENERATION FOR DIGITAL CIRCUITS USING ANALOG CIRCUITSENCRYPTION KEY GENERATION FOR DIGITAL CIRCUITS USING ANALOG CIRCUITS
ENCRYPTION KEY GENERATION FOR DIGITAL CIRCUITS USING ANALOG CIRCUITS
 
Android Malware 2020 (CCCS-CIC-AndMal-2020)
Android Malware 2020 (CCCS-CIC-AndMal-2020)Android Malware 2020 (CCCS-CIC-AndMal-2020)
Android Malware 2020 (CCCS-CIC-AndMal-2020)
 
Accelerating algorithmic and hardware advancements for power efficient on-dev...
Accelerating algorithmic and hardware advancements for power efficient on-dev...Accelerating algorithmic and hardware advancements for power efficient on-dev...
Accelerating algorithmic and hardware advancements for power efficient on-dev...
 
IRJET- Generating 3D Models Using 3D Generative Adversarial Network
IRJET- Generating 3D Models Using 3D Generative Adversarial NetworkIRJET- Generating 3D Models Using 3D Generative Adversarial Network
IRJET- Generating 3D Models Using 3D Generative Adversarial Network
 
[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf
[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf
[DSC Europe 23] Aleksandar Veljkovic - ZKML.pdf
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are Dangerous
 
Deep Learning Made Easy with Deep Features
Deep Learning Made Easy with Deep FeaturesDeep Learning Made Easy with Deep Features
Deep Learning Made Easy with Deep Features
 
The von Neumann Memory Barrier and Computer Architectures for the 21st Century
The von Neumann Memory Barrier and Computer Architectures for the 21st CenturyThe von Neumann Memory Barrier and Computer Architectures for the 21st Century
The von Neumann Memory Barrier and Computer Architectures for the 21st Century
 
Pushing the boundaries of AI research
Pushing the boundaries of AI researchPushing the boundaries of AI research
Pushing the boundaries of AI research
 
“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...
“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...
“Enabling Ultra-low Power Edge Inference and On-device Learning with Akida,” ...
 
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr TeterwakLearn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
 
C10ComputerEngg.pptx
C10ComputerEngg.pptxC10ComputerEngg.pptx
C10ComputerEngg.pptx
 
Data Analysis with TensorFlow in PostgreSQL
Data Analysis with TensorFlow in PostgreSQLData Analysis with TensorFlow in PostgreSQL
Data Analysis with TensorFlow in PostgreSQL
 
Machine learning for wireless networks @Bestcom2016
Machine learning for wireless networks @Bestcom2016Machine learning for wireless networks @Bestcom2016
Machine learning for wireless networks @Bestcom2016
 
Meetup 18/10/2018 - Artificiële intelligentie en mobiliteit
Meetup 18/10/2018 - Artificiële intelligentie en mobiliteitMeetup 18/10/2018 - Artificiële intelligentie en mobiliteit
Meetup 18/10/2018 - Artificiële intelligentie en mobiliteit
 
Creating a Machine Learning Model on the Cloud
Creating a Machine Learning Model on the CloudCreating a Machine Learning Model on the Cloud
Creating a Machine Learning Model on the Cloud
 
An LSTM-Based Neural Network Architecture for Model Transformations
An LSTM-Based Neural Network Architecture for Model TransformationsAn LSTM-Based Neural Network Architecture for Model Transformations
An LSTM-Based Neural Network Architecture for Model Transformations
 
Apache Spark for Cyber Security in an Enterprise Company
Apache Spark for Cyber Security in an Enterprise CompanyApache Spark for Cyber Security in an Enterprise Company
Apache Spark for Cyber Security in an Enterprise Company
 
Automata Processing: Accelerating Big Data
Automata Processing: Accelerating Big DataAutomata Processing: Accelerating Big Data
Automata Processing: Accelerating Big Data
 
Decoders.pptx
Decoders.pptxDecoders.pptx
Decoders.pptx
 

More from Patrick Michl

More from Patrick Michl (7)

Attention Please! 2022
Attention Please! 2022Attention Please! 2022
Attention Please! 2022
 
Spec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRSSpec2Vec - Energy based non-linear Calibration in NIRS
Spec2Vec - Energy based non-linear Calibration in NIRS
 
Attention please! Attention Mechanism in Neural Networks
Attention please! Attention Mechanism in Neural NetworksAttention please! Attention Mechanism in Neural Networks
Attention please! Attention Mechanism in Neural Networks
 
Regulation Analysis using Restricted Boltzmann Machines
Regulation Analysis using Restricted Boltzmann MachinesRegulation Analysis using Restricted Boltzmann Machines
Regulation Analysis using Restricted Boltzmann Machines
 
Concept of Regulation Analysis using Restricted Boltzmann Machines
Concept of Regulation Analysis using Restricted Boltzmann MachinesConcept of Regulation Analysis using Restricted Boltzmann Machines
Concept of Regulation Analysis using Restricted Boltzmann Machines
 
Synchronization and Collective Dynamics
Synchronization and Collective DynamicsSynchronization and Collective Dynamics
Synchronization and Collective Dynamics
 
Epidemic Spreading
Epidemic SpreadingEpidemic Spreading
Epidemic Spreading
 

Recently uploaded

Digital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptxDigital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptx
MohamedFarag457087
 
Module for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learningModule for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learning
levieagacer
 
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 bAsymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Sérgio Sacani
 
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune WaterworldsBiogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Sérgio Sacani
 
biology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGYbiology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGY
1301aanya
 
The Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptxThe Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptx
seri bangash
 
POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.
Silpa
 

Recently uploaded (20)

Call Girls Ahmedabad +917728919243 call me Independent Escort Service
Call Girls Ahmedabad +917728919243 call me Independent Escort ServiceCall Girls Ahmedabad +917728919243 call me Independent Escort Service
Call Girls Ahmedabad +917728919243 call me Independent Escort Service
 
COMPUTING ANTI-DERIVATIVES (Integration by SUBSTITUTION)
COMPUTING ANTI-DERIVATIVES(Integration by SUBSTITUTION)COMPUTING ANTI-DERIVATIVES(Integration by SUBSTITUTION)
COMPUTING ANTI-DERIVATIVES (Integration by SUBSTITUTION)
 
Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS ESCORT SERVICE In Bhiwan...
Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS  ESCORT SERVICE In Bhiwan...Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS  ESCORT SERVICE In Bhiwan...
Bhiwandi Bhiwandi ❤CALL GIRL 7870993772 ❤CALL GIRLS ESCORT SERVICE In Bhiwan...
 
Digital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptxDigital Dentistry.Digital Dentistryvv.pptx
Digital Dentistry.Digital Dentistryvv.pptx
 
Dr. E. Muralinath_ Blood indices_clinical aspects
Dr. E. Muralinath_ Blood indices_clinical  aspectsDr. E. Muralinath_ Blood indices_clinical  aspects
Dr. E. Muralinath_ Blood indices_clinical aspects
 
Module for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learningModule for Grade 9 for Asynchronous/Distance learning
Module for Grade 9 for Asynchronous/Distance learning
 
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 bAsymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
Asymmetry in the atmosphere of the ultra-hot Jupiter WASP-76 b
 
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
Human & Veterinary Respiratory Physilogy_DR.E.Muralinath_Associate Professor....
 
GBSN - Biochemistry (Unit 1)
GBSN - Biochemistry (Unit 1)GBSN - Biochemistry (Unit 1)
GBSN - Biochemistry (Unit 1)
 
Introduction of DNA analysis in Forensic's .pptx
Introduction of DNA analysis in Forensic's .pptxIntroduction of DNA analysis in Forensic's .pptx
Introduction of DNA analysis in Forensic's .pptx
 
Stages in the normal growth curve
Stages in the normal growth curveStages in the normal growth curve
Stages in the normal growth curve
 
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune WaterworldsBiogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
Biogenic Sulfur Gases as Biosignatures on Temperate Sub-Neptune Waterworlds
 
Grade 7 - Lesson 1 - Microscope and Its Functions
Grade 7 - Lesson 1 - Microscope and Its FunctionsGrade 7 - Lesson 1 - Microscope and Its Functions
Grade 7 - Lesson 1 - Microscope and Its Functions
 
FAIRSpectra - Enabling the FAIRification of Analytical Science
FAIRSpectra - Enabling the FAIRification of Analytical ScienceFAIRSpectra - Enabling the FAIRification of Analytical Science
FAIRSpectra - Enabling the FAIRification of Analytical Science
 
biology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGYbiology HL practice questions IB BIOLOGY
biology HL practice questions IB BIOLOGY
 
An introduction on sequence tagged site mapping
An introduction on sequence tagged site mappingAn introduction on sequence tagged site mapping
An introduction on sequence tagged site mapping
 
The Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptxThe Mariana Trench remarkable geological features on Earth.pptx
The Mariana Trench remarkable geological features on Earth.pptx
 
POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.POGONATUM : morphology, anatomy, reproduction etc.
POGONATUM : morphology, anatomy, reproduction etc.
 
GBSN - Microbiology (Unit 3)
GBSN - Microbiology (Unit 3)GBSN - Microbiology (Unit 3)
GBSN - Microbiology (Unit 3)
 
pumpkin fruit fly, water melon fruit fly, cucumber fruit fly
pumpkin fruit fly, water melon fruit fly, cucumber fruit flypumpkin fruit fly, water melon fruit fly, cucumber fruit fly
pumpkin fruit fly, water melon fruit fly, cucumber fruit fly
 

Structure learning with Deep Neural Networks

  • 1. Structure learning with deep neuronal networks 7th Network Modeling Workshop, 28/2/2014 Patrick Michl
  • 2. Page 228.02.2014 Patrick Michl Network Modeling Agenda AutoencodersAutoencoders Biological ModelBiological Model Validation & ImplementationValidation & Implementation
  • 3. Page 328.02.2014 Patrick Michl Network Modeling Real world data usually is high dimensional … x1 x2 Dataset Model Autoencoders
  • 4. Page 428.02.2014 Patrick Michl Network Modeling … which makes structural analysis and modeling complicated! x1 x2 x1 x2 Dataset Model ?  Autoencoders
  • 5. Page 528.02.2014 Patrick Michl Network Modeling Dimensionality reduction techinques like PCA … x1 x2 PCA Dataset Model Autoencoders
  • 6. Page 628.02.2014 Patrick Michl Network Modeling … can not preserve complex structures! x1 x2 PCA Dataset Model x1 x2 𝑥2=α 𝑥1+β  Autoencoders
  • 7. Page 728.02.2014 Patrick Michl Network Modeling Therefore the analysis of unknown structures … x1 x2 Dataset Model Autoencoders
  • 8. Page 828.02.2014 Patrick Michl Network Modeling … needs more considerate nonlinear techniques! x1 x2 Dataset Model x1 x2 𝑥2=𝑓(𝑥1)  Autoencoders
  • 9. Page 928.02.2014 Patrick Michl Network Modeling Autoencoders are artificial neuronal networks … Autoencoder • Artificial Neuronal Network Autoencoders input data X output data X‘ Perceptrons Gaussian Units
  • 10. Page 1028.02.2014 Patrick Michl Network Modeling Autoencoders are artificial neuronal networks … Autoencoder • Artificial Neuronal Network Autoencoders input data X output data X‘ Perceptrons Gaussian Units Perceptron 1 0 Gauss Units R
  • 11. Page 1128.02.2014 Patrick Michl Network Modeling Autoencoders are artificial neuronal networks … Autoencoder • Artificial Neuronal Network Autoencoders input data X output data X‘ Perceptrons Gaussian Units
  • 12. Page 1228.02.2014 Patrick Michl Network Modeling Autoencoder • Artificial Neuronal Network • Multiple hidden layers Autoencoders … with multiple hidden layers. Gaussian Units input data X output data X‘ Perceptrons (Visible layers) (Hidden layers)
  • 13. Page 1328.02.2014 Patrick Michl Network Modeling Autoencoder • Artificial Neuronal Network • Multiple hidden layers Autoencoders Such networks are called deep networks. Gaussian Units input data X output data X‘ Perceptrons (Visible layers) (Hidden layers)
  • 14. Page 1428.02.2014 Patrick Michl Network Modeling Autoencoder • Artificial Neuronal Network • Multiple hidden layers Autoencoders Such networks are called deep networks. Gaussian Units input data X output data X‘ Perceptrons (Visible layers) (Hidden layers)Definition (deep network) Deep networks are artificial neuronal networks with multiple hidden layers
  • 15. Page 1528.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders Gaussian Units input data X output data X‘ Perceptrons (Visible layers) (Hidden layers) Such networks are called deep networks. • Deep network
  • 16. Page 1628.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders Autoencoders have a symmetric topology … Gaussian Units input data X output data X‘ Perceptrons (Visible layers) (Hidden layers) • Deep network • Symmetric topology
  • 17. Page 1728.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders … with an odd number of hidden layers. Gaussian Units input data X output data X‘ Perceptrons (Visible layers) (Hidden layers) • Deep network • Symmetric topology
  • 18. Page 1828.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders The small layer in the center works lika an information bottleneck input data X output data X‘ • Deep network • Symmetric topology • Information bottleneck Bottleneck
  • 19. Page 1928.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders ... that creates a low dimensional code for each sample in the input data. input data X output data X‘ • Deep network • Symmetric topology • Information bottleneck Bottleneck
  • 20. Page 2028.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders The upper stack does the encoding … input data X output data X‘ • Deep network • Symmetric topology • Information bottleneck • Encoder Encoder
  • 21. Page 2128.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders … and the lower stack does the decoding. input data X output data X‘ • Deep network • Symmetric topology • Information bottleneck • Encoder • Decoder Encoder Decoder
  • 22. Page 2228.02.2014 Patrick Michl Network Modeling • Deep network • Symmetric topology • Information bottleneck • Encoder • Decoder Autoencoder Autoencoders … and the lower stack does the decoding. input data X output data X‘ Encoder Decoder Definition (deep network) Deep networks are artificial neuronal networks with multiple hidden layers Definition (autoencoder) Autoencoders are deep networks with a symmetric topology and an odd number of hiddern layers, containing a encoder, a low dimensional representation and a decoder.
  • 23. Page 2328.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders Autoencoders can be used to reduce the dimension of data … input data X output data X‘ Problem: dimensionality of data Idea: 1. Train autoencoder to minimize the distance between input X and output X‘ 2. Encode X to low dimensional code Y 3. Decode low dimensional code Y to output X‘ 4. Output X‘ is low dimensional
  • 24. Page 2428.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders … if we can train them! input data X output data X‘ Problem: dimensionality of data Idea: 1. Train autoencoder to minimize the distance between input X and output X‘ 2. Encode X to low dimensional code Y 3. Decode low dimensional code Y to output X‘ 4. Output X‘ is low dimensional
  • 25. Page 2528.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders In feedforward ANNs backpropagation is a good approach. input data X output data X‘ Training Backpropagation
  • 26. Page 2628.02.2014 Patrick Michl Network Modeling Backpropagation Autoencoder Autoencoders input data X output data X‘ Training Definition (autoencoder) Backpropagation (1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function error   In feedforward ANNs backpropagation is a good approach.
  • 27. Page 2728.02.2014 Patrick Michl Network Modeling Backpropagation Autoencoder Autoencoders In feedforward ANNs backpropagation is the choice input data X output data X‘ Training Definition (autoencoder) Backpropagation (1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function Example (linear neuronal unit with two inputs)
  • 28. Page 2828.02.2014 Patrick Michl Network Modeling Backpropagation Autoencoder Autoencoders input data X output data X‘ Training Definition (autoencoder) Backpropagation (1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function (2) By calculating we get a vector that shows in a direction which decreases the error (3) We update the parameters to decrease the error   In feedforward ANNs backpropagation is a good approach.
  • 29. Page 2928.02.2014 Patrick Michl Network Modeling Backpropagation Autoencoder Autoencoders In feedforward ANNs backpropagation is the choice input data X output data X‘ Training Definition (autoencoder) Backpropagation (1) The distance (error) between current output X‘ and wanted output Y is computed. This gives a error function (2) By calculating we get a vector that shows in a direction which decreases the error (3) We update the parameters to decrease the error (4) We repeat that  
  • 30. Page 3028.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders … the problem are the multiple hidden layers! input data X output data X‘ Training Backpropagation Problem: Deep Network
  • 31. Page 3128.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation is known to be slow far away from the output layer … Backpropagation Problem: Deep Network • Very slow training
  • 32. Page 3228.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training … and can converge to poor local minima. Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution
  • 33. Page 3328.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution The task is to initialize the parameters close to a good solution!
  • 34. Page 3428.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution • Pretraining Therefore the training of autoencoders has a pretraining phase …
  • 35. Page 3528.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution • Pretraining • Restricted Boltzmann Machines … which uses Restricted Boltzmann Machines (RBMs)
  • 36. Page 3628.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution • Pretraining • Restricted Boltzmann Machines … which uses Restricted Boltzmann Machines (RBMs) Restricted Boltzmann Machine • RBMs are Markov Random Fields
  • 37. Page 3728.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution • Pretraining • Restricted Boltzmann Machines … which uses Restricted Boltzmann Machines (RBMs) Restricted Boltzmann Machine • RBMs are Markov Random Fields Markov Random Field Every unit influences every neighbor The coupling is undirected Motivation (Ising Model) A set of magnetic dipoles (spins) is arranged in a graph (lattice) where neighbors are coupled with a given strengt
  • 38. Page 3828.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution • Pretraining • Restricted Boltzmann Machines … which uses Restricted Boltzmann Machines (RBMs) Restricted Boltzmann Machine • RBMs are Markov Random Fields • Bipartite topology: visible (v), hidden (h) • Use local energy to calculate the probabilities of values Training: contrastive divergency (Gibbs Sampling) h1 v1 v2 v3 v4 h2 h3
  • 39. Page 3928.02.2014 Patrick Michl Network Modeling Autoencoder Autoencoders input data X output data X‘ Training Backpropagation Problem: Deep Network • Very slow training • Maybe bad solution Idea: Initialize close to a good solution • Pretraining • Restricted Boltzmann Machines … which uses Restricted Boltzmann Machines (RBMs) Restricted Boltzmann Machine Gibbs Sampling
  • 40. Page 4028.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder The top layer RBM transforms real value data into binary codes.   Top Training
  • 41. Page 4128.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Top Therefore visible units are modeled with gaussians to encode data …   h2 v1 v2 v3 v4 h3 h4 h5h1 Training
  • 42. Page 4228.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Top … and many hidden units with simoids to encode dependencies   h2 v1 v2 v3 v4 h3 h4 h5h1 Training
  • 43. Page 4328.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Top The objective function is the sum of the local energies. Local Energy   𝐸 𝑣 ≔ −∑ h 𝑤 𝑣h 𝑥 𝑣 𝜎 𝑣 𝑥 h + ( 𝑥 𝑣 − 𝑏 𝑣 )2 2 𝜎 𝑣 2   h2 v1 v2 v3 v4 h3 h4 h5h1 Training
  • 44. Page 4428.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Reduction   The next RBM layer maps the dependency encoding… Training
  • 45. Page 4528.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Reduction … from the upper layer … v   h1 v1 v2 v3 v4 h2 h3 Training
  • 46. Page 4628.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Reduction … to a smaller number of simoids … h   h1 v1 v2 v3 v4 h2 h3 Training
  • 47. Page 4728.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Reduction … which can be trained faster than the top layer Local Energy 𝐸𝑣 ≔−∑ h 𝑤 𝑣h 𝑥 𝑣 𝑥h+𝑥h 𝑏h   𝐸h ≔−∑ 𝑣 𝑤 𝑣h 𝑥 𝑣 𝑥h+𝑥 𝑣 𝑏 𝑣   h1 v1 v2 v3 v4 h2 h3 Training
  • 48. Page 4828.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Unrolling The symmetric topology allows us to skip further training. Training
  • 49. Page 4928.02.2014 Patrick Michl Network Modeling Autoencoders Autoencoder Unrolling The symmetric topology allows us to skip further training. Training
  • 50. Page 5028.02.2014 Patrick Michl Network Modeling After pretraining backpropagation usually finds good solutions Autoencoders Autoencoder Training • Pretraining Top RBM (GRBM) Reduction RBMs Unrolling • Finetuning Backpropagation
  • 51. Page 5128.02.2014 Patrick Michl Network Modeling The algorithmic complexity of RBM training depends on the network size Autoencoders Autoencoder Training • Complexity: O(inw) i: number of iterations n: number of nodes w: number of weights • Memory Complexity: O(w)
  • 52. Page 5228.02.2014 Patrick Michl Network Modeling Agenda Autoencoders Biological Model Validation & Implementation
  • 53. Page 5328.02.2014 Patrick Michl Network Modeling Network Modeling Restricted Boltzmann Machines (RBM) How to model the topological structure? S E TF
  • 54. Page 5428.02.2014 Patrick Michl Network Modeling We define S and E as visible data Layer … S E TF Network Modeling Restricted Boltzmann Machines (RBM)
  • 55. Page 5528.02.2014 Patrick Michl Network Modeling S E TF Network Modeling Restricted Boltzmann Machines (RBM) We identify S and E with the visible layer …
  • 56. Page 5628.02.2014 Patrick Michl Network Modeling S E … and the TFs with the hidden layer in a RBM TF Network Modeling Restricted Boltzmann Machines (RBM)
  • 57. Page 5728.02.2014 Patrick Michl Network Modeling S E The training of the RBM gives us a model TF Network Modeling Restricted Boltzmann Machines (RBM)
  • 58. Page 5828.02.2014 Patrick Michl Network Modeling Agenda Autoencoder Biological Model Implementation & Results
  • 59. Page 5928.02.2014 Patrick Michl Network Modeling Results Validation of the results • Needs information about the true regulation • Needs information about the descriptive power of the data
  • 60. Page 6028.02.2014 Patrick Michl Network Modeling Results Validation of the results • Needs information about the true regulation • Needs information about the descriptive power of the data Without this infomation validation can only be done, using artificial datasets!
  • 61. Page 6128.02.2014 Patrick Michl Network Modeling Results Artificial datasets We simulate data in three steps:
  • 62. Page 6228.02.2014 Patrick Michl Network Modeling Results Artificial datasets We simulate data in three steps Step 1 Choose number of Genes (E+S) and create random bimodal distributed data
  • 63. Page 6328.02.2014 Patrick Michl Network Modeling Results Artificial datasets We simulate data in three steps Step 1 Choose number of Genes (E+S) and create random bimodal distributed data Step 2 Manipulate data in a fixed order
  • 64. Page 6428.02.2014 Patrick Michl Network Modeling Results Artificial datasets We simulate data in three steps Step 1 Choose number of Genes (E+S) and create random bimodal distributed data Step 2 Manipulate data in a fixed order Step 3 Add noise to manipulated data and normalize data
  • 65. Page 6528.02.2014 Patrick Michl Network Modeling Simulation Results Step 1 Number of visible nodes 8 (4E, 4S) Create random data: Random {-1, +1} + N(0,  
  • 66. Page 6628.02.2014 Patrick Michl Network Modeling Simulation Results Noise   Step 2 Manipulate data
  • 67. Page 6728.02.2014 Patrick Michl Network Modeling Simulation Results Step 3 Add noise: N(0,  
  • 68. Page 6828.02.2014 Patrick Michl Network Modeling Results We analyse the data X with an RBM
  • 69. Page 6928.02.2014 Patrick Michl Network Modeling Results We train an autoencoder with 9 hidden layers and 165 nodes: Layer 1 & 9: 32 hidden units Layer 2 & 8: 24 hidden units Layer 3 & 7: 16 hidden units Layer 4 & 6: 8 hidden units Layer 5: 5 hidden units input data X output data X‘
  • 70. Page 7028.02.2014 Patrick Michl Network Modeling Results We transform the data from X to X‘ And reduce the dimensionality
  • 71. Page 7128.02.2014 Patrick Michl Network Modeling Results We analyse the transformed data X‘ with an RBM
  • 72. Page 7228.02.2014 Patrick Michl Network Modeling Results Lets compare the models
  • 73. Page 7328.02.2014 Patrick Michl Network Modeling Results Another Example with more nodes and larger autoencoder
  • 74. Page 7428.02.2014 Patrick Michl Network Modeling Conclusion Conclusion • Autoencoders can improve modeling significantly by reducing the dimensionality of data • Autoencoders preserve complex structures in their multilayer perceptron network. Analysing those networks (for example with knockout tests) could give more structural information • The drawback are high computational costs Since the field of deep learning is getting more popular (Face recognition / Voice recognition, Image transformation). Many new improvements in facing the computational costs have been made.
  • 75. Page 7528.02.2014 Patrick Michl Network Modeling Acknowledgement eilsLABS Prof. Dr. Rainer König Prof. Dr. Roland Eils Network Modeling Group