岳華 杜, profile picture
Uploaded by岳華 杜
PDF, PPTX531 views

從 VAE 走向深度學習新理論

The document discusses the evolution of deep learning theories, particularly focusing on variations of autoencoders like Variational Autoencoders (VAEs) and their components such as latent variables and generative models. It highlights foundational concepts like the information bottleneck method, stochastic gradient descent, and various applications in representation learning and feature extraction. The document also examines the relationships between mutual information, entropy, and their relevance in training neural networks.

Technology◦
Related topics:
Deep Learning•

Embed presentation

Download as PDF, PPTX
從VAE ⾛向深度學習新理論從VAE ⾛向深度學習新理論 杜岳華
Deep Learning is a kind of Representational LearningDeep Learning is a kind of Representational Learning
Deep Learning is a kind of Representational LearningDeep Learning is a kind of Representational Learning picture source (https://www.deeplearningbook.org/)
Representational LearningRepresentational Learning x2 x3 x4 x5 z4 z3 z2 z1 x1 f woman ClassifierFeature extractor g
Representational LearningRepresentational Learning woman Representation Learning: A Review and New Perspectives (https://arxiv.org/abs/1206.5538)
AutoencoderAutoencoder x2 x3 x4 x5 z2 z1 x1 x2 x3 x4 x5 x1 z1 z2
Restricted Boltzmann MachinesRestricted Boltzmann Machines An unsupervised greedy way to extract featuresAn unsupervised greedy way to extract features 發明：發明： Smolensky, Paul (1986). Chapter 6: Information Processing in Dynamical Systems: Foundations of Harmony Theory. 應⽤：應⽤： 降維：Hinton, G. E.; Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science. 分類：Larochelle, H.; Bengio, Y. (2008). Classi cation using discriminative restricted Boltzmann machines. ICML '08. 協同過濾：Salakhutdinov, R.; Mnih, A.; Hinton, G. (2007). Restricted Boltzmann machines for collaborative ltering. ICML '07. 特徵學習：Coates, Adam; Lee, Honglak; Ng, Andrew Y. (2011). An analysis of single-layer networks in unsupervised feature learning. International Conference on Arti cial Intelligence and Statistics (AISTATS).
Restricted Boltzmann MachinesRestricted Boltzmann Machines x2 x3 x4 x5 z2 z1 x1 A Beginner's Guide to Restricted Boltzmann Machines (RBMs) (https://skymind.ai/wiki/restricted-boltzmann-machine)
Deep Belief Network [Hinton]Deep Belief Network [Hinton] A greedy layerwise unsupervised pre-training methodA greedy layerwise unsupervised pre-training method W1 W1 W2
Deep Belief NetworkDeep Belief Network W1 W2 W1 T W2 T
We need generative model!We need generative model! Discriminative model: Generative model: p(Y |X) p(X, Y )
Disentangle explanatory generative factorsDisentangle explanatory generative factors to disentangle as many factors as possible, discarding as little information about the data as is practical x2 x3 x4 x5 z2 z1 x1 x2 x3 x4 x5 x1 z1 z2
Variational AutoencoderVariational Autoencoder
A generative modelA generative model z x N We hope to learn generative factors by unsupervised method
The factorThe factor xi yi ^yi=axi+b mean ^y variance σ2
The factorThe factor x y N y=θ0+θ1 x θ θ=(θ0, θ1)
To learn latent random variablesTo learn latent random variables z x N θ
Introduce Bayesian theoremIntroduce Bayesian theorem (z|x) =pθ (x|z) (z)pθ pθ (x)pθ (x) = ∫ (x|z) (z)dzpθ pθ pθ is intractable.(x)pθ Variational inference: useVariational inference: use to approximateto approximate(z|x)qϕ (z|x)pθ
Kullback–Leibler divergenceKullback–Leibler divergence Relative entropy, to measure the dissimilarity between two distributions. Use data to approximate theoretical distributionp(X) q(X) (p(X)||q(X)) = − p( ) log DKL ∑ i xi q( )xi p( )xi 1. Asymmetry 2. Not distance 3. 4. and are equal (p(X)||q(X)) ≥ 0DKL (p(X)||q(X)) = 0DKL ⇔ p(X) q(X)
FormulationFormulation (z|x) =pθ (x|z) (z)pθ pθ (x)pθ arg ( (z|x)|| (z|x))min ϕ DKL qϕ pθ x zφ θ N x z θ N
ArchitectureArchitecture x z x z encoder decoder qϕ(z∣x) pθ(z∣x) z= f (x) x= g(z) θφ x z x gf
Evidence Lower Bound method (ELOB)Evidence Lower Bound method (ELOB) ( (z|x)|| (z|x))DKL qϕ pθ = ∫ q(z|x) log  dz q(z|x) p(z|x) = ∫ q(z|x) log  dz q(z|x)p(x) p(x, z) = ∫ q(z|x) log  dz + ∫ q(z|x) log p(x)dz q(z|x) p(x, z) = ∫ q(z|x)(log q(z|x) − log p(x, z))dz + log p(x) = − [log p(x, z) − log q(z|x)] + log p(x)Eq(z|x)
Evidence Lower Bound method (ELOB)Evidence Lower Bound method (ELOB) Let is called (variational) lower bound or evidence lower bound. L(θ, ϕ, x) = [log  (x, z) − log  (z|x)]E (z|x)qϕ pθ qϕ ( (z|x)|| (z|x)) = −L(θ, ϕ, x) + log p(x)DKL qϕ pθ log p(x) = ( (z|x)|| (z|x)) ↙ +L(θ, ϕ, x) ↗DKL qϕ pθ
Evidence Lower Bound method (ELOB)Evidence Lower Bound method (ELOB) Encoder: , Decoder: (z|x) =pθ (x|z) (z)pθ pθ (x)pθ arg ( (z|x)|| (z|x))min ϕ DKL qϕ pθ ⇓ (z|x) =pθ (x|z) (z)pθ pθ (x)pθ arg L(θ, ϕ, x)max θ,ϕ (z|x)qϕ (x|z)pθ
Hypothesis: gaussian mixture as latent representationHypothesis: gaussian mixture as latent representation z2 z1 μz2 μz1 σz 2 σz1 z2 z2 z1 z1
Encoder and decoderEncoder and decoder z2 z2 z1 z1 encoderencoder decoder
How to solve?How to solve? Mean eld variational approximation Sampling by Markov chain Monte Carlo More?
Sampling by MCMCSampling by MCMC picture source (https://www.youtube.com/watch? v=OTO1DygELpY)
Stochastic gradient descent?Stochastic gradient descent? L(θ, ϕ, x) = [log  (x, z) − log  (z|x)]E (z|x)qϕ pθ qϕ L(θ, ϕ, x) = [−log  (z|x)]∇ϕ ∇ϕ E (z|x)qϕ qϕ
Reparameterization trickReparameterization trick Encoder ( ) Decoder ( ) Sample from Encoder ( ) Decoder ( ) Sample from * + Tutorial on Variational Autoencoders (https://arxiv.org/abs/1606.05908)
Stochastic gradient variational bayes (SGVB)Stochastic gradient variational bayes (SGVB) ⾒Algorithm 1 in Auto-Encoding Variational Bayes⾒Algorithm 1 in Auto-Encoding Variational Bayes
Example: variational autoencoderExample: variational autoencoder z2 z2 z1 z1 encoderencoder decoder
ExperimentsExperiments (a) Learned Frey Face manifold (b) Learned MNIST manifold
-variational Autoencoder-variational Autoencoderβ
Achieve disentangled explainable generative factorAchieve disentangled explainable generative factor
Achieve disentangled explainable generative factorAchieve disentangled explainable generative factor
Achieve disentangled explainable generative factorAchieve disentangled explainable generative factor Figure 6 in β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK
What is the di erence between VAE andWhat is the di erence between VAE and -VAE?-VAE?β VAE: -VAE: arg max L(θ, ϕ, x) = [log  (x|z)] − ( (z|x)|| (z))E (z|x)qϕ pθ DKL qϕ pθ β arg max L(θ, ϕ, x) = [log  (x|z)] − β ( (z|x)|| (z))E (z|x)qϕ pθ DKL qϕ pθ L(θ, ϕ, x) = [log  (x, z) − log  (z|x)]E (z|x)qϕ pθ qϕ = ∫ (z|x)(log  (x, z) − log  (z|x))dzqϕ pθ qϕ = ∫ (z|x)(log  − log  )dzqϕ (x, z)pθ (z)pθ (z|x)qϕ (z)pθ = [log  (x|z)] − ( (z|x)|| (z))E (z|x)qϕ pθ DKL qϕ pθ
Why?Why? The higher encourages learning a disentangled representation. : encourage to learn good representations. : constraint the capacity of β [log  (x|z)]E (z|x)qϕ pθ ( (z|x)|| (z))DKL qϕ pθ z
The information bottleneck methodThe information bottleneck method arg max I (Z; Y ) − βI (X; Z) : maximize mutual information between Z and Y. : discard irrelevant information about Y from X. I (Z; Y ) I (X; Z) Learning is about forgetting irrelevant details.Learning is about forgetting irrelevant details.
ExperimentsExperiments Understanding disentangling in β-VAE (https://arxiv.org/abs/1804.03599)
Information Bottleneck TheoryInformation Bottleneck Theory
Basic Information theoryBasic Information theory EntropyEntropy Information entropy, Shannon entropy Measure the uncertainty of an event. H(X) = E(I (X)) = − p( ) log p( )∑ i=1 n xi xi 1. Nonnegativity: 2. Symmetry: 3. If and are independent random variable: H(X) ≥ 0 H(X, Y ) = H(Y , X) X Y H(X|Y ) = H(X)
EntropyEntropy 天氣預報100% 下⾬，0% 晴天： 天氣預報80% 下⾬，20% 晴天： 天氣預報50% 下⾬，50% 晴天： 1 lo  1 + 0 lo  0 = 0 + 0 = 0g2 g2 −0.8 lo  0.8 − 0.2 lo  0.2 = 0.258 + 0.464 = 0.722g2 g2 −0.5 lo 0.5 − 0.5 lo 0.5 = 0.5 + 0.5 = 1g2 g2
EntropyEntropy 0 0.5 10 0.5 1 Pr(X=1) H(X) ) picture source (https://en.wikipedia.org/wiki/Entropy_(information_theory)
Conditional entropyConditional entropy To measure how much information needed to describe the outcome of a random variable Y given that the value of another random variable X is known. H(Y |X) = p(x)H(Y |X = x)∑ x∈X = − p(x) p(y|x) log p(y|x)∑ x∈X ∑ y∈Y = − p(x, y) log ∑ x∈X ,y∈Y p(x, y) p(x)
Mutual informationMutual information To measure how much information obtained about one random variable through observing the other. I (X; Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X) = H(X) + H(Y ) − H(X, Y ) = p(x, y) log ∑ x,y p(x, y) p(x)p(y) 1. Nonnegativity: 2. Symmetry: I (X; Y ) ≥ 0 I (X; Y ) = I (Y ; X)
Relation to Kullback–Leibler divergenceRelation to Kullback–Leibler divergence I (X; Y ) = (p(X, Y )||p(X)p(Y ))DKL
RelationRelation picture source (https://en.wikipedia.org/wiki/Mutual_information)
RelationRelation picture source (https://en.wikipedia.org/wiki/Mutual_information)
Cross entropyCross entropy How much difference between two distributions. H(q, p) = H(q) + (q||p)DKL = − p(x) log q(x)∑ x DKL(q∣p) H (q) H (q, p) NOTION: notation confused with joint entropy.
Di erence between mutual information and cross entropyDi erence between mutual information and cross entropy Mutual information Measure the information share between two random variables. Cross entropy Measure the difference between two distributions.
Data processing inequality (DPI)Data processing inequality (DPI) Let be a Markov chain, thenX → Y → Z I (X; Y ) ≥ I (X; Z)
The neural network generates a successive Markov chainThe neural network generates a successive Markov chain Treat the whole layer as a single random variableTi Encoder Decoder I (X; Y ) ≥ I ( ; Y ) ≥ I ( ; Y ) ≥. . . ≥ I ( ; Y ) ≥ I ( ; Y )T1 T2 Tm Y^ H(X) ≥ I (X; ) ≥ I (X; ) ≥. . . ≥ I (X; ) ≥ I (X; )T1 T2 Tm Y^
Codebook and volumeCodebook and volume Let : signal source with xed probability measure : quantized codebook : a soft partition of , with probability with X p(x) X^ p( |x)x^ X p( ) = p(x)p( |x)x^ ∑ x x^
What determines the quality of a quantization?What determines the quality of a quantization? Rate, the average numbers of bits per message to encode the signal. The information to transmit from to is bounded from belowX X^ I (X; )X^
Rate distortion theoryRate distortion theory Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 1 Lossy compression n Lower the bit-rate R by allowing some acceptable distortion D of the signal. Distortion D Rate R Lossless coding D=0
Rate distortion theoryRate distortion theory Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 2 Types of lossy compression problems D R n Given maximum rate R, minimize distortion D n Given distortion D, minimize rate R D R Equivalent constrained optimization problems, often unwieldy due to constraint.
Rate distortion theoryRate distortion theory Def. rate distortion function as R(D) = min I (X; )X^ w. r. t. E[d(x, )] ≤ Dx^ Apply Lagrange multiplier: F (p( |x)) = I (X; ) + βE[d(x, )]x^ X^ x^
Information bottleneck methodInformation bottleneck method , thenX → → YX^ I (X; ) ≥ I (X; Y )X^ Information bottleneck: arg min L(x, ) = I (X; ) − βI ( ; Y )x^ X^ X^ We want this quantization to capture as much information about tradeoff between compress the representation and preserve meaningful information. Y
Information bottleneck methodInformation bottleneck method x2 x3 x4 x5 z4 z3 z2 z1 x1 x2 x3 x4 x5 x1
Opening the black box of Deep Neural Networks viaOpening the black box of Deep Neural Networks via InformationInformation
IssuesIssues 1. The SGD layer dynamics in the Information plane. 2. The effect of the training sample size on the layers. 3. What is the bene t of the hidden layers? 4. What is the nal location of the hidden layers? 5. Do the hidden layers form optimal IB representations?
SetupSetup standard DNN settings tanh as activation function sigmoid function in the nal layer train with SGD and cross-entropy loss 7 fully connected hidden layers with widths: 12-10-7-5-4-3-2 neurons
Information planeInformation plane Encoder Decoder Given , plot point on the information plane. Applied to the Markov chain of a k-layers of DNN, connected points form a unique information path. P (X; Y ) (I (X; T ), I (T ; Y ))
The dynamics of the training by Stochastic-Gradient-DecentThe dynamics of the training by Stochastic-Gradient-Decent 50 different randomized initializations with different randomized training samples init − 400epochs − 9000epochs The optimization process in the Information Plane (https://www.youtube.com/watch? v=P1A1yNsxMjc)
The two optimization phases in the Information PlaneThe two optimization phases in the Information Plane 5% - 45% - 85% training samples5% - 45% - 85% training samples Emperical risk minimization (ERM) phase (fast) increase layer learn the information while preserving the DPI order Representation compression phase (slow) decrease until convergence layer lose irrelevant information (compression) IY IX
The drift and di usion phases of SGD optimizationThe drift and di usion phases of SGD optimization Layer weight's gradient distributionsLayer weight's gradient distributions
The drift and di usion phases of SGD optimizationThe drift and di usion phases of SGD optimization Drift phase large gradient mean, small variance (high SNR) increase and reduce the emperical error ERM phase Diffusion phase small gradient mean, large uctuations (low SNR) the gradients behave like Gaussian noise, weights evolve like Wiener process compression phase Maximize the entropy of the weight distribution by addiing noise, known as stochastic relaxation compression by diffusion phase attempts to interpret single weights or even single neurons in such networks can be meaningless IY
The computational bene t of the hidden layersThe computational bene t of the hidden layers Train 6 different architecture with 1-6 hidden layers
The computational bene t of the hidden layersThe computational bene t of the hidden layers 1. Adding hidden layers dramatically reduces the number of training epochs for good generalization. 2. The compression phase of each layer is shorter when it starts from a previous compressed layer. 3. The compression is faster for the deeper (narrower and closer to the output) layers. 4. Even wide hidden layers eventually compress in the diffusion phase. Adding extra width does not help.
Convergence to the layers to the Information Bottleneck boundConvergence to the layers to the Information Bottleneck bound
Evolution of the layers with training sample sizeEvolution of the layers with training sample size 0 1 2 3 4 5 6 7 8 9 I(X;T) 0.3 0.4 0.5 0.6 0.7 I(T;Y) 4% 84% Training data
with increasing training size the layers’ true label information (generalization) is pushed up and gets closer to the theoretical IB bound for the rule distribution. IY
Are our ndings general enough?Are our ndings general enough?
Hinton 的評論Hinton 的評論 Hinton 在聽完Tishby 的talk 之後，給Tishby 發了email: “I have to listen to it another 10,000 times to really understand it, but it’s very rare nowadays to hear a talk with a really original idea in it that may be the answer to a really major puzzle.”
Caution!Caution! No, information bottleneck (probably) doesn’t open the “black-box” of deep neural n (https://severelytheoretical.wordpress.com/2017/09/28/no-information-bottlenec black-box-of-deep-neural-networks/) Tishby's 'Opening the Black Box of Deep Neural Networks via Information' received (https://www.reddit.com/r/MachineLearning/comments/72eau7/d_tishbys_opening On the Information Bottleneck Theory of Deep Learning [Harvard University] [ICLR (https://openreview.net/forum?id=ry_WPG-A-)
Thank you for attentionThank you for attention ReferenceReference 18. Information Theory of Deep Learning. Naftali Tishby (https://www.youtube.com/watch?v=bLqJHjXihK8)

More Related Content

PPTX
Training data-efficient image transformers & distillation through attention
bytaeseon ryu
 
PDF
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
PDF
The Transformer - Xavier Giró - UPC Barcelona 2021
byUniversitat Politècnica de Catalunya
 
PPTX
Transformers in Vision: From Zero to Hero
byBill Liu
 
XLS
Kochie budget
byCarlynnY
 
PDF
Latent diffusions vs DALL-E v2
byVitaly Bondar
 
PPTX
Transformer in Vision
bySangmin Woo
 
PPTX
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
bychangedaeoh
 
Training data-efficient image transformers & distillation through attention
bytaeseon ryu
 
Variational Autoencoders VAE - Santiago Pascual - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
The Transformer - Xavier Giró - UPC Barcelona 2021
byUniversitat Politècnica de Catalunya
 
Transformers in Vision: From Zero to Hero
byBill Liu
 
Kochie budget
byCarlynnY
 
Latent diffusions vs DALL-E v2
byVitaly Bondar
 
Transformer in Vision
bySangmin Woo
 
Vision Transformer(ViT) / An Image is Worth 16*16 Words: Transformers for Ima...
bychangedaeoh
 

What's hot

PPTX
Quantum Computing ppt
byOECLIB Odisha Electronics Control Library
 
PPTX
Generative Adversarial Networks (GAN)
byManohar Mukku
 
PPTX
Deep Learning Interview Questions And Answers | AI & Deep Learning Interview ...
bySimplilearn
 
PDF
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
PPTX
Day 1 (Lecture 3): Predictive Analytics in Healthcare
byAseda Owusua Addai-Deseh
 
PDF
Self-supervised Learning Lecture Note
bySangwoo Mo
 
PDF
Paraphrase Detection in NLP
byYuriy Guts
 
PPTX
Quantum Machine Learning for IBM AI
bySasha Lazarevic
 
PDF
PR-409: Denoising Diffusion Probabilistic Models
byHyeongmin Lee
 
PDF
Photodiode amplifers
bywtyru1989
 
PDF
Tutorial on Deep Generative Models
byMLReview
 
PDF
How deep generative models can help quants reduce the risk of overfitting?
byGautier Marti
 
PDF
Transformer Introduction (Seminar Material)
byYuta Niki
 
PDF
[Paper] Multiscale Vision Transformers(MVit)
bySusang Kim
 
PDF
Object Detection Beyond Mask R-CNN and RetinaNet I
byWanjin Yu
 
PDF
Quantum computation: EPR Paradox and Bell's Inequality
byStefano Franco
 
PDF
Review: Incremental Few-shot Instance Segmentation [CDM]
byDongmin Choi
 
PPTX
logic family
bySachit Kumar
 
PDF
2 diode applications
byعلي العراقي الوطني
 
PPTX
Attention in Deep Learning
by健程 杨
 
Quantum Computing ppt
byOECLIB Odisha Electronics Control Library
 
Generative Adversarial Networks (GAN)
byManohar Mukku
 
Deep Learning Interview Questions And Answers | AI & Deep Learning Interview ...
bySimplilearn
 
Loss Functions for Deep Learning - Javier Ruiz Hidalgo - UPC Barcelona 2018
byUniversitat Politècnica de Catalunya
 
Day 1 (Lecture 3): Predictive Analytics in Healthcare
byAseda Owusua Addai-Deseh
 
Self-supervised Learning Lecture Note
bySangwoo Mo
 
Paraphrase Detection in NLP
byYuriy Guts
 
Quantum Machine Learning for IBM AI
bySasha Lazarevic
 
PR-409: Denoising Diffusion Probabilistic Models
byHyeongmin Lee
 
Photodiode amplifers
bywtyru1989
 
Tutorial on Deep Generative Models
byMLReview
 
How deep generative models can help quants reduce the risk of overfitting?
byGautier Marti
 
Transformer Introduction (Seminar Material)
byYuta Niki
 
[Paper] Multiscale Vision Transformers(MVit)
bySusang Kim
 
Object Detection Beyond Mask R-CNN and RetinaNet I
byWanjin Yu
 
Quantum computation: EPR Paradox and Bell's Inequality
byStefano Franco
 
Review: Incremental Few-shot Instance Segmentation [CDM]
byDongmin Choi
 
logic family
bySachit Kumar
 
2 diode applications
byعلي العراقي الوطني
 
Attention in Deep Learning
by健程 杨
 

Similar to 從 VAE 走向深度學習新理論

PDF
Bayesian Deep Learning
byRayKim51
 
PDF
VAE-type Deep Generative Models
byKenta Oono
 
PDF
Probabilistic AI Lecture 1: Introduction to variational inference and the ELBO
bycniclsh1
 
PDF
Iclr2016 vaeまとめ
byDeep Learning JP
 
PDF
Uncertainty in Deep Learning
byRoberto Pereira Silveira
 
PPTX
Bayesian Neural Networks
byNatan Katz
 
PDF
Introduction to Variational Auto Encoder
byvaidehimadaan041
 
PDF
Generative models : VAE and GAN
bySEMINARGROOT
 
PDF
is anyone_interest_in_auto-encoding_variational-bayes
byNAVER Engineering
 
PDF
[DL輪読会]Recent Advances in Autoencoder-Based Representation Learning
byDeep Learning JP
 
PDF
Bayesian Learning- part of machine learning
bykensaleste
 
PDF
AI 로봇 아티스트의 비밀(창원대학교 정보통신공학과 특강)
byChangwon National University
 
PPTX
GAN for Bayesian Inference objectives
byNatan Katz
 
PDF
Learning stochastic neural networks with Chainer
bySeiya Tokui
 
PDF
Variational Autoencoder from scratch.pdf
bynamnguynhi30
 
PPT
Machine Learning and Inductive Inference
bybutest
 
PPTX
Deep_Learning.pptx on model training and evaluating
by227r1a7374
 
PDF
Variational autoencoders for speech processing d.bielievtsov dataconf 21 04 18
byOlga Zinkevych
 
PPTX
Machine Learning Algorithms Review(Part 2)
byZihui Li
 
PPTX
Generative models
byAvner Gidron
 
Bayesian Deep Learning
byRayKim51
 
VAE-type Deep Generative Models
byKenta Oono
 
Probabilistic AI Lecture 1: Introduction to variational inference and the ELBO
bycniclsh1
 
Iclr2016 vaeまとめ
byDeep Learning JP
 
Uncertainty in Deep Learning
byRoberto Pereira Silveira
 
Bayesian Neural Networks
byNatan Katz
 
Introduction to Variational Auto Encoder
byvaidehimadaan041
 
Generative models : VAE and GAN
bySEMINARGROOT
 
is anyone_interest_in_auto-encoding_variational-bayes
byNAVER Engineering
 
[DL輪読会]Recent Advances in Autoencoder-Based Representation Learning
byDeep Learning JP
 
Bayesian Learning- part of machine learning
bykensaleste
 
AI 로봇 아티스트의 비밀(창원대학교 정보통신공학과 특강)
byChangwon National University
 
GAN for Bayesian Inference objectives
byNatan Katz
 
Learning stochastic neural networks with Chainer
bySeiya Tokui
 
Variational Autoencoder from scratch.pdf
bynamnguynhi30
 
Machine Learning and Inductive Inference
bybutest
 
Deep_Learning.pptx on model training and evaluating
by227r1a7374
 
Variational autoencoders for speech processing d.bielievtsov dataconf 21 04 18
byOlga Zinkevych
 
Machine Learning Algorithms Review(Part 2)
byZihui Li
 
Generative models
byAvner Gidron
 

More from 岳華 杜

PDF
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
by岳華 杜
 
PPTX
Introduction to julia
by岳華 杜
 
PPTX
COSCUP: Introduction to Julia
by岳華 杜
 
PDF
Julia: The language for future
by岳華 杜
 
PDF
20170715 北Bio meetup
by岳華 杜
 
PDF
自然語言處理概覽
by岳華 杜
 
PPTX
Metaprogramming in julia
by岳華 杜
 
PPTX
20171127 當julia遇上資料科學
by岳華 杜
 
PPTX
Introduction to Julia
by岳華 杜
 
PDF
The Language for future-julia
by岳華 杜
 
PDF
COSCUP: Metaprogramming in Julia
by岳華 杜
 
PDF
20190907 Julia the language for future
by岳華 杜
 
PPTX
20171117 oop and design patterns in julia
by岳華 杜
 
PDF
[COSCUP 2023] 我的Julia軟體架構演進之旅
by岳華 杜
 
PPTX
Introduction to machine learning
by岳華 杜
 
PPTX
20180506 Introduction to machine learning
by岳華 杜
 
PDF
COSCUP: Foreign Function Call in Julia
by岳華 杜
 
PDF
Batch normalization 與他愉快的小伙伴
by岳華 杜
 
PPTX
20171014 tips for manipulating filesystem in julia
by岳華 杜
 
PDF
20170807 julia的簡單而高效資料處理
by岳華 杜
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
by岳華 杜
 
Introduction to julia
by岳華 杜
 
COSCUP: Introduction to Julia
by岳華 杜
 
Julia: The language for future
by岳華 杜
 
20170715 北Bio meetup
by岳華 杜
 
自然語言處理概覽
by岳華 杜
 
Metaprogramming in julia
by岳華 杜
 
20171127 當julia遇上資料科學
by岳華 杜
 
Introduction to Julia
by岳華 杜
 
The Language for future-julia
by岳華 杜
 
COSCUP: Metaprogramming in Julia
by岳華 杜
 
20190907 Julia the language for future
by岳華 杜
 
20171117 oop and design patterns in julia
by岳華 杜
 
[COSCUP 2023] 我的Julia軟體架構演進之旅
by岳華 杜
 
Introduction to machine learning
by岳華 杜
 
20180506 Introduction to machine learning
by岳華 杜
 
COSCUP: Foreign Function Call in Julia
by岳華 杜
 
Batch normalization 與他愉快的小伙伴
by岳華 杜
 
20171014 tips for manipulating filesystem in julia
by岳華 杜
 
20170807 julia的簡單而高效資料處理
by岳華 杜
 

Recently uploaded

PDF
What is Voice User Interface (VUI) Definition & Examples.pdf
byDesign Studio UI UX
 
PPTX
Scrape YouTube Video Data for Influencer Analytics.pptx
byWeb Data Crawler
 
PDF
How Application Performance Monitoring Tools Are Used in Performance Testing
byhenrycavill30521
 
PDF
Xemelgo - RFID Industry Predictions for 2026
byRich Rogers
 
PPTX
Search Engine Responses to Conspiratorial Search Practices AANZCA 2025 Presen...
bykkasianenko
 
PDF
Small Business Automation: A Comprehensive Cost and ROI Guide
byAmelia Swank
 
PDF
Digital Marketing Trends in 2026: AI, Data, Content & Future Growth
byConacent Consulting
 
PDF
Techbrains Baku 2025 by GoUP - all speaker session
byHusseinMalikMammadli
 
PDF
Artificial Intelligence and Barbarism - Conceptual Map
byalfadix
 
PDF
“Lessons from Yesterday's Tomorrowland” by Scott M. Graffius
byScott M. Graffius
 
PDF
How Azure DevOps Consultants Dubai Reduce Release Delays.pdf
byLogicEra
 
PDF
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
PPTX
UNIT III TYPOLOGY OF CRIME AND CRIMINAL BEHAVIOUR (1).pptx
bybloodyhumanoid
 
PDF
AI Driven & AI Native Development AI native development
byZainKarim7
 
PDF
Regenerative Agriculture Finance : Environmental impact
byrose mason
 
PPTX
Compare and contrast types of attacks.pptx
bysyednaqihassan14
 
PPTX
Cybersecurity Basics: Understanding Threats, Protection Methods, and Safe Dig...
byDhruvManiar3
 
PPT
Carole BirdCarole BirdCarole BirdCarole Bird.ppt
byk4n3kictf
 
PPTX
Salesforce Spring 26 Release Key .pptx
byMehediHasan939830
 
PPT
Information and Communication Technologies and Power
byJeffrey Hart
 
What is Voice User Interface (VUI) Definition & Examples.pdf
byDesign Studio UI UX
 
Scrape YouTube Video Data for Influencer Analytics.pptx
byWeb Data Crawler
 
How Application Performance Monitoring Tools Are Used in Performance Testing
byhenrycavill30521
 
Xemelgo - RFID Industry Predictions for 2026
byRich Rogers
 
Search Engine Responses to Conspiratorial Search Practices AANZCA 2025 Presen...
bykkasianenko
 
Small Business Automation: A Comprehensive Cost and ROI Guide
byAmelia Swank
 
Digital Marketing Trends in 2026: AI, Data, Content & Future Growth
byConacent Consulting
 
Techbrains Baku 2025 by GoUP - all speaker session
byHusseinMalikMammadli
 
Artificial Intelligence and Barbarism - Conceptual Map
byalfadix
 
“Lessons from Yesterday's Tomorrowland” by Scott M. Graffius
byScott M. Graffius
 
How Azure DevOps Consultants Dubai Reduce Release Delays.pdf
byLogicEra
 
A Brief Introduction About Christopher Elwell Woburn
byChristopher Elwell Woburn
 
UNIT III TYPOLOGY OF CRIME AND CRIMINAL BEHAVIOUR (1).pptx
bybloodyhumanoid
 
AI Driven & AI Native Development AI native development
byZainKarim7
 
Regenerative Agriculture Finance : Environmental impact
byrose mason
 
Compare and contrast types of attacks.pptx
bysyednaqihassan14
 
Cybersecurity Basics: Understanding Threats, Protection Methods, and Safe Dig...
byDhruvManiar3
 
Carole BirdCarole BirdCarole BirdCarole Bird.ppt
byk4n3kictf
 
Salesforce Spring 26 Release Key .pptx
byMehediHasan939830
 
Information and Communication Technologies and Power
byJeffrey Hart
 

從 VAE 走向深度學習新理論

  • 1.
  • 2.
    Deep Learning isa kind of Representational LearningDeep Learning is a kind of Representational Learning
  • 3.
    Deep Learning isa kind of Representational LearningDeep Learning is a kind of Representational Learning picture source (https://www.deeplearningbook.org/)
  • 4.
  • 5.
    Representational LearningRepresentational Learning woman RepresentationLearning: A Review and New Perspectives (https://arxiv.org/abs/1206.5538)
  • 6.
  • 7.
    Restricted Boltzmann MachinesRestrictedBoltzmann Machines An unsupervised greedy way to extract featuresAn unsupervised greedy way to extract features 發明：發明： Smolensky, Paul (1986). Chapter 6: Information Processing in Dynamical Systems: Foundations of Harmony Theory. 應⽤：應⽤： 降維：Hinton, G. E.; Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science. 分類：Larochelle, H.; Bengio, Y. (2008). Classi cation using discriminative restricted Boltzmann machines. ICML '08. 協同過濾：Salakhutdinov, R.; Mnih, A.; Hinton, G. (2007). Restricted Boltzmann machines for collaborative ltering. ICML '07. 特徵學習：Coates, Adam; Lee, Honglak; Ng, Andrew Y. (2011). An analysis of single-layer networks in unsupervised feature learning. International Conference on Arti cial Intelligence and Statistics (AISTATS).
  • 8.
    Restricted Boltzmann MachinesRestrictedBoltzmann Machines x2 x3 x4 x5 z2 z1 x1 A Beginner's Guide to Restricted Boltzmann Machines (RBMs) (https://skymind.ai/wiki/restricted-boltzmann-machine)
  • 9.
    Deep Belief Network[Hinton]Deep Belief Network [Hinton] A greedy layerwise unsupervised pre-training methodA greedy layerwise unsupervised pre-training method W1 W1 W2
  • 10.
    Deep Belief NetworkDeepBelief Network W1 W2 W1 T W2 T
  • 11.
    We need generativemodel!We need generative model! Discriminative model: Generative model: p(Y |X) p(X, Y )
  • 12.
    Disentangle explanatory generativefactorsDisentangle explanatory generative factors to disentangle as many factors as possible, discarding as little information about the data as is practical x2 x3 x4 x5 z2 z1 x1 x2 x3 x4 x5 x1 z1 z2
  • 13.
  • 14.
    A generative modelAgenerative model z x N We hope to learn generative factors by unsupervised method
  • 15.
  • 17.
    The factorThe factor xy N y=θ0+θ1 x θ θ=(θ0, θ1)
  • 18.
    To learn latentrandom variablesTo learn latent random variables z x N θ
  • 19.
    Introduce Bayesian theoremIntroduceBayesian theorem (z|x) =pθ (x|z) (z)pθ pθ (x)pθ (x) = ∫ (x|z) (z)dzpθ pθ pθ is intractable.(x)pθ Variational inference: useVariational inference: use to approximateto approximate(z|x)qϕ (z|x)pθ
  • 20.
    Kullback–Leibler divergenceKullback–Leibler divergence Relativeentropy, to measure the dissimilarity between two distributions. Use data to approximate theoretical distributionp(X) q(X) (p(X)||q(X)) = − p( ) log DKL ∑ i xi q( )xi p( )xi 1. Asymmetry 2. Not distance 3. 4. and are equal (p(X)||q(X)) ≥ 0DKL (p(X)||q(X)) = 0DKL ⇔ p(X) q(X)
  • 21.
    FormulationFormulation (z|x) =pθ (x|z) (z)pθ pθ (x)pθ arg( (z|x)|| (z|x))min ϕ DKL qϕ pθ x zφ θ N x z θ N
  • 22.
  • 23.
    Evidence Lower Boundmethod (ELOB)Evidence Lower Bound method (ELOB) ( (z|x)|| (z|x))DKL qϕ pθ = ∫ q(z|x) log  dz q(z|x) p(z|x) = ∫ q(z|x) log  dz q(z|x)p(x) p(x, z) = ∫ q(z|x) log  dz + ∫ q(z|x) log p(x)dz q(z|x) p(x, z) = ∫ q(z|x)(log q(z|x) − log p(x, z))dz + log p(x) = − [log p(x, z) − log q(z|x)] + log p(x)Eq(z|x)
  • 24.
    Evidence Lower Boundmethod (ELOB)Evidence Lower Bound method (ELOB) Let is called (variational) lower bound or evidence lower bound. L(θ, ϕ, x) = [log  (x, z) − log  (z|x)]E (z|x)qϕ pθ qϕ ( (z|x)|| (z|x)) = −L(θ, ϕ, x) + log p(x)DKL qϕ pθ log p(x) = ( (z|x)|| (z|x)) ↙ +L(θ, ϕ, x) ↗DKL qϕ pθ
  • 25.
    Evidence Lower Boundmethod (ELOB)Evidence Lower Bound method (ELOB) Encoder: , Decoder: (z|x) =pθ (x|z) (z)pθ pθ (x)pθ arg ( (z|x)|| (z|x))min ϕ DKL qϕ pθ ⇓ (z|x) =pθ (x|z) (z)pθ pθ (x)pθ arg L(θ, ϕ, x)max θ,ϕ (z|x)qϕ (x|z)pθ
  • 26.
    Hypothesis: gaussian mixtureas latent representationHypothesis: gaussian mixture as latent representation z2 z1 μz2 μz1 σz 2 σz1 z2 z2 z1 z1
  • 27.
    Encoder and decoderEncoderand decoder z2 z2 z1 z1 encoderencoder decoder
  • 28.
    How to solve?Howto solve? Mean eld variational approximation Sampling by Markov chain Monte Carlo More?
  • 29.
    Sampling by MCMCSamplingby MCMC picture source (https://www.youtube.com/watch? v=OTO1DygELpY)
  • 30.
    Stochastic gradient descent?Stochasticgradient descent? L(θ, ϕ, x) = [log  (x, z) − log  (z|x)]E (z|x)qϕ pθ qϕ L(θ, ϕ, x) = [−log  (z|x)]∇ϕ ∇ϕ E (z|x)qϕ qϕ
  • 31.
    Reparameterization trickReparameterization trick Encoder () Decoder ( ) Sample from Encoder ( ) Decoder ( ) Sample from * + Tutorial on Variational Autoencoders (https://arxiv.org/abs/1606.05908)
  • 32.
    Stochastic gradient variationalbayes (SGVB)Stochastic gradient variational bayes (SGVB) ⾒Algorithm 1 in Auto-Encoding Variational Bayes⾒Algorithm 1 in Auto-Encoding Variational Bayes
  • 33.
    Example: variational autoencoderExample:variational autoencoder z2 z2 z1 z1 encoderencoder decoder
  • 34.
    ExperimentsExperiments (a) Learned FreyFace manifold (b) Learned MNIST manifold
  • 35.
  • 36.
    Achieve disentangled explainablegenerative factorAchieve disentangled explainable generative factor
  • 37.
    Achieve disentangled explainablegenerative factorAchieve disentangled explainable generative factor
  • 38.
    Achieve disentangled explainablegenerative factorAchieve disentangled explainable generative factor Figure 6 in β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK
  • 39.
    What is thedi erence between VAE andWhat is the di erence between VAE and -VAE?-VAE?β VAE: -VAE: arg max L(θ, ϕ, x) = [log  (x|z)] − ( (z|x)|| (z))E (z|x)qϕ pθ DKL qϕ pθ β arg max L(θ, ϕ, x) = [log  (x|z)] − β ( (z|x)|| (z))E (z|x)qϕ pθ DKL qϕ pθ L(θ, ϕ, x) = [log  (x, z) − log  (z|x)]E (z|x)qϕ pθ qϕ = ∫ (z|x)(log  (x, z) − log  (z|x))dzqϕ pθ qϕ = ∫ (z|x)(log  − log  )dzqϕ (x, z)pθ (z)pθ (z|x)qϕ (z)pθ = [log  (x|z)] − ( (z|x)|| (z))E (z|x)qϕ pθ DKL qϕ pθ
  • 40.
    Why?Why? The higher encourageslearning a disentangled representation. : encourage to learn good representations. : constraint the capacity of β [log  (x|z)]E (z|x)qϕ pθ ( (z|x)|| (z))DKL qϕ pθ z
  • 41.
    The information bottleneckmethodThe information bottleneck method arg max I (Z; Y ) − βI (X; Z) : maximize mutual information between Z and Y. : discard irrelevant information about Y from X. I (Z; Y ) I (X; Z) Learning is about forgetting irrelevant details.Learning is about forgetting irrelevant details.
  • 42.
    ExperimentsExperiments Understanding disentangling inβ-VAE (https://arxiv.org/abs/1804.03599)
  • 43.
  • 44.
    Basic Information theoryBasicInformation theory EntropyEntropy Information entropy, Shannon entropy Measure the uncertainty of an event. H(X) = E(I (X)) = − p( ) log p( )∑ i=1 n xi xi 1. Nonnegativity: 2. Symmetry: 3. If and are independent random variable: H(X) ≥ 0 H(X, Y ) = H(Y , X) X Y H(X|Y ) = H(X)
  • 45.
    EntropyEntropy 天氣預報100% 下⾬，0% 晴天： 天氣預報80%下⾬，20% 晴天： 天氣預報50% 下⾬，50% 晴天： 1 lo  1 + 0 lo  0 = 0 + 0 = 0g2 g2 −0.8 lo  0.8 − 0.2 lo  0.2 = 0.258 + 0.464 = 0.722g2 g2 −0.5 lo 0.5 − 0.5 lo 0.5 = 0.5 + 0.5 = 1g2 g2
  • 46.
    EntropyEntropy 0 0.5 10 0.5 1 Pr(X=1) H(X) ) picturesource (https://en.wikipedia.org/wiki/Entropy_(information_theory)
  • 47.
    Conditional entropyConditional entropy Tomeasure how much information needed to describe the outcome of a random variable Y given that the value of another random variable X is known. H(Y |X) = p(x)H(Y |X = x)∑ x∈X = − p(x) p(y|x) log p(y|x)∑ x∈X ∑ y∈Y = − p(x, y) log ∑ x∈X ,y∈Y p(x, y) p(x)
  • 48.
    Mutual informationMutual information Tomeasure how much information obtained about one random variable through observing the other. I (X; Y ) = H(X) − H(X|Y ) = H(Y ) − H(Y |X) = H(X) + H(Y ) − H(X, Y ) = p(x, y) log ∑ x,y p(x, y) p(x)p(y) 1. Nonnegativity: 2. Symmetry: I (X; Y ) ≥ 0 I (X; Y ) = I (Y ; X)
  • 49.
    Relation to Kullback–LeiblerdivergenceRelation to Kullback–Leibler divergence I (X; Y ) = (p(X, Y )||p(X)p(Y ))DKL
  • 50.
  • 51.
  • 52.
    Cross entropyCross entropy Howmuch difference between two distributions. H(q, p) = H(q) + (q||p)DKL = − p(x) log q(x)∑ x DKL(q∣p) H (q) H (q, p) NOTION: notation confused with joint entropy.
  • 53.
    Di erence betweenmutual information and cross entropyDi erence between mutual information and cross entropy Mutual information Measure the information share between two random variables. Cross entropy Measure the difference between two distributions.
  • 54.
    Data processing inequality(DPI)Data processing inequality (DPI) Let be a Markov chain, thenX → Y → Z I (X; Y ) ≥ I (X; Z)
  • 55.
    The neural networkgenerates a successive Markov chainThe neural network generates a successive Markov chain Treat the whole layer as a single random variableTi Encoder Decoder I (X; Y ) ≥ I ( ; Y ) ≥ I ( ; Y ) ≥. . . ≥ I ( ; Y ) ≥ I ( ; Y )T1 T2 Tm Y^ H(X) ≥ I (X; ) ≥ I (X; ) ≥. . . ≥ I (X; ) ≥ I (X; )T1 T2 Tm Y^
  • 56.
    Codebook and volumeCodebookand volume Let : signal source with xed probability measure : quantized codebook : a soft partition of , with probability with X p(x) X^ p( |x)x^ X p( ) = p(x)p( |x)x^ ∑ x x^
  • 57.
    What determines thequality of a quantization?What determines the quality of a quantization? Rate, the average numbers of bits per message to encode the signal. The information to transmit from to is bounded from belowX X^ I (X; )X^
  • 58.
    Rate distortion theoryRatedistortion theory Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 1 Lossy compression n Lower the bit-rate R by allowing some acceptable distortion D of the signal. Distortion D Rate R Lossless coding D=0
  • 59.
    Rate distortion theoryRatedistortion theory Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 2 Types of lossy compression problems D R n Given maximum rate R, minimize distortion D n Given distortion D, minimize rate R D R Equivalent constrained optimization problems, often unwieldy due to constraint.
  • 60.
    Rate distortion theoryRatedistortion theory Def. rate distortion function as R(D) = min I (X; )X^ w. r. t. E[d(x, )] ≤ Dx^ Apply Lagrange multiplier: F (p( |x)) = I (X; ) + βE[d(x, )]x^ X^ x^
  • 61.
    Information bottleneck methodInformationbottleneck method , thenX → → YX^ I (X; ) ≥ I (X; Y )X^ Information bottleneck: arg min L(x, ) = I (X; ) − βI ( ; Y )x^ X^ X^ We want this quantization to capture as much information about tradeoff between compress the representation and preserve meaningful information. Y
  • 62.
    Information bottleneck methodInformationbottleneck method x2 x3 x4 x5 z4 z3 z2 z1 x1 x2 x3 x4 x5 x1
  • 63.
    Opening the blackbox of Deep Neural Networks viaOpening the black box of Deep Neural Networks via InformationInformation
  • 64.
    IssuesIssues 1. The SGDlayer dynamics in the Information plane. 2. The effect of the training sample size on the layers. 3. What is the bene t of the hidden layers? 4. What is the nal location of the hidden layers? 5. Do the hidden layers form optimal IB representations?
  • 65.
    SetupSetup standard DNN settings tanhas activation function sigmoid function in the nal layer train with SGD and cross-entropy loss 7 fully connected hidden layers with widths: 12-10-7-5-4-3-2 neurons
  • 66.
    Information planeInformation plane EncoderDecoder Given , plot point on the information plane. Applied to the Markov chain of a k-layers of DNN, connected points form a unique information path. P (X; Y ) (I (X; T ), I (T ; Y ))
  • 67.
    The dynamics ofthe training by Stochastic-Gradient-DecentThe dynamics of the training by Stochastic-Gradient-Decent 50 different randomized initializations with different randomized training samples init − 400epochs − 9000epochs The optimization process in the Information Plane (https://www.youtube.com/watch? v=P1A1yNsxMjc)
  • 68.
    The two optimizationphases in the Information PlaneThe two optimization phases in the Information Plane 5% - 45% - 85% training samples5% - 45% - 85% training samples Emperical risk minimization (ERM) phase (fast) increase layer learn the information while preserving the DPI order Representation compression phase (slow) decrease until convergence layer lose irrelevant information (compression) IY IX
  • 69.
    The drift anddi usion phases of SGD optimizationThe drift and di usion phases of SGD optimization Layer weight's gradient distributionsLayer weight's gradient distributions
  • 70.
    The drift anddi usion phases of SGD optimizationThe drift and di usion phases of SGD optimization Drift phase large gradient mean, small variance (high SNR) increase and reduce the emperical error ERM phase Diffusion phase small gradient mean, large uctuations (low SNR) the gradients behave like Gaussian noise, weights evolve like Wiener process compression phase Maximize the entropy of the weight distribution by addiing noise, known as stochastic relaxation compression by diffusion phase attempts to interpret single weights or even single neurons in such networks can be meaningless IY
  • 71.
    The computational benet of the hidden layersThe computational bene t of the hidden layers Train 6 different architecture with 1-6 hidden layers
  • 72.
    The computational benet of the hidden layersThe computational bene t of the hidden layers 1. Adding hidden layers dramatically reduces the number of training epochs for good generalization. 2. The compression phase of each layer is shorter when it starts from a previous compressed layer. 3. The compression is faster for the deeper (narrower and closer to the output) layers. 4. Even wide hidden layers eventually compress in the diffusion phase. Adding extra width does not help.
  • 73.
    Convergence to thelayers to the Information Bottleneck boundConvergence to the layers to the Information Bottleneck bound
  • 74.
    Evolution of thelayers with training sample sizeEvolution of the layers with training sample size 0 1 2 3 4 5 6 7 8 9 I(X;T) 0.3 0.4 0.5 0.6 0.7 I(T;Y) 4% 84% Training data
  • 75.
    with increasing trainingsize the layers’ true label information (generalization) is pushed up and gets closer to the theoretical IB bound for the rule distribution. IY
  • 76.
    Are our ndingsgeneral enough?Are our ndings general enough?
  • 77.
    Hinton 的評論Hinton 的評論 Hinton在聽完Tishby 的talk 之後，給Tishby 發了email: “I have to listen to it another 10,000 times to really understand it, but it’s very rare nowadays to hear a talk with a really original idea in it that may be the answer to a really major puzzle.”
  • 78.
    Caution!Caution! No, information bottleneck(probably) doesn’t open the “black-box” of deep neural n (https://severelytheoretical.wordpress.com/2017/09/28/no-information-bottlenec black-box-of-deep-neural-networks/) Tishby's 'Opening the Black Box of Deep Neural Networks via Information' received (https://www.reddit.com/r/MachineLearning/comments/72eau7/d_tishbys_opening On the Information Bottleneck Theory of Deep Learning [Harvard University] [ICLR (https://openreview.net/forum?id=ry_WPG-A-)
  • 79.
    Thank you forattentionThank you for attention ReferenceReference 18. Information Theory of Deep Learning. Naftali Tishby (https://www.youtube.com/watch?v=bLqJHjXihK8)