SlideShare a Scribd company logo
1 of 27
Download to read offline
Associative Memory Model について
2020/10/04
1
Referrence
1. The Capacity of Hopfield Associated Memory (1987)
2. Meta Learning Deep Energy-Based Memory Models (ICLR 2020)
3. Overparameterized Neural Networks Implement Associated Memory (2019)
4. 関連︓Identity Crisis: Memorization and Generalization under Extreme
Overparameterization (ICLR 2020)
5. 後で読みたい︓Associative Memory in Iterated Overparameterized Sigmoid
Autoencoders (ICML2020)
2
Mathmatical Preliminaries
Dynamical systems on state space
behavior of a state with transition given by
Fixed points
state such that
Attractors (≒ stable fixed points)
A fixed point is (locally) stable if any states near converge by
successively applying .
(本当は集合に対して定める, e.g. limit cycle)
Theorem
For differentiable , a fixed point of is stable Jacobian of at has
maximum (absolute) eigenvalue less than 1.
V (= R , {0, 1} ,  manifolds, etc.)n n
x ∈ V x ← F(x) F : V → V
x x = F(x)
x x x
F
F x F ⟺ F x
3
Associative Memory Model
Model to retrieve remembered patterns from distorted/incomplete version
STORE patterns as attractors of network dynamics
RETRIEVE by running dynamics
Retrieval often written as optimization procedure of Energy function
Hopfield network
(Deep) Boltzman machine
Energy based deep network (Referrence 2)
(モチベーションがよくわからないという話はある)
4
Hopfield Network
Hopfield Network consists of
Binary neurons (state vector)
Symmetric matrix (parameter)
State transition (dynamics)
x ∈ {0, 1}n
T ∈ Rn×n
x ←i sgn(Tx) =i sgn( T x )∑j ij j
sgn(0) := +1
5
Hopfield Network for Memory Model
: vectors to be stored ( should be small )
Encoding rule
Retrieval
Hopfield's Asynchronous Algorithm
a. take an initial state
b. choose randomly
c.
d. repeat 2,3
returns
{x , ⋯ , x } ⊂(1) (m)
{+1, −1}n
m < n
T = (x x −∑α=1
m (α) (α)T
I )n
i ∈ {1, ⋯ , n}
x ←i sgn(Tx)i
x
6
Hopfield Network for Memory Model
The algorithm converges, but limits are NOT necessarily s
For small , s tend to be stable attractors (w.r.t. Hamming distance)
Energy of Hopfield Network
Theorem (Hopfield)
: symmetric, diagonal , then the energy does not increase by state
transition, and asynchronous algorithm converges.
x(α)
m < n x(α)
E := − T x x∑i,j ij i j
T ≥ 0 E
7
Meta-Learning Deep Energy-Based Memory Models
S. Batrunov, J. W. Rae, S. Osindero, T. P. Lillicrap (Google Brain)
Construct memory models for more complex data (e.g. images)
Represent higher order dependency in real-world data
Need compressive (≒expressive) & fast writing rule with energy
Use deep networks
Apply gradient-based meta-learning methods (Finn et al., 2017)
8
Energy-Based Memory Models
9
Energy-Based Memory Models
Parametric model differentiable in both
Aims to compress patterns into parameters so that each
becomes a local minimum of
Retrieve from distorted by calling (energy-minimization)
Practically quantified by reconstruction error:
(expection taken over distortion ?)
E(x; θ) x, θ
X = {x , ⋯ , x }1 N θ
xi E(x; θ)
xi x~i read( ; θ)x~i
x ↦ x~
10
Meta-Learning Gradient-Based Writing Rules
Naive EBMM requires many iterations for to converge (i.e. writing is slow)...
Want to find good initial parameter for fast optimization
Hard to evaluate and differentiate expectation over distortion...
Introduce writing loss
Including only 1st-order information (w/o Hessian) is empirically sufficient
Limits deviation from initial is empirically helpful
Define (explicit) writing rule
(continued...)
θ
θˉ
θˉ
write
11
Meta-Learning Gradient-Based Writing Rules
Hard to evaluate and differentiate expectation over distortion...
(...continued)
meta-learn by
where
(remark : need access to whole dataset, not only one set to store)
Use (number of iteration for write/read) in the experiment
r = ({γ }, {η }), τ =(k) (t)
(α, β)
X
K = T = 5
12
Experiments : Retrieval for real-world image
Baseline
LSTM (failed)
Hopfield networks (failed)
Memory-Augmented Networks (Santoro et al., 2016)
Memory Networks (Wetson et al., 2014)
Differentiable Plasticity model (Miconi et al., 2018)
Dynamic Kanerva Machine (Wu et al., 2018)
Datasets
Omniglot characters
CIFAR-10
ImageNet 64x64 13
Experiments : Retrieval for real-world image
Procedure (varying memory size)
Write a fixed-sized batch of images
Form queries by corrupting a random block of the images
Retrieve the original image.
Use FC (only for Omniglot) or Conv in 3-block ResNet for proposed model.
Energy is computed as a linear combination of units in the last layer.
14
Results
MemNet, EBMMはResNetでidentityを学習しやすくなって簡単になる
EBMM can detect the distorted part (why??)
15
Results
16
Results
17
Results
perceptual lossで改善が⾒込めるか︖
18
Result for storing random bit sequence of length 128
19
Overparameterized Neural Networks Implement
Associative Memory
A.Radhakrishnan (MIT), M.Belkin (Ohio State Univ.), C.Uhler (MIT)
Empirically show:
Overparameterized autoencoders has associated memory as attractors (w/o
explicit energy!)
Efficient sequence encoding with the same mechanism
ICLR 2020 reject
Not convincing for applicability to classifier or more general models
⾯⽩いが、インパクトや⽴ち位置が不⼗分。もうちょっと結果が欲しい
20
Dynamics defined by autoencoder
Autoencoder can be iterated
Hence define a dynamical system on the data space.
Sequence encoder can be trained by modifying the MSE loss:
L = ∣∣f(x ) −(i)
x ∣∣(i+1 mod n) 2
Sequential counterpart of stable fixed points are called a limit cycle
In this paper, the authors analyze
the dynamics defined by AEs trained to achieve MSE
varying activation / optimizer / initialization / depth and width
Remark : Reference 4 analyzes AEs with 1 training datum focusing on architectures,
but not on dynamics.
f : R →d
Rd
< 10−8
21
Retrieval via iteration
Spurious (i.e. out of stored data) attractors sometimes appear (depending on
dataset & optimization).
22
23
Impact of optimizers and activation functions
24
Analysis for Convolutional Networks
25
Impact of depth/width
26
Efficiency of Sequence Encoder
27

More Related Content

What's hot

proposal_pura
proposal_puraproposal_pura
proposal_puraErick Lin
 
Fast Unbalanced Optimal Transport on a Tree
Fast Unbalanced Optimal Transport on a TreeFast Unbalanced Optimal Transport on a Tree
Fast Unbalanced Optimal Transport on a Treejoisino
 
Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...
Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...
Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...ijtsrd
 
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
An introduction to deep learning
An introduction to deep learningAn introduction to deep learning
An introduction to deep learningVan Thanh
 
A brief introduction to recent segmentation methods
A brief introduction to recent segmentation methodsA brief introduction to recent segmentation methods
A brief introduction to recent segmentation methodsShunta Saito
 
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Universitat Politècnica de Catalunya
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronMostafa G. M. Mostafa
 
Extremely Low Bit Transformer Quantization for On-Device NMT
Extremely Low Bit Transformer Quantization for On-Device NMTExtremely Low Bit Transformer Quantization for On-Device NMT
Extremely Low Bit Transformer Quantization for On-Device NMTInsoo Chung
 
Neural Networks Ver1
Neural  Networks  Ver1Neural  Networks  Ver1
Neural Networks Ver1ncct
 
Introduction to Applied Machine Learning
Introduction to Applied Machine LearningIntroduction to Applied Machine Learning
Introduction to Applied Machine LearningSheilaJimenezMorejon
 
Deep learning lecture - part 1 (basics, CNN)
Deep learning lecture - part 1 (basics, CNN)Deep learning lecture - part 1 (basics, CNN)
Deep learning lecture - part 1 (basics, CNN)SungminYou
 
An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...
An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...
An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...ijtsrd
 
VIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape EstimationVIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape EstimationArithmer Inc.
 
Image compression and reconstruction using a new approach by artificial neura...
Image compression and reconstruction using a new approach by artificial neura...Image compression and reconstruction using a new approach by artificial neura...
Image compression and reconstruction using a new approach by artificial neura...Hưng Đặng
 
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 14
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 14Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 14
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 14Dr. Muhammad Ali Tirmizi., Ph.D.
 
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 13
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 13Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 13
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 13Dr. Muhammad Ali Tirmizi., Ph.D.
 
NEURAL NETWORKS
NEURAL NETWORKSNEURAL NETWORKS
NEURAL NETWORKSESCOM
 
A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...
A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...
A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...ijdms
 

What's hot (20)

proposal_pura
proposal_puraproposal_pura
proposal_pura
 
Fast Unbalanced Optimal Transport on a Tree
Fast Unbalanced Optimal Transport on a TreeFast Unbalanced Optimal Transport on a Tree
Fast Unbalanced Optimal Transport on a Tree
 
Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...
Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...
Mobile Network Coverage Determination at 900MHz for Abuja Rural Areas using A...
 
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
Unsupervised Learning (D2L6 2017 UPC Deep Learning for Computer Vision)
 
An introduction to deep learning
An introduction to deep learningAn introduction to deep learning
An introduction to deep learning
 
A brief introduction to recent segmentation methods
A brief introduction to recent segmentation methodsA brief introduction to recent segmentation methods
A brief introduction to recent segmentation methods
 
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
Convolutional Neural Networks (D1L3 2017 UPC Deep Learning for Computer Vision)
 
Neural networks
Neural networksNeural networks
Neural networks
 
Neural Networks: Multilayer Perceptron
Neural Networks: Multilayer PerceptronNeural Networks: Multilayer Perceptron
Neural Networks: Multilayer Perceptron
 
Extremely Low Bit Transformer Quantization for On-Device NMT
Extremely Low Bit Transformer Quantization for On-Device NMTExtremely Low Bit Transformer Quantization for On-Device NMT
Extremely Low Bit Transformer Quantization for On-Device NMT
 
Neural Networks Ver1
Neural  Networks  Ver1Neural  Networks  Ver1
Neural Networks Ver1
 
Introduction to Applied Machine Learning
Introduction to Applied Machine LearningIntroduction to Applied Machine Learning
Introduction to Applied Machine Learning
 
Deep learning lecture - part 1 (basics, CNN)
Deep learning lecture - part 1 (basics, CNN)Deep learning lecture - part 1 (basics, CNN)
Deep learning lecture - part 1 (basics, CNN)
 
An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...
An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...
An Artificial Intelligence Approach to Ultra High Frequency Path Loss Modelli...
 
VIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape EstimationVIBE: Video Inference for Human Body Pose and Shape Estimation
VIBE: Video Inference for Human Body Pose and Shape Estimation
 
Image compression and reconstruction using a new approach by artificial neura...
Image compression and reconstruction using a new approach by artificial neura...Image compression and reconstruction using a new approach by artificial neura...
Image compression and reconstruction using a new approach by artificial neura...
 
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 14
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 14Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 14
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 14
 
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 13
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 13Dr. Syed Muhammad Ali Tirmizi - Special topics in finance   lec 13
Dr. Syed Muhammad Ali Tirmizi - Special topics in finance lec 13
 
NEURAL NETWORKS
NEURAL NETWORKSNEURAL NETWORKS
NEURAL NETWORKS
 
A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...
A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...
A SURVEY OF SPIKING NEURAL NETWORKS AND SUPPORT VECTOR MACHINE PERFORMANCE BY...
 

Similar to Associative Memory Models for Storing and Retrieving Images

Machine learning in science and industry — day 4
Machine learning in science and industry — day 4Machine learning in science and industry — day 4
Machine learning in science and industry — day 4arogozhnikov
 
Recent advances of AI for medical imaging : Engineering perspectives
Recent advances of AI for medical imaging : Engineering perspectivesRecent advances of AI for medical imaging : Engineering perspectives
Recent advances of AI for medical imaging : Engineering perspectivesNamkug Kim
 
NIPS2007: deep belief nets
NIPS2007: deep belief netsNIPS2007: deep belief nets
NIPS2007: deep belief netszukun
 
Deep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and HypeDeep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and HypeSiby Jose Plathottam
 
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...IOSR Journals
 
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...Chris Rackauckas
 
EchoBay: optimization of Echo State Networks under memory and time constraints
EchoBay: optimization of Echo State Networks under memory and time constraintsEchoBay: optimization of Echo State Networks under memory and time constraints
EchoBay: optimization of Echo State Networks under memory and time constraintsNECST Lab @ Politecnico di Milano
 
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...Numenta
 
Implementing a neural network potential for exascale molecular dynamics
Implementing a neural network potential for exascale molecular dynamicsImplementing a neural network potential for exascale molecular dynamics
Implementing a neural network potential for exascale molecular dynamicsPFHub PFHub
 
Convolutional auto-encoded extreme learning machine for incremental learning ...
Convolutional auto-encoded extreme learning machine for incremental learning ...Convolutional auto-encoded extreme learning machine for incremental learning ...
Convolutional auto-encoded extreme learning machine for incremental learning ...IJECEIAES
 
A simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsA simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsDevansh16
 
Lecture 7: Recurrent Neural Networks
Lecture 7: Recurrent Neural NetworksLecture 7: Recurrent Neural Networks
Lecture 7: Recurrent Neural NetworksSang Jun Lee
 
Citython presentation
Citython presentationCitython presentation
Citython presentationAnkit Tewari
 
Reservoir computing fast deep learning for sequences
Reservoir computing   fast deep learning for sequencesReservoir computing   fast deep learning for sequences
Reservoir computing fast deep learning for sequencesClaudio Gallicchio
 
Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...IJECEIAES
 
Reading group nfm - 20170312
Reading group  nfm - 20170312Reading group  nfm - 20170312
Reading group nfm - 20170312Shuai Zhang
 
Big Data Intelligence: from Correlation Discovery to Causal Reasoning
Big Data Intelligence: from Correlation Discovery to Causal Reasoning Big Data Intelligence: from Correlation Discovery to Causal Reasoning
Big Data Intelligence: from Correlation Discovery to Causal Reasoning Wanjin Yu
 
Tutorial on Markov Random Fields (MRFs) for Computer Vision Applications
Tutorial on Markov Random Fields (MRFs) for Computer Vision ApplicationsTutorial on Markov Random Fields (MRFs) for Computer Vision Applications
Tutorial on Markov Random Fields (MRFs) for Computer Vision ApplicationsAnmol Dwivedi
 

Similar to Associative Memory Models for Storing and Retrieving Images (20)

Machine learning in science and industry — day 4
Machine learning in science and industry — day 4Machine learning in science and industry — day 4
Machine learning in science and industry — day 4
 
Recent advances of AI for medical imaging : Engineering perspectives
Recent advances of AI for medical imaging : Engineering perspectivesRecent advances of AI for medical imaging : Engineering perspectives
Recent advances of AI for medical imaging : Engineering perspectives
 
NIPS2007: deep belief nets
NIPS2007: deep belief netsNIPS2007: deep belief nets
NIPS2007: deep belief nets
 
Deep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and HypeDeep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and Hype
 
E04423133
E04423133E04423133
E04423133
 
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
 
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
Generalizing Scientific Machine Learning and Differentiable Simulation Beyond...
 
EchoBay: optimization of Echo State Networks under memory and time constraints
EchoBay: optimization of Echo State Networks under memory and time constraintsEchoBay: optimization of Echo State Networks under memory and time constraints
EchoBay: optimization of Echo State Networks under memory and time constraints
 
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...
 
Implementing a neural network potential for exascale molecular dynamics
Implementing a neural network potential for exascale molecular dynamicsImplementing a neural network potential for exascale molecular dynamics
Implementing a neural network potential for exascale molecular dynamics
 
G013124354
G013124354G013124354
G013124354
 
Convolutional auto-encoded extreme learning machine for incremental learning ...
Convolutional auto-encoded extreme learning machine for incremental learning ...Convolutional auto-encoded extreme learning machine for incremental learning ...
Convolutional auto-encoded extreme learning machine for incremental learning ...
 
A simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representationsA simple framework for contrastive learning of visual representations
A simple framework for contrastive learning of visual representations
 
Lecture 7: Recurrent Neural Networks
Lecture 7: Recurrent Neural NetworksLecture 7: Recurrent Neural Networks
Lecture 7: Recurrent Neural Networks
 
Citython presentation
Citython presentationCitython presentation
Citython presentation
 
Reservoir computing fast deep learning for sequences
Reservoir computing   fast deep learning for sequencesReservoir computing   fast deep learning for sequences
Reservoir computing fast deep learning for sequences
 
Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...Background Estimation Using Principal Component Analysis Based on Limited Mem...
Background Estimation Using Principal Component Analysis Based on Limited Mem...
 
Reading group nfm - 20170312
Reading group  nfm - 20170312Reading group  nfm - 20170312
Reading group nfm - 20170312
 
Big Data Intelligence: from Correlation Discovery to Causal Reasoning
Big Data Intelligence: from Correlation Discovery to Causal Reasoning Big Data Intelligence: from Correlation Discovery to Causal Reasoning
Big Data Intelligence: from Correlation Discovery to Causal Reasoning
 
Tutorial on Markov Random Fields (MRFs) for Computer Vision Applications
Tutorial on Markov Random Fields (MRFs) for Computer Vision ApplicationsTutorial on Markov Random Fields (MRFs) for Computer Vision Applications
Tutorial on Markov Random Fields (MRFs) for Computer Vision Applications
 

More from ohken

Qiskit Advocate 自己紹介
Qiskit Advocate 自己紹介Qiskit Advocate 自己紹介
Qiskit Advocate 自己紹介ohken
 
Sliced Wasserstein距離と生成モデル
Sliced Wasserstein距離と生成モデルSliced Wasserstein距離と生成モデル
Sliced Wasserstein距離と生成モデルohken
 
最適輸送の計算アルゴリズムの研究動向
最適輸送の計算アルゴリズムの研究動向最適輸送の計算アルゴリズムの研究動向
最適輸送の計算アルゴリズムの研究動向ohken
 
ICLR2020読み会 Stable Rank Normalization
ICLR2020読み会 Stable Rank NormalizationICLR2020読み会 Stable Rank Normalization
ICLR2020読み会 Stable Rank Normalizationohken
 
Multivariate Time series analysis via interpretable RNNs
Multivariate Time series analysis via interpretable RNNsMultivariate Time series analysis via interpretable RNNs
Multivariate Time series analysis via interpretable RNNsohken
 
ICML 2020 最適輸送まとめ
ICML 2020 最適輸送まとめICML 2020 最適輸送まとめ
ICML 2020 最適輸送まとめohken
 

More from ohken (6)

Qiskit Advocate 自己紹介
Qiskit Advocate 自己紹介Qiskit Advocate 自己紹介
Qiskit Advocate 自己紹介
 
Sliced Wasserstein距離と生成モデル
Sliced Wasserstein距離と生成モデルSliced Wasserstein距離と生成モデル
Sliced Wasserstein距離と生成モデル
 
最適輸送の計算アルゴリズムの研究動向
最適輸送の計算アルゴリズムの研究動向最適輸送の計算アルゴリズムの研究動向
最適輸送の計算アルゴリズムの研究動向
 
ICLR2020読み会 Stable Rank Normalization
ICLR2020読み会 Stable Rank NormalizationICLR2020読み会 Stable Rank Normalization
ICLR2020読み会 Stable Rank Normalization
 
Multivariate Time series analysis via interpretable RNNs
Multivariate Time series analysis via interpretable RNNsMultivariate Time series analysis via interpretable RNNs
Multivariate Time series analysis via interpretable RNNs
 
ICML 2020 最適輸送まとめ
ICML 2020 最適輸送まとめICML 2020 最適輸送まとめ
ICML 2020 最適輸送まとめ
 

Recently uploaded

Citronella presentation SlideShare mani upadhyay
Citronella presentation SlideShare mani upadhyayCitronella presentation SlideShare mani upadhyay
Citronella presentation SlideShare mani upadhyayupadhyaymani499
 
Base editing, prime editing, Cas13 & RNA editing and organelle base editing
Base editing, prime editing, Cas13 & RNA editing and organelle base editingBase editing, prime editing, Cas13 & RNA editing and organelle base editing
Base editing, prime editing, Cas13 & RNA editing and organelle base editingNetHelix
 
Harmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms PresentationHarmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms Presentationtahreemzahra82
 
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests GlycosidesGLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests GlycosidesNandakishor Bhaurao Deshmukh
 
Quarter 4_Grade 8_Digestive System Structure and Functions
Quarter 4_Grade 8_Digestive System Structure and FunctionsQuarter 4_Grade 8_Digestive System Structure and Functions
Quarter 4_Grade 8_Digestive System Structure and FunctionsCharlene Llagas
 
Pests of soyabean_Binomics_IdentificationDr.UPR.pdf
Pests of soyabean_Binomics_IdentificationDr.UPR.pdfPests of soyabean_Binomics_IdentificationDr.UPR.pdf
Pests of soyabean_Binomics_IdentificationDr.UPR.pdfPirithiRaju
 
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...D. B. S. College Kanpur
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxRitchAndruAgustin
 
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024Jene van der Heide
 
User Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather StationUser Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather StationColumbia Weather Systems
 
CHROMATOGRAPHY PALLAVI RAWAT.pptx
CHROMATOGRAPHY  PALLAVI RAWAT.pptxCHROMATOGRAPHY  PALLAVI RAWAT.pptx
CHROMATOGRAPHY PALLAVI RAWAT.pptxpallavirawat456
 
Microphone- characteristics,carbon microphone, dynamic microphone.pptx
Microphone- characteristics,carbon microphone, dynamic microphone.pptxMicrophone- characteristics,carbon microphone, dynamic microphone.pptx
Microphone- characteristics,carbon microphone, dynamic microphone.pptxpriyankatabhane
 
FREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by naFREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by naJASISJULIANOELYNV
 
Ai in communication electronicss[1].pptx
Ai in communication electronicss[1].pptxAi in communication electronicss[1].pptx
Ai in communication electronicss[1].pptxsubscribeus100
 
《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》rnrncn29
 
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...Universidade Federal de Sergipe - UFS
 
Biological classification of plants with detail
Biological classification of plants with detailBiological classification of plants with detail
Biological classification of plants with detailhaiderbaloch3
 
Environmental Biotechnology Topic:- Microbial Biosensor
Environmental Biotechnology Topic:- Microbial BiosensorEnvironmental Biotechnology Topic:- Microbial Biosensor
Environmental Biotechnology Topic:- Microbial Biosensorsonawaneprad
 

Recently uploaded (20)

AZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTXAZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTX
 
Citronella presentation SlideShare mani upadhyay
Citronella presentation SlideShare mani upadhyayCitronella presentation SlideShare mani upadhyay
Citronella presentation SlideShare mani upadhyay
 
Base editing, prime editing, Cas13 & RNA editing and organelle base editing
Base editing, prime editing, Cas13 & RNA editing and organelle base editingBase editing, prime editing, Cas13 & RNA editing and organelle base editing
Base editing, prime editing, Cas13 & RNA editing and organelle base editing
 
Let’s Say Someone Did Drop the Bomb. Then What?
Let’s Say Someone Did Drop the Bomb. Then What?Let’s Say Someone Did Drop the Bomb. Then What?
Let’s Say Someone Did Drop the Bomb. Then What?
 
Harmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms PresentationHarmful and Useful Microorganisms Presentation
Harmful and Useful Microorganisms Presentation
 
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests GlycosidesGLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
 
Quarter 4_Grade 8_Digestive System Structure and Functions
Quarter 4_Grade 8_Digestive System Structure and FunctionsQuarter 4_Grade 8_Digestive System Structure and Functions
Quarter 4_Grade 8_Digestive System Structure and Functions
 
Pests of soyabean_Binomics_IdentificationDr.UPR.pdf
Pests of soyabean_Binomics_IdentificationDr.UPR.pdfPests of soyabean_Binomics_IdentificationDr.UPR.pdf
Pests of soyabean_Binomics_IdentificationDr.UPR.pdf
 
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
Fertilization: Sperm and the egg—collectively called the gametes—fuse togethe...
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
 
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024GenAI talk for Young at Wageningen University & Research (WUR) March 2024
GenAI talk for Young at Wageningen University & Research (WUR) March 2024
 
User Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather StationUser Guide: Capricorn FLX™ Weather Station
User Guide: Capricorn FLX™ Weather Station
 
CHROMATOGRAPHY PALLAVI RAWAT.pptx
CHROMATOGRAPHY  PALLAVI RAWAT.pptxCHROMATOGRAPHY  PALLAVI RAWAT.pptx
CHROMATOGRAPHY PALLAVI RAWAT.pptx
 
Microphone- characteristics,carbon microphone, dynamic microphone.pptx
Microphone- characteristics,carbon microphone, dynamic microphone.pptxMicrophone- characteristics,carbon microphone, dynamic microphone.pptx
Microphone- characteristics,carbon microphone, dynamic microphone.pptx
 
FREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by naFREE NURSING BUNDLE FOR NURSES.PDF by na
FREE NURSING BUNDLE FOR NURSES.PDF by na
 
Ai in communication electronicss[1].pptx
Ai in communication electronicss[1].pptxAi in communication electronicss[1].pptx
Ai in communication electronicss[1].pptx
 
《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》《Queensland毕业文凭-昆士兰大学毕业证成绩单》
《Queensland毕业文凭-昆士兰大学毕业证成绩单》
 
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
REVISTA DE BIOLOGIA E CIÊNCIAS DA TERRA ISSN 1519-5228 - Artigo_Bioterra_V24_...
 
Biological classification of plants with detail
Biological classification of plants with detailBiological classification of plants with detail
Biological classification of plants with detail
 
Environmental Biotechnology Topic:- Microbial Biosensor
Environmental Biotechnology Topic:- Microbial BiosensorEnvironmental Biotechnology Topic:- Microbial Biosensor
Environmental Biotechnology Topic:- Microbial Biosensor
 

Associative Memory Models for Storing and Retrieving Images

  • 1. Associative Memory Model について 2020/10/04 1
  • 2. Referrence 1. The Capacity of Hopfield Associated Memory (1987) 2. Meta Learning Deep Energy-Based Memory Models (ICLR 2020) 3. Overparameterized Neural Networks Implement Associated Memory (2019) 4. 関連︓Identity Crisis: Memorization and Generalization under Extreme Overparameterization (ICLR 2020) 5. 後で読みたい︓Associative Memory in Iterated Overparameterized Sigmoid Autoencoders (ICML2020) 2
  • 3. Mathmatical Preliminaries Dynamical systems on state space behavior of a state with transition given by Fixed points state such that Attractors (≒ stable fixed points) A fixed point is (locally) stable if any states near converge by successively applying . (本当は集合に対して定める, e.g. limit cycle) Theorem For differentiable , a fixed point of is stable Jacobian of at has maximum (absolute) eigenvalue less than 1. V (= R , {0, 1} ,  manifolds, etc.)n n x ∈ V x ← F(x) F : V → V x x = F(x) x x x F F x F ⟺ F x 3
  • 4. Associative Memory Model Model to retrieve remembered patterns from distorted/incomplete version STORE patterns as attractors of network dynamics RETRIEVE by running dynamics Retrieval often written as optimization procedure of Energy function Hopfield network (Deep) Boltzman machine Energy based deep network (Referrence 2) (モチベーションがよくわからないという話はある) 4
  • 5. Hopfield Network Hopfield Network consists of Binary neurons (state vector) Symmetric matrix (parameter) State transition (dynamics) x ∈ {0, 1}n T ∈ Rn×n x ←i sgn(Tx) =i sgn( T x )∑j ij j sgn(0) := +1 5
  • 6. Hopfield Network for Memory Model : vectors to be stored ( should be small ) Encoding rule Retrieval Hopfield's Asynchronous Algorithm a. take an initial state b. choose randomly c. d. repeat 2,3 returns {x , ⋯ , x } ⊂(1) (m) {+1, −1}n m < n T = (x x −∑α=1 m (α) (α)T I )n i ∈ {1, ⋯ , n} x ←i sgn(Tx)i x 6
  • 7. Hopfield Network for Memory Model The algorithm converges, but limits are NOT necessarily s For small , s tend to be stable attractors (w.r.t. Hamming distance) Energy of Hopfield Network Theorem (Hopfield) : symmetric, diagonal , then the energy does not increase by state transition, and asynchronous algorithm converges. x(α) m < n x(α) E := − T x x∑i,j ij i j T ≥ 0 E 7
  • 8. Meta-Learning Deep Energy-Based Memory Models S. Batrunov, J. W. Rae, S. Osindero, T. P. Lillicrap (Google Brain) Construct memory models for more complex data (e.g. images) Represent higher order dependency in real-world data Need compressive (≒expressive) & fast writing rule with energy Use deep networks Apply gradient-based meta-learning methods (Finn et al., 2017) 8
  • 10. Energy-Based Memory Models Parametric model differentiable in both Aims to compress patterns into parameters so that each becomes a local minimum of Retrieve from distorted by calling (energy-minimization) Practically quantified by reconstruction error: (expection taken over distortion ?) E(x; θ) x, θ X = {x , ⋯ , x }1 N θ xi E(x; θ) xi x~i read( ; θ)x~i x ↦ x~ 10
  • 11. Meta-Learning Gradient-Based Writing Rules Naive EBMM requires many iterations for to converge (i.e. writing is slow)... Want to find good initial parameter for fast optimization Hard to evaluate and differentiate expectation over distortion... Introduce writing loss Including only 1st-order information (w/o Hessian) is empirically sufficient Limits deviation from initial is empirically helpful Define (explicit) writing rule (continued...) θ θˉ θˉ write 11
  • 12. Meta-Learning Gradient-Based Writing Rules Hard to evaluate and differentiate expectation over distortion... (...continued) meta-learn by where (remark : need access to whole dataset, not only one set to store) Use (number of iteration for write/read) in the experiment r = ({γ }, {η }), τ =(k) (t) (α, β) X K = T = 5 12
  • 13. Experiments : Retrieval for real-world image Baseline LSTM (failed) Hopfield networks (failed) Memory-Augmented Networks (Santoro et al., 2016) Memory Networks (Wetson et al., 2014) Differentiable Plasticity model (Miconi et al., 2018) Dynamic Kanerva Machine (Wu et al., 2018) Datasets Omniglot characters CIFAR-10 ImageNet 64x64 13
  • 14. Experiments : Retrieval for real-world image Procedure (varying memory size) Write a fixed-sized batch of images Form queries by corrupting a random block of the images Retrieve the original image. Use FC (only for Omniglot) or Conv in 3-block ResNet for proposed model. Energy is computed as a linear combination of units in the last layer. 14
  • 19. Result for storing random bit sequence of length 128 19
  • 20. Overparameterized Neural Networks Implement Associative Memory A.Radhakrishnan (MIT), M.Belkin (Ohio State Univ.), C.Uhler (MIT) Empirically show: Overparameterized autoencoders has associated memory as attractors (w/o explicit energy!) Efficient sequence encoding with the same mechanism ICLR 2020 reject Not convincing for applicability to classifier or more general models ⾯⽩いが、インパクトや⽴ち位置が不⼗分。もうちょっと結果が欲しい 20
  • 21. Dynamics defined by autoencoder Autoencoder can be iterated Hence define a dynamical system on the data space. Sequence encoder can be trained by modifying the MSE loss: L = ∣∣f(x ) −(i) x ∣∣(i+1 mod n) 2 Sequential counterpart of stable fixed points are called a limit cycle In this paper, the authors analyze the dynamics defined by AEs trained to achieve MSE varying activation / optimizer / initialization / depth and width Remark : Reference 4 analyzes AEs with 1 training datum focusing on architectures, but not on dynamics. f : R →d Rd < 10−8 21
  • 22. Retrieval via iteration Spurious (i.e. out of stored data) attractors sometimes appear (depending on dataset & optimization). 22
  • 23. 23
  • 24. Impact of optimizers and activation functions 24