Resticted bolzmann machine (RBM) is a type of a stochastic neural network. It can be used as a building block for deep learning algorithms. One of the great examples of RBM applications is unsupervised features learning for hand-written digits. Not going deep into the theory, let's look at RBM structure, and walk through the basic learning algorithm (so-called Contrastive Divergence).
5. Bolzmann machine
Stochastic neural network,
Has probability distribution defined:
Low energy states have higher probability
“Partition function”: normalization constant
(Boltzmann distribution is a probability distribution
of particles in a system over various possible states)
6. Bolzmann machine
Same form of energy as for Hopfield networks:
But, probabilistic units state update:
(Sigmoid function)
7. Bolzmann machine
v: Visible units, connected to observed data
h: Hidden units, capture dependancies and
patterns in visible variables
Interested in modeling observed data, so
marginalize over hidden variables (x for
visivle variables):
Defining “Free energy”
Allows to define
distribution
of visible states: