Arunabha Saha, profile picture
Uploaded byArunabha Saha
PDF, PPTX12,507 views

Brief Introduction to Boltzmann Machine

The document provides an overview of Boltzmann Machines and their relation to Hopfield Nets, covering their structure, learning algorithms, and applications. It discusses the importance of the global energy function and how hidden units in stochastic Hopfield nets can enhance functionality. Additionally, it addresses the challenges of local minima and introduces simulated annealing as a solution for escaping shallow minima.

Embed presentation

Download as PDF, PPTX
Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Boltzmann Machine A Brief Introduction Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Ritajit Majumdar Arunabha Saha Restricted Boltzmann Machine Reference University of Calcutta November 6, 2013 Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 1 / 44
Boltzmann Machine Ritajit Majumdar Arunabha Saha 1 Hopfield Net Outline Hopfield Net 2 Stochastic Hopfield Nets with Hidden Units Stochastic Hopfield Nets with Hidden Units Boltzmann Machine 3 Learning Algorithm for Boltzmann Machine Boltzmann Machine Applications of Boltzmann Machine 4 Learning Algorithm for Boltzmann Machine 5 Applications of Boltzmann Machine 6 Restricted Boltzmann Machine 7 Reference Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 2 / 44
Boltzmann Machine Ritajit Majumdar Arunabha Saha Hopfield Network Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Figure: Two dimensional representation of motion in state space Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 3 / 44
Boltzmann Machine Hopfield Net Ritajit Majumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Ritajit Majumdar Arunabha Saha (CU) Reference Boltzmann Machine November 6, 2013 4 / 44
Boltzmann Machine Hopfield Net Ritajit Majumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Reference John Hopfield introduced a global energy function for network with symmetric connections. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 4 / 44
Boltzmann Machine Hopfield Net Ritajit Majumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Reference John Hopfield introduced a global energy function for network with symmetric connections. 1 Each binary configuration of the whole network has an Energy. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 4 / 44
Boltzmann Machine Hopfield Net Ritajit Majumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Reference John Hopfield introduced a global energy function for network with symmetric connections. 1 Each binary configuration of the whole network has an Energy. 2 The binary threshold decision rule causes the network to settle to a minimum of this energy function. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 4 / 44
Boltzmann Machine Energy Function Ritajit Majumdar Arunabha Saha Outline The global energy is defined as - Hopfield Net E =− si bi − i si sj wij (1) i<j where bi is the bias of the i th unit, s is 0 or 1 1 depending on whether the unit is turned off or on respectively. And wij is the weight of the connection between units i and j. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference 1 For Bipolar Inputs the states will be −1 and 1 respectively. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 5 / 44
Boltzmann Machine Energy Function Ritajit Majumdar Arunabha Saha Outline The global energy is defined as - Hopfield Net E =− si bi − i si sj wij (1) i<j where bi is the bias of the i th unit, s is 0 or 1 1 depending on whether the unit is turned off or on respectively. And wij is the weight of the connection between units i and j. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference From this energy function, each unit computes locally how changing their state will affect the global energy. 1 For Bipolar Inputs the states will be −1 and 1 respectively. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 5 / 44
Boltzmann Machine Energy Function Ritajit Majumdar Arunabha Saha Outline The global energy is defined as - Hopfield Net E =− si bi − i si sj wij (1) i<j Stochastic Hopfield Nets with Hidden Units Boltzmann Machine where bi is the bias of the i th unit, s is 0 or 1 1 depending on whether the unit is turned off or on respectively. And wij is the weight of the connection between units i and j. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference From this energy function, each unit computes locally how changing their state will affect the global energy. The energy gap is defined as Ei = E (si = 0) − E (si = 1) = bi + sj wij (2) j 1 For Bipolar Inputs the states will be −1 and 1 respectively. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 5 / 44
Settling to an Energy Minima Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units The net is initially in a random state i.e., the units are on or off randomly. The binary threshold decision rule updates units one at a time in a random order. Update each unit to whichever of its two states minimizes the global energy. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Use binary threshold units, i.e., the states can be either 0 or 1. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 6 / 44
Settling to an Energy Minima Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units The net is initially in a random state i.e., the units are on or off randomly. The binary threshold decision rule updates units one at a time in a random order. Update each unit to whichever of its two states minimizes the global energy. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Use binary threshold units, i.e., the states can be either 0 or 1. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 7 / 44
Settling to an Energy Minima Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units The net is initially in a random state i.e., the units are on or off randomly. The binary threshold decision rule updates units one at a time in a random order. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Update each unit to whichever of its two states minimizes the global energy. Use binary threshold units, i.e., the states can be either 0 or 1. Ritajit Majumdar Arunabha Saha (CU) Reference -E = goodness = 4 Boltzmann Machine November 6, 2013 8 / 44
Using this type of Computation as memory Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Hopfield proposed that memories could be energy minima of a neural net. The binary threshold decision rule can be used to “clean up” incomplete or corrupted memory. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 9 / 44
Using this type of Computation as memory Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Hopfield proposed that memories could be energy minima of a neural net. The binary threshold decision rule can be used to “clean up” incomplete or corrupted memory. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Using energy minima to represent memories gives a content-addressable memory. An item can be accessed just by knowing parts of it. It is robust against hardware damage. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 9 / 44
Boltzmann Machine Ritajit Majumdar Arunabha Saha Stochastic Hopfield Nets with Hidden Units Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 10 / 44
Hopfield Net with Hidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 11 / 44
Hopfield Net with Hidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Use the net to construct interpretations of sensory input. Ritajit Majumdar Arunabha Saha (CU) Learning Algorithm for Boltzmann Machine Reference Boltzmann Machine November 6, 2013 11 / 44
Hopfield Net with Hidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Use the net to construct interpretations of sensory input. Reference The input is represented by the visible units. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 11 / 44
Hopfield Net with Hidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Use the net to construct interpretations of sensory input. Reference The input is represented by the visible units. The interpretation is represented by the states of the hidden units. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 11 / 44
Boltzmann Machine Noisy Networks Ritajit Majumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
Boltzmann Machine Noisy Networks Ritajit Majumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
Boltzmann Machine Noisy Networks Ritajit Majumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Ritajit Majumdar Arunabha Saha (CU) Restricted Boltzmann Machine Reference Boltzmann Machine November 6, 2013 12 / 44
Boltzmann Machine Noisy Networks Ritajit Majumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Restricted Boltzmann Machine Reference Start with a lot of noise to escape the energy barriers of poor local minima. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
Boltzmann Machine Noisy Networks Ritajit Majumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Restricted Boltzmann Machine Reference Start with a lot of noise to escape the energy barriers of poor local minima. Slowly reduce the noise so that the system ends up in a deep minima. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
Boltzmann Machine Noisy Networks Ritajit Majumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Restricted Boltzmann Machine Reference Start with a lot of noise to escape the energy barriers of poor local minima. Slowly reduce the noise so that the system ends up in a deep minima. This process is called “Simulated Annealing”. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
Boltzmann Machine Stochastic Binary Units Ritajit Majumdar Arunabha Saha Outline How to add noise? Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
Boltzmann Machine Stochastic Binary Units Ritajit Majumdar Arunabha Saha Outline How to add noise? 1 Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
Boltzmann Machine Stochastic Binary Units Ritajit Majumdar Arunabha Saha Outline How to add noise? 1 2 Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units The “temperature” controls the amount of noise. Learning Algorithm for Boltzmann Machine Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
Boltzmann Machine Stochastic Binary Units Ritajit Majumdar Arunabha Saha Outline How to add noise? Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units 2 The “temperature” controls the amount of noise. Learning Algorithm for Boltzmann Machine 3 Unit i then turns on with the probability given by the logistic function - Applications of Boltzmann Machine 1 Boltzmann Machine Restricted Boltzmann Machine Reference prob(si = 1) = Ritajit Majumdar Arunabha Saha (CU) 1 1 + e− Ei T Boltzmann Machine (3) November 6, 2013 13 / 44
Boltzmann Machine Stochastic Binary Units Ritajit Majumdar Arunabha Saha Outline How to add noise? Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units 2 The “temperature” controls the amount of noise. Learning Algorithm for Boltzmann Machine 3 Unit i then turns on with the probability given by the logistic function - Applications of Boltzmann Machine 1 Boltzmann Machine Restricted Boltzmann Machine Reference prob(si = 1) = T =0 T →∞ T =1 1 1 + e− Ei T (3) Deterministic (Hopfield Net) Complete Chaos Approaches Boltzmann Distribution Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 14 / 44
Boltzmann Machine Statistical Mechanics Ritajit Majumdar Arunabha Saha Consider a physical system with large number of possible states and many degrees of freedom. Let pi be the occurrence probability of state i Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine pi ≥ 0 for all i and pi = 1. i At thermal equilibrium state i occurs with probability 1 Z exp pi = Ei − kB T Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference where Ei is the energy of the system Boltzmann constant, kB = 1.38x10−23 J/K Ei exp − kB T is the Boltzmann Factor Partition function,(Zustadsumme) exp − Z= i Ritajit Majumdar Arunabha Saha (CU) Ei kB T Boltzmann Machine November 6, 2013 15 / 44
Boltzmann Machine Statistical Mechanics Ritajit Majumdar Arunabha Saha Outline Hopfield Net Gibbs Distribution Stochastic Hopfield Nets with Hidden Units 1 States of low energy have a higher occurrence probability than states of high energy. 2 As T is reduced, the probability is concentrated on a smaller subset of low-energy states. In the context of neural nets, the parameter T (which controls thermal fluctuations) represents the effect of synaptic noise. Hence kB = 1 is set and pi and Z getting the form 1 Z exp pi = i Ritajit Majumdar Arunabha Saha (CU) Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference − Ei T exp − Z= Boltzmann Machine Ei T Boltzmann Machine November 6, 2013 16 / 44
What is Boltzmann Machine? Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Boltzmann Machine Learning Algorithm for Boltzmann Machine A Hopfield Net consisting of Binary Stochastic Neuron with hidden units is called Boltzmann Machine. Applications of Boltzmann Machine Restricted Boltzmann Machine Reference A Boltzmann Machine is a network of symmetrically connected, neuron like units that make stochastic decisions about whether to be on or off. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 17 / 44
Structure of Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The stochastic neurons of Boltzmann machine are in two groups: vissible and hidden. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 18 / 44
Structure of Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The stochastic neurons of Boltzmann machine are in two groups: vissible and hidden. visible neurons provide an interface between the net and its environment. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 18 / 44
Structure of Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 19 / 44
Structure of Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. the hidden units do this (explain underlying constraints) by capturing higher-order correlations between the clamping vectors. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 19 / 44
Structure of Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. the hidden units do this (explain underlying constraints) by capturing higher-order correlations between the clamping vectors. Boltzmann machine learning may be viewed as an unsupervised learning procedure for modelling a distribution that is specified by the clamping patterns. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 19 / 44
Structure of Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. the hidden units do this (explain underlying constraints) by capturing higher-order correlations between the clamping vectors. Boltzmann machine learning may be viewed as an unsupervised learning procedure for modelling a distribution that is specified by the clamping patterns. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference the network can perform pattern completion: when a vector bearing part of the information is clamped onto a subset of the visible neurons, the network performs completion of the pattern on the remaining visible neurons (if it has learnt properly). Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 19 / 44
Boltzmann Machine Modelling Binary Data Ritajit Majumdar Arunabha Saha The objective of Boltzmann Machine is - Outline Hopfield Net Modelling Binary Data Stochastic Hopfield Nets with Hidden Units Given a training set of binary vectors, fit the model that will assign a probability to every possible binary vector. When unit i is given opportunity to update its state, it first computes its total input zi , zi = bi + sj wij Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine (4) Reference j Unit i turns on with probability prob(si = 1) = 1 1 + e −zi (5) If the units are updated sequentially in random order, the network will eventually reach a Boltzmann Distribution, also called its stationary distribution. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 20 / 44
How Boltzmann Machine Generates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline It is not a causal generative model. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 21 / 44
How Boltzmann Machine Generates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline It is not a causal generative model. Everything is defined in terms of energies of joint configurations of the visible and hidden units. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 21 / 44
How Boltzmann Machine Generates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net It is not a causal generative model. Everything is defined in terms of energies of joint configurations of the visible and hidden units. The energies of the joint configurations are related to their probabilities by two ways: −E (v,h) - either by defining the probability p(v, h) ∝ e - define the probability to be the probability of finding the network in that joint configuration after we have updated all of the stochastic binary units many times. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 21 / 44
How Boltzmann Machine Generates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net It is not a causal generative model. Everything is defined in terms of energies of joint configurations of the visible and hidden units. The energies of the joint configurations are related to their probabilities by two ways: −E (v,h) - either by defining the probability p(v, h) ∝ e - define the probability to be the probability of finding the network in that joint configuration after we have updated all of the stochastic binary units many times. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 21 / 44
Boltzmann Machine How Boltzmann Machine Generates Data Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units In Boltzmann Machine, everything is defined in terms of the energies of joint configurations of the visible (v) and hidden (h) units. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine The energies of joint configurations are related to their probabilities as p(v , h) ∝ e −E (v ,h) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Reference (6) November 6, 2013 22 / 44
Probabilities in term of Energies Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net The probability of a joint configuration over both visible and hidden units depends on the energy of that joint configuration compared with the energies of all other joint configurations. Ritajit Majumdar Arunabha Saha (CU) Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Boltzmann Machine November 6, 2013 23 / 44
Probabilities in term of Energies Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net The probability of a joint configuration over both visible and hidden units depends on the energy of that joint configuration compared with the energies of all other joint configurations. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The probability of a configuration of the visible units is the sum of the probabilities of all the joint configurations that contain it. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 23 / 44
Boltzmann Machine An Example Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Figure: Credit: Geoffrey Hinton, Neural Networks for Machine Learning Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 24 / 44
Boltzmann Machine Limitation and Solution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
Boltzmann Machine Limitation and Solution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? If there are large number of hidden units then we cannot calculate partition function, as it is exponentially many terms. We need to sample the data and to do this we can use Markov Chain Monte Carlo(MCMC). Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
Boltzmann Machine Limitation and Solution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? If there are large number of hidden units then we cannot calculate partition function, as it is exponentially many terms. We need to sample the data and to do this we can use Markov Chain Monte Carlo(MCMC). starting from a random global configuration pick units at random and allow them to stochastically update their states based on their energy gaps. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Run MCMC until it reaches it stationary distribution(thermal eqb. at temp is 1) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
Boltzmann Machine Limitation and Solution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? If there are large number of hidden units then we cannot calculate partition function, as it is exponentially many terms. We need to sample the data and to do this we can use Markov Chain Monte Carlo(MCMC). starting from a random global configuration pick units at random and allow them to stochastically update their states based on their energy gaps. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Run MCMC until it reaches it stationary distribution(thermal eqb. at temp is 1) the probability is related to its energy p(v , h) ∝ e −E (v ,h) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Boltzmann Learning Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine “A surprising feature of this rule is that it uses only locally available information. The change of weight depends only on the behaviour of the two units it connects, even though the change optimizes a global measure.” Applications of Boltzmann Machine Restricted Boltzmann Machine Reference - Ackley, Hinton 1985 Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 26 / 44
Boltzmann Machine Goal of Learning Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine is an unsupervised learning algorithm. Unlike Backpropagation Algorithm, where the training set consists of input vector and desired output, in Boltzmann Machine only the input vector is provided. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine We want to maximize the product of the probabilities the Boltzmann Machine assigns to the binary vectors in training set. Reference It is equivalent to maximizing the probability that we would obtain exactly the N training cases. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 27 / 44
Boltzmann Machine Assumptions Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The goal of Boltzmann learning is to produce a NN that categorize input patterns according to Boltzmann distribution. Two assumptions are made: Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 28 / 44
Boltzmann Machine Assumptions Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The goal of Boltzmann learning is to produce a NN that categorize input patterns according to Boltzmann distribution. Two assumptions are made: 1 Each environmental vector persists long enough for the network to reach thermal equilibrium. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 28 / 44
Boltzmann Machine Assumptions Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The goal of Boltzmann learning is to produce a NN that categorize input patterns according to Boltzmann distribution. Two assumptions are made: 1 Each environmental vector persists long enough for the network to reach thermal equilibrium. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference 2 There is no structure in the sequence in which environmental vectors are clamped to the visible units of the network. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 28 / 44
Why Learning could be difficult Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Consider a chain of hidden units with visibile units attached at two ends - Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Training: We want the two visibile units to be in opposite states. Solution: The product of all the weights must be negative. If all are positive, then turning on one unit will turn on the next unit and eventually the two visibile units will be in same state. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 29 / 44
Why Learning could be difficult Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Consider a chain of hidden units with visibile units attached at two ends - Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Difficulty: To modify w1 and w5, we need to know w3 (and the weights of other hidden units too). Because, if w3 is negative, then we need to modify w1 in a different way than what we would do if w3 is positive. So to change one weight in a right direction, we need to know all the other weights. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 30 / 44
Boltzmann Learning Algorithm Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 31 / 44
Boltzmann Learning Algorithm Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: 1 Clamping phase Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 31 / 44
Boltzmann Learning Algorithm Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: 1 Clamping phase Restricted Boltzmann Machine 2 Free-running phase Ritajit Majumdar Arunabha Saha (CU) Applications of Boltzmann Machine Reference Boltzmann Machine November 6, 2013 31 / 44
Boltzmann Learning Algorithm Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: 1 Clamping phase Applications of Boltzmann Machine Restricted Boltzmann Machine 2 Free-running phase Reference 3 Learning phase Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 31 / 44
Boltzmann Machine Algorithm Ritajit Majumdar Arunabha Saha Outline Hopfield Net 1 Initialization: Set weights to random numbers in [-1, 1] Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 32 / 44
Boltzmann Machine Algorithm Ritajit Majumdar Arunabha Saha Outline Hopfield Net 1 Initialization: Set weights to random numbers in [-1, 1] 2 Clamping Phase: Present the net with the mapping it is supposed to learn by clamping input and output units to patterns. For each pattern perform simulated annealing on the hidden units in the sequence T0 , T1 , ..., Tfinal . At the final temperature, collect statistics to estimate the correlations ρji + = sj si here sj si + + Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference (j = i) P(Sβ = sβ |Sα = sα )sj si = sα ∈ sβ where sα and sβ represents the vector of visible and hidden neurons respectively Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 32 / 44
Boltzmann Machine Algorithm Ritajit Majumdar Arunabha Saha Outline 3 Free-running Phase:Repeat calculations in step 2, but this time only clamp the input units. Hence, at the final temperature, estimate the correlations ρji here sj si − − = sj si − (j = i) Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine P(Sβ = sβ )sj si = sα ∈ sβ Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 33 / 44
Boltzmann Machine Algorithm Ritajit Majumdar Arunabha Saha Outline 3 Free-running Phase:Repeat calculations in step 2, but this time only clamp the input units. Hence, at the final temperature, estimate the correlations ρji here sj si − − = sj si − Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine (j = i) Applications of Boltzmann Machine P(Sβ = sβ )sj si = sα ∈ Restricted Boltzmann Machine sβ 4 Learning Phase: updating the weights using the learning rule Reference wji = η(ρji + − ρ− ) ji η is learning parameter depending upon T (η = Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine ε T ) November 6, 2013 33 / 44
Boltzmann Machine Algorithm Ritajit Majumdar Arunabha Saha Outline 3 Free-running Phase:Repeat calculations in step 2, but this time only clamp the input units. Hence, at the final temperature, estimate the correlations ρji here sj si − − = sj si − Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine (j = i) Applications of Boltzmann Machine P(Sβ = sβ )sj si = sα ∈ Restricted Boltzmann Machine sβ 4 Learning Phase: updating the weights using the learning rule Reference wji = η(ρji + − ρ− ) ji η is learning parameter depending upon T (η = ε T ) 5 Iteration: Iterate steps 2 to 4 until the learning procedure converges with no more changes with synaptic weight wji ∀ j, i. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 33 / 44
Boltzmann Machine Learning Algorithm Ritajit Majumdar Arunabha Saha Outline Correlation Hopfield Net Everything that one weight needs to know of other weights and the data is contained in the difference of two correlations ∂logp(v ) = si sj ∂wij v − si sj model (7) Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine where . is the expectation value. The first term in R.H.S. denotes the expectation value of product of states at equilibrium when the state vector (or data) v is clamped on the visibile units. Reference The second term in R.H.S. denotes the expectation value of product of states at equilibrium without any clamping. So we can make the change in weight wij ∝ si sj Ritajit Majumdar Arunabha Saha (CU) v − si sj model Boltzmann Machine (8) November 6, 2013 34 / 44
Boltzmann Machine Why is it so? Ritajit Majumdar Arunabha Saha Outline Hopfield Net We know the probability of a global configuration at equilibrium is - Stochastic Hopfield Nets with Hidden Units Boltzmann Machine p(v , h) ∝ e −E (v ,h) Learning Algorithm for Boltzmann Machine So, the logarithm of probability is a linear function of the energy. Applications of Boltzmann Machine Restricted Boltzmann Machine And energy, on its own term, is a linear function of weights and states. E =− si bi − i Reference si sj wij i<j Hence, ∂E = −si sj ∂wij Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine (9) November 6, 2013 35 / 44
Boltzmann Machine Why is it so? Ritajit Majumdar Arunabha Saha Outline Hopfield Net Differentiating equation 1, we get ∂E ∂wij Stochastic Hopfield Nets with Hidden Units Boltzmann Machine = −si sj Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine 1 The process of settling to equilibrium state propagates information about the weights. 2 No need of back-propagation. The following two stages are required - 3 Restricted Boltzmann Machine Reference The machine needs to settle to equilibrium with data. The machine needs to settle to equilibrium without data. 4 However, in both cases the learning process is similar with different boundary conditions. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 36 / 44
Why we need negative phase Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The combine use of positive and negative phase stabilizes the distribution of synaptic weights. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 37 / 44
Why we need negative phase Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The combine use of positive and negative phase stabilizes the distribution of synaptic weights. The both phases are important equally due to the presence of partition function Z. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 37 / 44
Why we need negative phase Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The combine use of positive and negative phase stabilizes the distribution of synaptic weights. The both phases are important equally due to the presence of partition function Z. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The direction of steepest descent in energy space in not the same as the direction of steepest ascent in probability space. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 37 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 38 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference It runs very slow. Take very much time to learn. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference It runs very slow. Take very much time to learn. Weight explosion: If weights get too big too early, then the network get struck in one goodness optimum. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
Boltzmann Machine Shortcomings Ritajit Majumdar Arunabha Saha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference It runs very slow. Take very much time to learn. Weight explosion: If weights get too big too early, then the network get struck in one goodness optimum. - This shortcomings can be eliminated by sigmoid belief network Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Cancer Detection Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Cancer Detection Loan Application Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Boltzmann Machine Applications Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Cancer Detection Loan Application Decision making Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
Restricted Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Restricted Boltzmann Machine is a stochastic neural network (random behaviour when activated). It consist of one layer of visible units (neurons) and one layer of hidden units. Units in each layer have no connections between them and are connected to all other units in other layer. Connections between neurons are bidirectional and symmetric . This means that information flows in both directions during the training and during the usage of the network and that weights are the same in both directions. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 40 / 44
Boltzmann Machine How RBM Works Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine First the network is trained by using some data set and setting the neurons on visible layer to match data points in this data set. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 41 / 44
Boltzmann Machine How RBM Works Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine First the network is trained by using some data set and setting the neurons on visible layer to match data points in this data set. After the network is trained we can use it on new unknown data to make classification of the data (this is known as unsupervised learning) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 41 / 44
Continuous Restricted Boltzmann Machine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net CRBM have very close implementation to original RBM with binomial neurons (0,1) as possible values of activation. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Training data2 2 500 Reconstructed data training data were taken Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 42 / 44
Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Geoffrey Hinton Neural Networks for Machine Learning www.coursera.org Geoffrey Hinton http://www.scholarpedia.org/article/Boltzmann_ machine Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Simon Haykin, 2ed Geoffrey Hinton, David Ackley A learning Algorithm for Boltzmann Machine Cognitive Science 9, 147-169 (1985) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 43 / 44
Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine THANK YOU Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 44 / 44

More Related Content

PDF
Artificial Neural Network Lect4 : Single Layer Perceptron Classifiers
byMohammed Bennamoun
 
PDF
Hebb network
bySiksha 'O' Anusandhan (Deemed to be University )
 
PPT
Adaline madaline
byNagarajan
 
PPT
neural networks
byInstitute of Technology Telkom
 
PPTX
HOPFIELD NETWORK
byankita pandey
 
PPTX
Hopfield Networks
byKanchana Rani G
 
PPTX
hopfield neural network
byAbhishikha Sinha
 
PPTX
Extreme learning machine:Theory and applications
byJames Chou
 
Artificial Neural Network Lect4 : Single Layer Perceptron Classifiers
byMohammed Bennamoun
 
Hebb network
bySiksha 'O' Anusandhan (Deemed to be University )
 
Adaline madaline
byNagarajan
 
neural networks
byInstitute of Technology Telkom
 
HOPFIELD NETWORK
byankita pandey
 
Hopfield Networks
byKanchana Rani G
 
hopfield neural network
byAbhishikha Sinha
 
Extreme learning machine:Theory and applications
byJames Chou
 

What's hot

PPTX
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
PPTX
Feedforward neural network
bySopheaktra YONG
 
PDF
Restricted Boltzmann Machine | Neural Network Tutorial | Deep Learning Tutori...
byEdureka!
 
PDF
P, NP, NP-Complete, and NP-Hard
byAnimesh Chaturvedi
 
PPT
Adaptive Resonance Theory
byNaveen Kumar
 
PPTX
Activation functions
byPRATEEK SAHU
 
PPTX
Restricted boltzmann machine
by강민국 강민국
 
PDF
Restricted Boltzmann Machine - A comprehensive study with a focus on Deep Bel...
byIndraneel Pole
 
PPT
backpropagation in neural networks
byAkash Goel
 
PPTX
K-Nearest Neighbor(KNN)
byAbdullah al Mamun
 
PPT
Perceptron
byNagarajan
 
PDF
Introduction to Recurrent Neural Network
byKnoldus Inc.
 
PPT
Bayseian decision theory
bysia16
 
PDF
Support Vector Machines for Classification
byPrakash Pimpale
 
PPTX
Autoencoders in Deep Learning
bymilad abbasi
 
PPT
2.5 backpropagation
byKrish_ver2
 
PPTX
Convolutional Neural Network and Its Applications
byKasun Chinthaka Piyarathna
 
PPT
Perceptron algorithm
byZul Kawsar
 
PDF
Convolutional Neural Networks (CNN)
byGaurav Mittal
 
PPTX
Multilayer perceptron
byomaraldabash
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
Feedforward neural network
bySopheaktra YONG
 
Restricted Boltzmann Machine | Neural Network Tutorial | Deep Learning Tutori...
byEdureka!
 
P, NP, NP-Complete, and NP-Hard
byAnimesh Chaturvedi
 
Adaptive Resonance Theory
byNaveen Kumar
 
Activation functions
byPRATEEK SAHU
 
Restricted boltzmann machine
by강민국 강민국
 
Restricted Boltzmann Machine - A comprehensive study with a focus on Deep Bel...
byIndraneel Pole
 
backpropagation in neural networks
byAkash Goel
 
K-Nearest Neighbor(KNN)
byAbdullah al Mamun
 
Perceptron
byNagarajan
 
Introduction to Recurrent Neural Network
byKnoldus Inc.
 
Bayseian decision theory
bysia16
 
Support Vector Machines for Classification
byPrakash Pimpale
 
Autoencoders in Deep Learning
bymilad abbasi
 
2.5 backpropagation
byKrish_ver2
 
Convolutional Neural Network and Its Applications
byKasun Chinthaka Piyarathna
 
Perceptron algorithm
byZul Kawsar
 
Convolutional Neural Networks (CNN)
byGaurav Mittal
 
Multilayer perceptron
byomaraldabash
 

Similar to Brief Introduction to Boltzmann Machine

PPTX
Boltzmann Machines in Deep learning and machine learning also used for traini...
byvenkatasaisumanth74
 
PDF
Artificial Neural Networks Lect8: Neural networks for constrained optimization
byMohammed Bennamoun
 
PPTX
Unit iii update
byIndira Priyadarsini
 
PPTX
Restricted Boltzman Machine (RBM) presentation of fundamental theory
bySeongwon Hwang
 
PDF
Neural Networks on Steroids (Poster)
byAdam Blevins
 
PPTX
LEC 04 ANN Boltzmann Machines Restricted
byAiniBasit
 
PDF
Dl1 deep learning_algorithms
byArmando Vieira
 
PDF
Concept of Regulation Analysis using Restricted Boltzmann Machines
byPatrick Michl
 
PDF
Learning RBM(Restricted Boltzmann Machine in Practice)
byMad Scientists
 
PDF
Quantum Deep Learning
byWilly Marroquin (WillyDevNET)
 
PPT
NIPS2007: deep belief nets
byzukun
 
PDF
Regulation Analysis using Restricted Boltzmann Machines
byPatrick Michl
 
PDF
rbm_hls
byDavid Lau
 
PDF
Learning to discover monte carlo algorithm on spin ice manifold
byKai-Wen Zhao
 
Boltzmann Machines in Deep learning and machine learning also used for traini...
byvenkatasaisumanth74
 
Artificial Neural Networks Lect8: Neural networks for constrained optimization
byMohammed Bennamoun
 
Unit iii update
byIndira Priyadarsini
 
Restricted Boltzman Machine (RBM) presentation of fundamental theory
bySeongwon Hwang
 
Neural Networks on Steroids (Poster)
byAdam Blevins
 
LEC 04 ANN Boltzmann Machines Restricted
byAiniBasit
 
Dl1 deep learning_algorithms
byArmando Vieira
 
Concept of Regulation Analysis using Restricted Boltzmann Machines
byPatrick Michl
 
Learning RBM(Restricted Boltzmann Machine in Practice)
byMad Scientists
 
Quantum Deep Learning
byWilly Marroquin (WillyDevNET)
 
NIPS2007: deep belief nets
byzukun
 
Regulation Analysis using Restricted Boltzmann Machines
byPatrick Michl
 
rbm_hls
byDavid Lau
 
Learning to discover monte carlo algorithm on spin ice manifold
byKai-Wen Zhao
 

More from Arunabha Saha

PDF
Naive Bayes Classifier
byArunabha Saha
 
PDF
Baseline Wandering
byArunabha Saha
 
PDF
Polar-NRZ & Polar-RZ encoding schemes
byArunabha Saha
 
PDF
Manchester & Differential Manchester encoding scheme
byArunabha Saha
 
PDF
Introduction to Quantum Computation. Part - 1
byArunabha Saha
 
PDF
Unipolar Encoding Techniques: NRZ & RZ
byArunabha Saha
 
PDF
Introduction to Quantum Computation. Part - 2
byArunabha Saha
 
PDF
Introduction to Quantum Secret Sharing
byArunabha Saha
 
PDF
Introduction to Quantum Teleportation & BBCJPW Protocol
byArunabha Saha
 
PDF
Introduction to Channel Capacity | DCNIT-LDTalks-1
byArunabha Saha
 
PDF
Introduction to Line Coding Schemes
byArunabha Saha
 
PDF
Data Communication, Networking & Internet Technology Lecture Series(DCNIT-LDT...
byArunabha Saha
 
Naive Bayes Classifier
byArunabha Saha
 
Baseline Wandering
byArunabha Saha
 
Polar-NRZ & Polar-RZ encoding schemes
byArunabha Saha
 
Manchester & Differential Manchester encoding scheme
byArunabha Saha
 
Introduction to Quantum Computation. Part - 1
byArunabha Saha
 
Unipolar Encoding Techniques: NRZ & RZ
byArunabha Saha
 
Introduction to Quantum Computation. Part - 2
byArunabha Saha
 
Introduction to Quantum Secret Sharing
byArunabha Saha
 
Introduction to Quantum Teleportation & BBCJPW Protocol
byArunabha Saha
 
Introduction to Channel Capacity | DCNIT-LDTalks-1
byArunabha Saha
 
Introduction to Line Coding Schemes
byArunabha Saha
 
Data Communication, Networking & Internet Technology Lecture Series(DCNIT-LDT...
byArunabha Saha
 

Recently uploaded

PDF
Pharmaceutical Biotechnology unitI Notes
byBhagyashree Prakash Gajbhare
 
PDF
Profiling, Placement, Pathways, and Pedagogies.pdf
byRHEAMARIMLA2
 
PDF
PULSE according to Physiology special pdf for all medical sectors students & ...
byhttps://www.facebook.com/profile.php?id=61578457231735
 
PPTX
When UDL is no longer enough: Examining post-UDL reflections on inclusive des...
byFrederic Fovet
 
PPTX
ANTI-TUSSIVE CHAPTER NO.05 PHARMACOGNOSY
byGanu Gavade
 
PPTX
INSTAGRAM: Where Moment Turns Into Stories.pptx
byandriemark51
 
PPT
NCompass Live: Best New Teen Reads of 2025
byNebraska Library Commission
 
PDF
2025-CB-NCAE-slide-deck.pptx-1 new kamote
byLuckyReyes6
 
PDF
Application of ICT Lecture 6 Digital Citizenship and Online Etiquette.pdf
byKhalil Khan Marwat
 
PPTX
Capacity Building Programme for Teachers on Equitable and Inclusive Education
byMUKHTAR133762
 
PDF
Oxygen-Containing-Compounds.pdf/IITJEE,NEET CLASS 12/CHEMISTRY /BY K SANDEEP ...
bySandeep Swamy
 
PDF
Error-Analysis-in-Analytical-Chemistry-and-statistics ppt.pdf/by k sandeep swamy
bySandeep Swamy
 
PPTX
Road Safety as a shared responsibility, 2026
bySougata Pal
 
PPTX
Indian_Economy_Professional_Charts_1947_2025.pptx
byVishalTyagi304086
 
PPTX
Shipping Policies & Picking Policies in Odoo 18 Inventory
byCeline George
 
PPTX
OverView of AI in Products in Odoo 19
byCeline George
 
PPTX
How to Manage Sorting Cart by Category in Odoo 18 POS
byCeline George
 
PPTX
CHAPTER NO. 05 DIURETICS PHARMACOGNOSY.pptx
byGanu Gavade
 
PPTX
How to use filtered() method in Odoo 18
byCeline George
 
PPTX
How to use sorted() method in Odoo 18
byCeline George
 
Pharmaceutical Biotechnology unitI Notes
byBhagyashree Prakash Gajbhare
 
Profiling, Placement, Pathways, and Pedagogies.pdf
byRHEAMARIMLA2
 
PULSE according to Physiology special pdf for all medical sectors students & ...
byhttps://www.facebook.com/profile.php?id=61578457231735
 
When UDL is no longer enough: Examining post-UDL reflections on inclusive des...
byFrederic Fovet
 
ANTI-TUSSIVE CHAPTER NO.05 PHARMACOGNOSY
byGanu Gavade
 
INSTAGRAM: Where Moment Turns Into Stories.pptx
byandriemark51
 
NCompass Live: Best New Teen Reads of 2025
byNebraska Library Commission
 
2025-CB-NCAE-slide-deck.pptx-1 new kamote
byLuckyReyes6
 
Application of ICT Lecture 6 Digital Citizenship and Online Etiquette.pdf
byKhalil Khan Marwat
 
Capacity Building Programme for Teachers on Equitable and Inclusive Education
byMUKHTAR133762
 
Oxygen-Containing-Compounds.pdf/IITJEE,NEET CLASS 12/CHEMISTRY /BY K SANDEEP ...
bySandeep Swamy
 
Error-Analysis-in-Analytical-Chemistry-and-statistics ppt.pdf/by k sandeep swamy
bySandeep Swamy
 
Road Safety as a shared responsibility, 2026
bySougata Pal
 
Indian_Economy_Professional_Charts_1947_2025.pptx
byVishalTyagi304086
 
Shipping Policies & Picking Policies in Odoo 18 Inventory
byCeline George
 
OverView of AI in Products in Odoo 19
byCeline George
 
How to Manage Sorting Cart by Category in Odoo 18 POS
byCeline George
 
CHAPTER NO. 05 DIURETICS PHARMACOGNOSY.pptx
byGanu Gavade
 
How to use filtered() method in Odoo 18
byCeline George
 
How to use sorted() method in Odoo 18
byCeline George
 

Brief Introduction to Boltzmann Machine

  • 1.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Boltzmann Machine A Brief Introduction Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Ritajit Majumdar Arunabha Saha Restricted Boltzmann Machine Reference University of Calcutta November 6, 2013 Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 1 / 44
  • 2.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha 1 Hopfield Net Outline Hopfield Net 2 Stochastic Hopfield Nets with Hidden Units Stochastic Hopfield Nets with Hidden Units Boltzmann Machine 3 Learning Algorithm for Boltzmann Machine Boltzmann Machine Applications of Boltzmann Machine 4 Learning Algorithm for Boltzmann Machine 5 Applications of Boltzmann Machine 6 Restricted Boltzmann Machine 7 Reference Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 2 / 44
  • 3.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha Hopfield Network Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Figure: Two dimensional representation of motion in state space Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 3 / 44
  • 4.
    Boltzmann Machine Hopfield Net RitajitMajumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Ritajit Majumdar Arunabha Saha (CU) Reference Boltzmann Machine November 6, 2013 4 / 44
  • 5.
    Boltzmann Machine Hopfield Net RitajitMajumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Reference John Hopfield introduced a global energy function for network with symmetric connections. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 4 / 44
  • 6.
    Boltzmann Machine Hopfield Net RitajitMajumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Reference John Hopfield introduced a global energy function for network with symmetric connections. 1 Each binary configuration of the whole network has an Energy. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 4 / 44
  • 7.
    Boltzmann Machine Hopfield Net RitajitMajumdar Arunabha Saha Outline A Hopfield Net is composed of binary threshold units with recurrent connections between them. Hopfield Net Recurrent networks of non-linear units are hard to analyze, since they can behave in many different ways - Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine 1 Settle to a stable state. Applications of Boltzmann Machine 2 Oscillate Follow chaotic trajectory. Restricted Boltzmann Machine 3 Reference John Hopfield introduced a global energy function for network with symmetric connections. 1 Each binary configuration of the whole network has an Energy. 2 The binary threshold decision rule causes the network to settle to a minimum of this energy function. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 4 / 44
  • 8.
    Boltzmann Machine Energy Function RitajitMajumdar Arunabha Saha Outline The global energy is defined as - Hopfield Net E =− si bi − i si sj wij (1) i<j where bi is the bias of the i th unit, s is 0 or 1 1 depending on whether the unit is turned off or on respectively. And wij is the weight of the connection between units i and j. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference 1 For Bipolar Inputs the states will be −1 and 1 respectively. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 5 / 44
  • 9.
    Boltzmann Machine Energy Function RitajitMajumdar Arunabha Saha Outline The global energy is defined as - Hopfield Net E =− si bi − i si sj wij (1) i<j where bi is the bias of the i th unit, s is 0 or 1 1 depending on whether the unit is turned off or on respectively. And wij is the weight of the connection between units i and j. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference From this energy function, each unit computes locally how changing their state will affect the global energy. 1 For Bipolar Inputs the states will be −1 and 1 respectively. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 5 / 44
  • 10.
    Boltzmann Machine Energy Function RitajitMajumdar Arunabha Saha Outline The global energy is defined as - Hopfield Net E =− si bi − i si sj wij (1) i<j Stochastic Hopfield Nets with Hidden Units Boltzmann Machine where bi is the bias of the i th unit, s is 0 or 1 1 depending on whether the unit is turned off or on respectively. And wij is the weight of the connection between units i and j. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference From this energy function, each unit computes locally how changing their state will affect the global energy. The energy gap is defined as Ei = E (si = 0) − E (si = 1) = bi + sj wij (2) j 1 For Bipolar Inputs the states will be −1 and 1 respectively. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 5 / 44
  • 11.
    Settling to anEnergy Minima Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units The net is initially in a random state i.e., the units are on or off randomly. The binary threshold decision rule updates units one at a time in a random order. Update each unit to whichever of its two states minimizes the global energy. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Use binary threshold units, i.e., the states can be either 0 or 1. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 6 / 44
  • 12.
    Settling to anEnergy Minima Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units The net is initially in a random state i.e., the units are on or off randomly. The binary threshold decision rule updates units one at a time in a random order. Update each unit to whichever of its two states minimizes the global energy. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Use binary threshold units, i.e., the states can be either 0 or 1. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 7 / 44
  • 13.
    Settling to anEnergy Minima Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units The net is initially in a random state i.e., the units are on or off randomly. The binary threshold decision rule updates units one at a time in a random order. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Update each unit to whichever of its two states minimizes the global energy. Use binary threshold units, i.e., the states can be either 0 or 1. Ritajit Majumdar Arunabha Saha (CU) Reference -E = goodness = 4 Boltzmann Machine November 6, 2013 8 / 44
  • 14.
    Using this typeof Computation as memory Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Hopfield proposed that memories could be energy minima of a neural net. The binary threshold decision rule can be used to “clean up” incomplete or corrupted memory. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 9 / 44
  • 15.
    Using this typeof Computation as memory Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Hopfield proposed that memories could be energy minima of a neural net. The binary threshold decision rule can be used to “clean up” incomplete or corrupted memory. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Using energy minima to represent memories gives a content-addressable memory. An item can be accessed just by knowing parts of it. It is robust against hardware damage. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 9 / 44
  • 16.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha Stochastic Hopfield Nets with Hidden Units Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 10 / 44
  • 17.
    Hopfield Net withHidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 11 / 44
  • 18.
    Hopfield Net withHidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Use the net to construct interpretations of sensory input. Ritajit Majumdar Arunabha Saha (CU) Learning Algorithm for Boltzmann Machine Reference Boltzmann Machine November 6, 2013 11 / 44
  • 19.
    Hopfield Net withHidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Use the net to construct interpretations of sensory input. Reference The input is represented by the visible units. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 11 / 44
  • 20.
    Hopfield Net withHidden Units Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Why add hidden units? Stochastic Hopfield Nets with Hidden Units In Hopfield Net, there is no hidden layer of units. By adding hidden layers, the attention can be shifted from just storing memories to various types of interpretations of the inputs. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Use the net to construct interpretations of sensory input. Reference The input is represented by the visible units. The interpretation is represented by the states of the hidden units. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 11 / 44
  • 21.
    Boltzmann Machine Noisy Networks RitajitMajumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
  • 22.
    Boltzmann Machine Noisy Networks RitajitMajumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
  • 23.
    Boltzmann Machine Noisy Networks RitajitMajumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Ritajit Majumdar Arunabha Saha (CU) Restricted Boltzmann Machine Reference Boltzmann Machine November 6, 2013 12 / 44
  • 24.
    Boltzmann Machine Noisy Networks RitajitMajumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Restricted Boltzmann Machine Reference Start with a lot of noise to escape the energy barriers of poor local minima. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
  • 25.
    Boltzmann Machine Noisy Networks RitajitMajumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Restricted Boltzmann Machine Reference Start with a lot of noise to escape the energy barriers of poor local minima. Slowly reduce the noise so that the system ends up in a deep minima. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
  • 26.
    Boltzmann Machine Noisy Networks RitajitMajumdar Arunabha Saha Outline The Binary Threshold Decision rule always goes downhill, i.e., reduces energy. Hopfield Net Hence it is impossible to escape from a local minima. Learning Algorithm for Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Applications of Boltzmann Machine Solution - Use random noise to escape from poor, shallow minima. Restricted Boltzmann Machine Reference Start with a lot of noise to escape the energy barriers of poor local minima. Slowly reduce the noise so that the system ends up in a deep minima. This process is called “Simulated Annealing”. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 12 / 44
  • 27.
    Boltzmann Machine Stochastic BinaryUnits Ritajit Majumdar Arunabha Saha Outline How to add noise? Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
  • 28.
    Boltzmann Machine Stochastic BinaryUnits Ritajit Majumdar Arunabha Saha Outline How to add noise? 1 Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
  • 29.
    Boltzmann Machine Stochastic BinaryUnits Ritajit Majumdar Arunabha Saha Outline How to add noise? 1 2 Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units The “temperature” controls the amount of noise. Learning Algorithm for Boltzmann Machine Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
  • 30.
    Boltzmann Machine Stochastic BinaryUnits Ritajit Majumdar Arunabha Saha Outline How to add noise? Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units 2 The “temperature” controls the amount of noise. Learning Algorithm for Boltzmann Machine 3 Unit i then turns on with the probability given by the logistic function - Applications of Boltzmann Machine 1 Boltzmann Machine Restricted Boltzmann Machine Reference prob(si = 1) = Ritajit Majumdar Arunabha Saha (CU) 1 1 + e− Ei T Boltzmann Machine (3) November 6, 2013 13 / 44
  • 31.
    Boltzmann Machine Stochastic BinaryUnits Ritajit Majumdar Arunabha Saha Outline How to add noise? Hopfield Net Replace the binary threshold units by binary stochastic units that make biased random decisions. Stochastic Hopfield Nets with Hidden Units 2 The “temperature” controls the amount of noise. Learning Algorithm for Boltzmann Machine 3 Unit i then turns on with the probability given by the logistic function - Applications of Boltzmann Machine 1 Boltzmann Machine Restricted Boltzmann Machine Reference prob(si = 1) = T =0 T →∞ T =1 1 1 + e− Ei T (3) Deterministic (Hopfield Net) Complete Chaos Approaches Boltzmann Distribution Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 13 / 44
  • 32.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 14 / 44
  • 33.
    Boltzmann Machine Statistical Mechanics RitajitMajumdar Arunabha Saha Consider a physical system with large number of possible states and many degrees of freedom. Let pi be the occurrence probability of state i Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine pi ≥ 0 for all i and pi = 1. i At thermal equilibrium state i occurs with probability 1 Z exp pi = Ei − kB T Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference where Ei is the energy of the system Boltzmann constant, kB = 1.38x10−23 J/K Ei exp − kB T is the Boltzmann Factor Partition function,(Zustadsumme) exp − Z= i Ritajit Majumdar Arunabha Saha (CU) Ei kB T Boltzmann Machine November 6, 2013 15 / 44
  • 34.
    Boltzmann Machine Statistical Mechanics RitajitMajumdar Arunabha Saha Outline Hopfield Net Gibbs Distribution Stochastic Hopfield Nets with Hidden Units 1 States of low energy have a higher occurrence probability than states of high energy. 2 As T is reduced, the probability is concentrated on a smaller subset of low-energy states. In the context of neural nets, the parameter T (which controls thermal fluctuations) represents the effect of synaptic noise. Hence kB = 1 is set and pi and Z getting the form 1 Z exp pi = i Ritajit Majumdar Arunabha Saha (CU) Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference − Ei T exp − Z= Boltzmann Machine Ei T Boltzmann Machine November 6, 2013 16 / 44
  • 35.
    What is BoltzmannMachine? Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Boltzmann Machine Learning Algorithm for Boltzmann Machine A Hopfield Net consisting of Binary Stochastic Neuron with hidden units is called Boltzmann Machine. Applications of Boltzmann Machine Restricted Boltzmann Machine Reference A Boltzmann Machine is a network of symmetrically connected, neuron like units that make stochastic decisions about whether to be on or off. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 17 / 44
  • 36.
    Structure of BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The stochastic neurons of Boltzmann machine are in two groups: vissible and hidden. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 18 / 44
  • 37.
    Structure of BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The stochastic neurons of Boltzmann machine are in two groups: vissible and hidden. visible neurons provide an interface between the net and its environment. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 18 / 44
  • 38.
    Structure of BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 19 / 44
  • 39.
    Structure of BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. the hidden units do this (explain underlying constraints) by capturing higher-order correlations between the clamping vectors. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 19 / 44
  • 40.
    Structure of BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. the hidden units do this (explain underlying constraints) by capturing higher-order correlations between the clamping vectors. Boltzmann machine learning may be viewed as an unsupervised learning procedure for modelling a distribution that is specified by the clamping patterns. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 19 / 44
  • 41.
    Structure of BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline during the training phase, the visible neurons are clamped; the hidden neurons always operate freely, they are used to explain underlying constraints in the environmental input vectors. the hidden units do this (explain underlying constraints) by capturing higher-order correlations between the clamping vectors. Boltzmann machine learning may be viewed as an unsupervised learning procedure for modelling a distribution that is specified by the clamping patterns. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference the network can perform pattern completion: when a vector bearing part of the information is clamped onto a subset of the visible neurons, the network performs completion of the pattern on the remaining visible neurons (if it has learnt properly). Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 19 / 44
  • 42.
    Boltzmann Machine Modelling BinaryData Ritajit Majumdar Arunabha Saha The objective of Boltzmann Machine is - Outline Hopfield Net Modelling Binary Data Stochastic Hopfield Nets with Hidden Units Given a training set of binary vectors, fit the model that will assign a probability to every possible binary vector. When unit i is given opportunity to update its state, it first computes its total input zi , zi = bi + sj wij Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine (4) Reference j Unit i turns on with probability prob(si = 1) = 1 1 + e −zi (5) If the units are updated sequentially in random order, the network will eventually reach a Boltzmann Distribution, also called its stationary distribution. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 20 / 44
  • 43.
    How Boltzmann MachineGenerates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline It is not a causal generative model. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 21 / 44
  • 44.
    How Boltzmann MachineGenerates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline It is not a causal generative model. Everything is defined in terms of energies of joint configurations of the visible and hidden units. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 21 / 44
  • 45.
    How Boltzmann MachineGenerates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net It is not a causal generative model. Everything is defined in terms of energies of joint configurations of the visible and hidden units. The energies of the joint configurations are related to their probabilities by two ways: −E (v,h) - either by defining the probability p(v, h) ∝ e - define the probability to be the probability of finding the network in that joint configuration after we have updated all of the stochastic binary units many times. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 21 / 44
  • 46.
    How Boltzmann MachineGenerates Data Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net It is not a causal generative model. Everything is defined in terms of energies of joint configurations of the visible and hidden units. The energies of the joint configurations are related to their probabilities by two ways: −E (v,h) - either by defining the probability p(v, h) ∝ e - define the probability to be the probability of finding the network in that joint configuration after we have updated all of the stochastic binary units many times. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 21 / 44
  • 47.
    Boltzmann Machine How BoltzmannMachine Generates Data Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units In Boltzmann Machine, everything is defined in terms of the energies of joint configurations of the visible (v) and hidden (h) units. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine The energies of joint configurations are related to their probabilities as p(v , h) ∝ e −E (v ,h) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Reference (6) November 6, 2013 22 / 44
  • 48.
    Probabilities in termof Energies Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net The probability of a joint configuration over both visible and hidden units depends on the energy of that joint configuration compared with the energies of all other joint configurations. Ritajit Majumdar Arunabha Saha (CU) Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Boltzmann Machine November 6, 2013 23 / 44
  • 49.
    Probabilities in termof Energies Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net The probability of a joint configuration over both visible and hidden units depends on the energy of that joint configuration compared with the energies of all other joint configurations. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The probability of a configuration of the visible units is the sum of the probabilities of all the joint configurations that contain it. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 23 / 44
  • 50.
    Boltzmann Machine An Example RitajitMajumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Figure: Credit: Geoffrey Hinton, Neural Networks for Machine Learning Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 24 / 44
  • 51.
    Boltzmann Machine Limitation andSolution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
  • 52.
    Boltzmann Machine Limitation andSolution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? If there are large number of hidden units then we cannot calculate partition function, as it is exponentially many terms. We need to sample the data and to do this we can use Markov Chain Monte Carlo(MCMC). Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
  • 53.
    Boltzmann Machine Limitation andSolution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? If there are large number of hidden units then we cannot calculate partition function, as it is exponentially many terms. We need to sample the data and to do this we can use Markov Chain Monte Carlo(MCMC). starting from a random global configuration pick units at random and allow them to stochastically update their states based on their energy gaps. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Run MCMC until it reaches it stationary distribution(thermal eqb. at temp is 1) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
  • 54.
    Boltzmann Machine Limitation andSolution Ritajit Majumdar Arunabha Saha Outline Hopfield Net What if the network is remarkably large? If there are large number of hidden units then we cannot calculate partition function, as it is exponentially many terms. We need to sample the data and to do this we can use Markov Chain Monte Carlo(MCMC). starting from a random global configuration pick units at random and allow them to stochastically update their states based on their energy gaps. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Run MCMC until it reaches it stationary distribution(thermal eqb. at temp is 1) the probability is related to its energy p(v , h) ∝ e −E (v ,h) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 25 / 44
  • 55.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Boltzmann Learning Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine “A surprising feature of this rule is that it uses only locally available information. The change of weight depends only on the behaviour of the two units it connects, even though the change optimizes a global measure.” Applications of Boltzmann Machine Restricted Boltzmann Machine Reference - Ackley, Hinton 1985 Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 26 / 44
  • 56.
    Boltzmann Machine Goal ofLearning Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Learning Algorithm for Boltzmann Machine is an unsupervised learning algorithm. Unlike Backpropagation Algorithm, where the training set consists of input vector and desired output, in Boltzmann Machine only the input vector is provided. Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine We want to maximize the product of the probabilities the Boltzmann Machine assigns to the binary vectors in training set. Reference It is equivalent to maximizing the probability that we would obtain exactly the N training cases. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 27 / 44
  • 57.
    Boltzmann Machine Assumptions Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The goal of Boltzmann learning is to produce a NN that categorize input patterns according to Boltzmann distribution. Two assumptions are made: Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 28 / 44
  • 58.
    Boltzmann Machine Assumptions Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The goal of Boltzmann learning is to produce a NN that categorize input patterns according to Boltzmann distribution. Two assumptions are made: 1 Each environmental vector persists long enough for the network to reach thermal equilibrium. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 28 / 44
  • 59.
    Boltzmann Machine Assumptions Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The goal of Boltzmann learning is to produce a NN that categorize input patterns according to Boltzmann distribution. Two assumptions are made: 1 Each environmental vector persists long enough for the network to reach thermal equilibrium. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference 2 There is no structure in the sequence in which environmental vectors are clamped to the visible units of the network. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 28 / 44
  • 60.
    Why Learning couldbe difficult Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Consider a chain of hidden units with visibile units attached at two ends - Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Training: We want the two visibile units to be in opposite states. Solution: The product of all the weights must be negative. If all are positive, then turning on one unit will turn on the next unit and eventually the two visibile units will be in same state. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 29 / 44
  • 61.
    Why Learning couldbe difficult Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Consider a chain of hidden units with visibile units attached at two ends - Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Difficulty: To modify w1 and w5, we need to know w3 (and the weights of other hidden units too). Because, if w3 is negative, then we need to modify w1 in a different way than what we would do if w3 is positive. So to change one weight in a right direction, we need to know all the other weights. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 30 / 44
  • 62.
    Boltzmann Learning Algorithm BoltzmannMachine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 31 / 44
  • 63.
    Boltzmann Learning Algorithm BoltzmannMachine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: 1 Clamping phase Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 31 / 44
  • 64.
    Boltzmann Learning Algorithm BoltzmannMachine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: 1 Clamping phase Restricted Boltzmann Machine 2 Free-running phase Ritajit Majumdar Arunabha Saha (CU) Applications of Boltzmann Machine Reference Boltzmann Machine November 6, 2013 31 / 44
  • 65.
    Boltzmann Learning Algorithm BoltzmannMachine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine The learning procedure mainly divided into three phases: 1 Clamping phase Applications of Boltzmann Machine Restricted Boltzmann Machine 2 Free-running phase Reference 3 Learning phase Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 31 / 44
  • 66.
    Boltzmann Machine Algorithm Ritajit Majumdar ArunabhaSaha Outline Hopfield Net 1 Initialization: Set weights to random numbers in [-1, 1] Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 32 / 44
  • 67.
    Boltzmann Machine Algorithm Ritajit Majumdar ArunabhaSaha Outline Hopfield Net 1 Initialization: Set weights to random numbers in [-1, 1] 2 Clamping Phase: Present the net with the mapping it is supposed to learn by clamping input and output units to patterns. For each pattern perform simulated annealing on the hidden units in the sequence T0 , T1 , ..., Tfinal . At the final temperature, collect statistics to estimate the correlations ρji + = sj si here sj si + + Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference (j = i) P(Sβ = sβ |Sα = sα )sj si = sα ∈ sβ where sα and sβ represents the vector of visible and hidden neurons respectively Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 32 / 44
  • 68.
    Boltzmann Machine Algorithm Ritajit Majumdar ArunabhaSaha Outline 3 Free-running Phase:Repeat calculations in step 2, but this time only clamp the input units. Hence, at the final temperature, estimate the correlations ρji here sj si − − = sj si − (j = i) Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine P(Sβ = sβ )sj si = sα ∈ sβ Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 33 / 44
  • 69.
    Boltzmann Machine Algorithm Ritajit Majumdar ArunabhaSaha Outline 3 Free-running Phase:Repeat calculations in step 2, but this time only clamp the input units. Hence, at the final temperature, estimate the correlations ρji here sj si − − = sj si − Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine (j = i) Applications of Boltzmann Machine P(Sβ = sβ )sj si = sα ∈ Restricted Boltzmann Machine sβ 4 Learning Phase: updating the weights using the learning rule Reference wji = η(ρji + − ρ− ) ji η is learning parameter depending upon T (η = Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine ε T ) November 6, 2013 33 / 44
  • 70.
    Boltzmann Machine Algorithm Ritajit Majumdar ArunabhaSaha Outline 3 Free-running Phase:Repeat calculations in step 2, but this time only clamp the input units. Hence, at the final temperature, estimate the correlations ρji here sj si − − = sj si − Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine (j = i) Applications of Boltzmann Machine P(Sβ = sβ )sj si = sα ∈ Restricted Boltzmann Machine sβ 4 Learning Phase: updating the weights using the learning rule Reference wji = η(ρji + − ρ− ) ji η is learning parameter depending upon T (η = ε T ) 5 Iteration: Iterate steps 2 to 4 until the learning procedure converges with no more changes with synaptic weight wji ∀ j, i. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 33 / 44
  • 71.
    Boltzmann Machine Learning Algorithm RitajitMajumdar Arunabha Saha Outline Correlation Hopfield Net Everything that one weight needs to know of other weights and the data is contained in the difference of two correlations ∂logp(v ) = si sj ∂wij v − si sj model (7) Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine where . is the expectation value. The first term in R.H.S. denotes the expectation value of product of states at equilibrium when the state vector (or data) v is clamped on the visibile units. Reference The second term in R.H.S. denotes the expectation value of product of states at equilibrium without any clamping. So we can make the change in weight wij ∝ si sj Ritajit Majumdar Arunabha Saha (CU) v − si sj model Boltzmann Machine (8) November 6, 2013 34 / 44
  • 72.
    Boltzmann Machine Why isit so? Ritajit Majumdar Arunabha Saha Outline Hopfield Net We know the probability of a global configuration at equilibrium is - Stochastic Hopfield Nets with Hidden Units Boltzmann Machine p(v , h) ∝ e −E (v ,h) Learning Algorithm for Boltzmann Machine So, the logarithm of probability is a linear function of the energy. Applications of Boltzmann Machine Restricted Boltzmann Machine And energy, on its own term, is a linear function of weights and states. E =− si bi − i Reference si sj wij i<j Hence, ∂E = −si sj ∂wij Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine (9) November 6, 2013 35 / 44
  • 73.
    Boltzmann Machine Why isit so? Ritajit Majumdar Arunabha Saha Outline Hopfield Net Differentiating equation 1, we get ∂E ∂wij Stochastic Hopfield Nets with Hidden Units Boltzmann Machine = −si sj Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine 1 The process of settling to equilibrium state propagates information about the weights. 2 No need of back-propagation. The following two stages are required - 3 Restricted Boltzmann Machine Reference The machine needs to settle to equilibrium with data. The machine needs to settle to equilibrium without data. 4 However, in both cases the learning process is similar with different boundary conditions. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 36 / 44
  • 74.
    Why we neednegative phase Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The combine use of positive and negative phase stabilizes the distribution of synaptic weights. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 37 / 44
  • 75.
    Why we neednegative phase Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The combine use of positive and negative phase stabilizes the distribution of synaptic weights. The both phases are important equally due to the presence of partition function Z. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 37 / 44
  • 76.
    Why we neednegative phase Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine The combine use of positive and negative phase stabilizes the distribution of synaptic weights. The both phases are important equally due to the presence of partition function Z. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference The direction of steepest descent in energy space in not the same as the direction of steepest ascent in probability space. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 37 / 44
  • 77.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
  • 78.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
  • 79.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
  • 80.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 38 / 44
  • 81.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference It runs very slow. Take very much time to learn. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
  • 82.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference It runs very slow. Take very much time to learn. Weight explosion: If weights get too big too early, then the network get struck in one goodness optimum. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
  • 83.
    Boltzmann Machine Shortcomings Ritajit Majumdar ArunabhaSaha Outline There are few problems with the Boltzmann algorithm It is not prefixed that how many times we need to iterate. Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Due to the presence of negative phase it takes a greater computation time This algorithm computes averages of two phases and take their difference. When these two correlations similar, the presence of sampling noise makes the difference more noisy. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference It runs very slow. Take very much time to learn. Weight explosion: If weights get too big too early, then the network get struck in one goodness optimum. - This shortcomings can be eliminated by sigmoid belief network Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 38 / 44
  • 84.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 85.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 86.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 87.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 88.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Cancer Detection Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 89.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Cancer Detection Loan Application Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 90.
    Boltzmann Machine Applications Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine stock market trend prediction Learning Algorithm for Boltzmann Machine character recognition Applications of Boltzmann Machine Face recognition Restricted Boltzmann Machine Internet Application Reference Cancer Detection Loan Application Decision making Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 39 / 44
  • 91.
    Restricted Boltzmann Machine BoltzmannMachine Ritajit Majumdar Arunabha Saha Outline Restricted Boltzmann Machine is a stochastic neural network (random behaviour when activated). It consist of one layer of visible units (neurons) and one layer of hidden units. Units in each layer have no connections between them and are connected to all other units in other layer. Connections between neurons are bidirectional and symmetric . This means that information flows in both directions during the training and during the usage of the network and that weights are the same in both directions. Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 40 / 44
  • 92.
    Boltzmann Machine How RBMWorks Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine First the network is trained by using some data set and setting the neurons on visible layer to match data points in this data set. Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 41 / 44
  • 93.
    Boltzmann Machine How RBMWorks Ritajit Majumdar Arunabha Saha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine First the network is trained by using some data set and setting the neurons on visible layer to match data points in this data set. After the network is trained we can use it on new unknown data to make classification of the data (this is known as unsupervised learning) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference November 6, 2013 41 / 44
  • 94.
    Continuous Restricted BoltzmannMachine Boltzmann Machine Ritajit Majumdar Arunabha Saha Outline Hopfield Net CRBM have very close implementation to original RBM with binomial neurons (0,1) as possible values of activation. Stochastic Hopfield Nets with Hidden Units Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Training data2 2 500 Reconstructed data training data were taken Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 42 / 44
  • 95.
    Boltzmann Machine Reference Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Geoffrey Hinton Neural Networks for Machine Learning www.coursera.org Geoffrey Hinton http://www.scholarpedia.org/article/Boltzmann_ machine Boltzmann Machine Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Simon Haykin, 2ed Geoffrey Hinton, David Ackley A learning Algorithm for Boltzmann Machine Cognitive Science 9, 147-169 (1985) Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 43 / 44
  • 96.
    Boltzmann Machine Ritajit Majumdar ArunabhaSaha Outline Hopfield Net Stochastic Hopfield Nets with Hidden Units Boltzmann Machine THANK YOU Learning Algorithm for Boltzmann Machine Applications of Boltzmann Machine Restricted Boltzmann Machine Reference Ritajit Majumdar Arunabha Saha (CU) Boltzmann Machine November 6, 2013 44 / 44