PowerPoint presentation of my 2015 research project.

1. Mathematical model
determining the optimal
parameters for the highest
possible learning efficiency in
Artificial Neural Networks
Sophia Kioulaphides The Bronx High School of Science
Sophia Kioulaphides The Bronx High School of Science

2. BACKGROUND
 The human brain and its capabilities are still unsolved mysteries.
 Scientists conceptualize technologies that will replicate the
behavior of the human brain.
 Various companies, including Facebook, have shown a keen
interest in understanding the thinking processes of the brain.
 Computers built on a system of neurons can learn and remember as
efficiently and quickly as the human brain.
 Neurons, the “building blocks” of the brain are connected by
synapses and perform different functions together through a
“chain-like process”.
 Neurons fire action potentials (electric pulses), sending messages
and causing the phenomena of learning and memory retrieval.
 This process is similar to how a metal wire conducts a current.
 This allows the Artificial Neural Network (ANN) to be formed.
Introduction Methodology Results Discussion

3. LITERATURE REVIEW
 The first ANN consisted of only one neuron.
 The model described mathematically human neural behavior.
 The individual properties of neurons are also collective!
 The ANN behaves very much like a ferromagnetic system, which is
how metals become magnets. One kind of ferromagnetic system is
the Ising spin system or Ising Model.
 The state of the Ising Model units is either “up” or “down”, similar
to how a neuron either completely fires or doesn’t fire at all.
 When artificial neurons are connected, they tend towards the most
ordered state, similar to how biological neurons tend to a memory.
 There has to be a low level of disorder to maximize learning.
Introduction Methodology Results Discussion

4. Introduction Methodology Results Discussion
LINK BETWEEN BIOLOGICAL AND PHYSICAL MODELS
Biological Physical
The membrane potential
optimizes towards the stable
global minimum of the brain.
The energy function describes
optimization in the brain in
mathematical terms.
Order: The extracellular electric
fields create order in the brain and
help with memory formation.
Order: The external magnetic field
affects cooperative magnetism in
a magnetic system.
Disorder: The weights of the
inputs of neurons are altered and
changes how the output is
reached.
Disorder: The pseudo-temperature
causes the system to tend to the
most stable unit.

5. RESEARCH PROBLEM
The fundamental question of my research project was, “Under what
conditions does the Ising Model (a model for the ANN) tend to the so-
called global minimum, or what we would commonly call a memory, the
fastest?” In other words, what do we need to do in order to maximize
learning?
Those optimal conditions are necessary for the ANN to retain a
particular memory the fastest, to store the maximum amount of
information in the most efficient manner.
The continuing study of those conditions will perpetuate the use and
further development of ANN computers that aim to operate with the
same efficiency as the human brain.
Introduction Methodology Results Discussion

6. RESEARCH HYPOTHESIS
Now, for some technical talk:
The pseudo-temperatures, denoted by T, optimize the system
when they are below the Curie Point, represented by TC (about
2.27 K). However, learning is faster at temperatures on the higher
end of 0 K to 2.27 K. So it is most likely that the fastest learning
will occur near 2.27 K, but once the pseudo-temperature goes
above this value, the system will not tend towards any particular
value.
Introduction Methodology Results Discussion

7. SIGNIFICANCE
 The theoretical limit for efficient learning had to be found.
 Once that is known, it gives engineers a head start in creating a new
generation of neural computers.
 These computers will be able to learn, make decisions, and
remember just like humans.
 Humans make use of these abilities to perform everyday tasks such
as discerning handwriting, and even to save lives by detecting the
presence of a bomb.
 In addition to studying the functions of a healthy brain, further
studies of the brain that is affected by neurodegenerative
disorders can be pursued, possibly leading to a cure.
Introduction Methodology Results Discussion

8. HOW TO MODEL A LEARNING PROCESS
 Recent studies dealt with the biology of neural networks.
 They experimentally showed how neural networks respond to
chemical impulses, such as drugs; when drugs are profusely
consumed, the firing rate of neurons is rapidly increased.
 We need to break the complex neural network down to the simplest
model that still retains all the properties of neurons.
 If we represent the brain as a simple computer, we see its basic
binary function, where “neurons” are either firing or not firing at
all; in other words, the familiar 0 vs. 1 computing relationship.
 A neural network stores information, and the maximum storage
occurs in an ordered system.
 Parameters of the ANN will have values that optimize learning
capabilities and maximize the phenomena of learning.
Introduction Methodology Results Discussion

9. Introduction Methodology Results Discussion
 The Energy Function  Mathematically shows optimization.
 The variable J represents the strength of the connection between
two neurons Si and Sj.
 The variable h or H represents the external magnetic field that is
acting on one particular “neuron”.
Ok. Here comes the math.

10. Introduction Methodology Results Discussion
 The strength of the synapses, J
 N is the number of neurons in the system
 μ is the number from 1 to p assigned to a specific memory.
 The magnetic field, h
 J is the strength of the synapses.
 Θi is the action potential that the system has to overcome to
fire a message.
ORDER PARAMETERS

11.  The entropy, S, is the degree of disorder in the system.
 kB is the Boltzmann Constant, which is the relationship
between the temperature and energy of one neuron.
 n, just like N, is the number of neurons in the system.
 The temperature, T, is defined as the reciprocal of the derivative
of the system’s entropy with respect to the system’s total neural
energy.
 The Boltzmann Distribution, β, is the level of disorder that
responds to an increase in energy.
Introduction Methodology Results Discussion
DISORDER PARAMETER

12.  In short, learning is order; entropy is disorder.
 Learning is never pure  There is never perfect order.
 Order and disorder are connected because they coexist.
 The order parameters show how we can come to the
most ordered state of our mental processes.
 Disorder arises because some neurons do not connect
entirely, when the connection on which the message
is being transmitted is faulty.
Introduction Methodology Results Discussion
Back to English:

13.  The Metropolis-Hastings
Algorithm
 A type of Monte Carlo
algorithm.
 Shows that if the pseudo-
temperature is lowered
slowly, then thermal
equilibrium is reached.
 The Boltzmann
Distribution chose the
pseudo-temperatures.
Introduction Methodology Results Discussion
OPTIMIZING ALGORITHM

14. THE SIMPLEST MODEL
 The smallest dimensions of the ANN that still capture essential
neural features  3x3 matrix.
 The configurations of the matrix have to do with how many
“neurons” are completely firing or not firing at all and where they
are located in the matrix.
 Each configuration was given a number from 1 to 2N, (N is the
number of “neurons”—in this case, 9). The numbers were 1 to 512.
 J was held constant at -0.7.
 H was held constant at 1.
 T was increased by increments of 0.03 in order to get a steady
curve.
Introduction Methodology Results Discussion
“Everything should be made as simple
as possible, but not simpler.”
–Albert Einstein

15. Introduction Methodology Results Discussion
-2
-1.5
-1
-0.5
0
0.5
1
1.5
1
32
63
94
125
156
187
218
249
280
311
342
373
404
435
466
497
TotalNeuralEnergy(E)
Configuration #
Series1
-1.5
-1
-0.5
0
0.5
1
1.5
2
1
33
65
97
129
161
193
225
257
289
321
353
385
417
449
481
TotalNeuralEnergy(E)
Configuration #
Series1

16. Introduction Methodology Results Discussion
0
5
10
15
20
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81
#ofSteps
Pseudo Temperature #
Series1
0
0.5
1
1.5
2
2.5
3
3.5
4
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81
Frequency
Pseudo Temperature #
Series1

17. Introduction Methodology Results Discussion
WHAT IS NEW?
 The goal of this project was to create guidelines that would help
scientists and engineers find the highest possible learning efficiency
of a neural network.
 There are optimal parameters for learning efficiency; we just need to
find them. I have developed a theoretical model, so now it is the time
to move on to the application stage.
 There is a new understanding of how quickly a memory can be
retrieved; if we know that, we can improve the retrieval mechanism.
 The simplest, smallest model that behaves the same way as a
biological neural network has been created.
 This model was able to learn and retrieve memories, both of which
are phenomena characteristic of the human brain.

18. WHAT REMAINS TO BE LEARNED?
 The small 3x3 model only captures general features of the
neuron.
 A more sophisticated model would capture the detailed
properties of a neuron.
 What else would you like to know?
Introduction Methodology Results Discussion

19. References
[1] Markoff, J. (2013, Dec 28). Brainlike Computers, Learning From Experience. The New York Times. Retrieved from http://mobile.nytimes.com/2013/12/29/science/brainlike-computers-learning-from-
experience.html?_r=1
[2] Neural Networks for Machine Learning. Coursera. Retrieved from https://www.coursera.org/course/neuralnets
[3] Obama, B. (2013, Apr 2). Remarks by the President on the BRAIN Initiative and American Innovation. The White House. Retrieved from http://www.whitehouse.gov/the-press-
office/2013/04/02/remarks-president-brain-initiative-and-american-innovation
[4] Cumming, J.G. (2010, Dec 23). Let’s build Babbage’s ultimate mechanical computer. NewScientist Opinion. (2791). Retrieved from http://www.newscientist.com/article/mg20827915.500-lets-build-
babbages-ultimate-mechanical-computer.html#.VCDuV0sq37U
[5] Conventional Computer Organization. Retrieved from http://people.cs.clemson.edu/~turner/courses/cs428/current/webct/content/pz/ch2/ch2_1.html
[6] Computer Parts: How do quantum computers differ from conventional computers? Discovery. Retrieved from http://curiosity.discovery.com/question/quantum-differ-conventional-computers
[7] DeGroff, D. and Neelakanta, P.S. (1994, Jan 7). Neural Network Modeling: Statistical Mechanics and Cybernetic Perspectives. Earthweb. Retrieved from
http://www.ru.lv/~peter/zinatne/ebooks/Neural_Network_Modeling.pdf
[8] Siganos, D. Why neural networks? Retrieved from http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol1/ds12/article1.html
[9] Siegelmann, H.T. and Sontag, E.D. (1991, Jun 17). Turing Computability with Neural Nets. Appl. Math. Lett. 0 (0). Retrieved from http://www.math.rutgers.edu/~sontag/FTP_DIR/aml-turing.pdf
[10] (2013, Oct 27). Neuroscientists discover new ‘mini-neural computer’ in the brain. ScienceDaily. Retrieved from http://www.sciencedaily.com/releases/2013/10/131027185027.htm
[11] Bushwick, S. (2012, Jan 19). How exactly do neurons pass signals through your nervous system? io9. Retrieved from http://io9.com/5877531/how-exactly-do-neurons-pass-signals-through-your-
nervous-system
[12] Sompolinsky, H. (1988, Dec). Statistical Mechanics of Neural Networks. Physics Today. Retreived from http://neurophysics.huji.ac.il/sites/default/files/Sompolinsky_PhysicsToday.pdf
[13] The biological model: The human brain. (2004). Neural Networks with Java. Retrieved from http://www.nnwj.de/biological-model-human-brain.html
[14] Kuhn, Michael. Manual for the implementation of neural networks in MATLAB. Norderstedt: GRIN Verlag GmbH, 2005. Print.
[15] Introduction to Neural Networks. Retrieved from http://gandalf.psych.umn.edu/users/kersten/kersten-
lab/courses/NeuralNetworksKoreaUF2012/MathematicaNotebooks/Lect_12_Hopfield/Lect_12_Hopfield.nb.pdf
[16] Svitil, K. (2011, Feb 2). Neurobiologists Find that Weak Electric Fields in the Brain Help Neurons Fire Together. Caltech. Retrieved from http://www.caltech.edu/content/neurobiologists-find-weak-
electrical-fields-brain-help-neurons-fire-together
[17] Cooperative Magnetism. Retrieved from http://www.chemie-biologie.uni-siegen.de/ac/hjd/lehre/ws0708/seminar_ws0708/klotz_zusammenfassung_cooperative_magnetism_korr_.pdf
[18] Simon, B. The Ising Model. Retrieved from http://math.arizona.edu/~tgk/541/chap1.pdf
[19] Joo, J.M. (2006). A Virtual Lab to Visualize the Performance of the Hopfield’s Neural Network for Associative Content-Addressable Memory. Revista de Investigación de Física. 9 (pp. 36-45).
Retrieved from http://www.rif-fisica.org/images/4/40/36-45_Montenegro.pdf
[20] Hopfield, J.J. (1982, Jan 15). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences of the United States of
America. 79 (pp. 2554-2558). Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC346238/pdf/pnas00447-0135.pdf
[21] Coxworth, B. (2011, Nov 24).“Analogue computer chip mimics brain’s neural function.” Gizmag. Retrieved from http://www.gizmag.com/computer-chip-mimics-neurons/20608/
[22] Erickson, C. Neurons and Neurotransmitters: The “Brains” of the Nervous System. The University of Texas. Retrieved from http://www.utexas.edu/research/asrec/neuron.html
[23] Hathaway, B. (2013, Feb 20). Human cognition depends upon slow-firing neurons, Yale researchers find. YaleNews. Retrieved from http://news.yale.edu/2013/02/20/human-cognition-depends-
upon-slow-firing-neurons-yale-researchers-find
[24] Shonkwiler, Ronald W. and Mendivil, Franklin. Explorations in Monte Carlo Methods. New York: Springer Dordrecht Heidelberg, 2000. Print.
[25] Kumar, P.M. and Torr, P.H.S. Fast Memory-Efficient Generalized Belief Propagation. University of Oxford. Retrieved from http://www.robots.ox.ac.uk/~tvg/publications/old_stuff/2006/GBP.pdf