Self Organisation: Inspiring Neural Network & IT Design

903 views

Published on

In an attempt to build more sophisticated neural networks and other Information Technology (I.T.) products, the industry constantly turns to the world of Biology for inspiration. The most advanced
computers in the World today, are of course humans.

This paper looks at Self Organisation in the Human Nervous System and aims to highlight the means by which the understanding gained, from the study of this issue, can influence and inspire the design of Neural Networks and I.T. products and services.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
903
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Self Organisation: Inspiring Neural Network & IT Design

  1. 1. Self OrganisationInspiring Neural Network & IT Design www.oliviamoran.me
  2. 2. About The AuthorsOlivia Moran is a training specialist who Submitted For Cognitive Computing,specialises in E-Learning instructional Msc in Computing, University ofdesign and is a certified Moodle expert. Ulster 2006She has been working as a trainer andcourse developer for 3 years developing Authors included:and delivery training courses for traditionalclassroom, blended learning and E-learning.  Olivia Moran  Eric NicholsSelf Organisation: Inspiring Neural Network  Barry Feehily& IT Design was written as part of a group  Lisa Murphycollaboration. www.oliviamoran.me
  3. 3. Self Organisation: Inspiring Neural Networks & IT Design1. ABSTRACT 2. INTRODUCTIONIn an attempt to build more sophisticated neural The term self-organisation is used to describe thenetworks and other Information Technology (I.T.) process by which “Internal structures can evolveproducts, the industry constantly turns to the world of without the intervention of an external designer or theBiology for inspiration. The most advanced presence of some centralised form of internal control.computers in the World today, are of course humans. If the capacities of the system satisfy a number ofIt is therefore no wonder that engineers and constraints, it can develop a distributed form ofcomputer scientists invest such a large amount of internal structure through a process of self-their time examining theses biological machines and organisation” Cilliers (1998). In basic terms self-the way in which they operate. I.T. products are often organisation is a process that involves theconstructed based on the same principles or concepts organisation of group behaviour to achieve globalon which the human body is built. These concepts order. This process occurs through interactionsrelate to areas such as sensory perception and among the group and not through external influences.processing, the motor system and social cognition etc, According to Cilliers (1998) “This process is such thatthe list is endless. For the purpose of this report, structure is neither a passive reflection of the outside,however, only one issue will be explored in depth, nor a result of active, pre-programmed internalself-organisation. factors, but the result of a complex interaction between the environment, the present state of theThis report aims to highlight the means by which the system and the history of the system”.understanding gained, from the study of this issue,can influence and inspire the design of I.T. products The concept of self-organisation is easily illustratedand services. It will examine the role of sleep and its using an example from nature. Examples include theeffects on self-organisation. Subsequently, the reaction of hair dye to our hair and the type of activitydevelopment of the Nervous System (N.S.) and the that occurs as well as the growth of plants andimportance of self-organisation to the development animals and the creation of a sculpture by an artist.process will be explored in depth. This document will One of the more widely used examples is the hare andbriefly consider how connections are made between the lynx. A study was carried out and recorded by theneurons and furthermore how these can be rewired. Hudson Bay Trading Company in Canada betweenSelf-organisation occurs at a number of levels, which 1849 and 1930. This recorded and examined specificwill be highlighted. A comparison will be made statistics relating to the populations of hares andbetween the N.S. of invertebrates and vertebrates in lynxes. In this example, the lynx is the predator of thean attempt to determine the effect, if any, that the hare. It was concluded from this study that asizes of these systems exert on the self-organisation decrease in the number of prey, would cause aprocess. It will be illustrated how self-organisation corresponding decrease in the number of predators.can be computationally modeled. Finally, this This was due to the fact that a reduction in preydocument will give some thought to future work in resulted in limited food sources and so there was notthis field of research. enough food to sustain current predator numbers. After a period of time, the numbers of prey grewKEYWORDS: Cognitive Computing, Self-Organisation, because the amount of predators was low. However,Neural Networks, The Nervous System, Self- this replenished food stocks for the predators and soOrganising Maps. there numbers began to grow yet again. This process begins over again and continues in a cyclical manner. This behaviour is seen emerging from the interaction between the lynxes and hares.
  4. 4. Self Organisation: Inspiring Neural Networks & IT DesignThis self-organisation process also takes place in our deprivation is viewed as a lack of the necessarybodies. One can easily forget that our bodies are amount of sleep that your body requires for healthysome of the most complex systems around. functioning.Consequently it is only obvious that this is a goodplace in which to study the concept and gain a better The occurrence of sleep deprivation throughoutunderstanding of it. This document aims to explore modern day society is incredibly higher than thatthe concept of self-organisation in depth. Firstly the found four or five decades ago. This is partly due torole of sleep is considered and the part that it plays in our hectic lifestyles, jobs and of course electricalthe self-organisation process. Self-organisation is lighting. People are staying awake for longer lengthsextremely important in the development of the of time. Consequently, there has been a substantialnervous system. It is also crucial to understanding decrease in the average amount of sleep each personhow connections between neurons are made and gets. A person can be deprived of sleep by their ownrewired. This development process is explored at mind and body. Sleep is extremely important and islength. The occurrence of self-organisation at needed for regeneration of certain parts of the bodydifferent levels is considered briefly focusing on the in particular the brain so that it can function properly.single and networked cell levels. A comparison ismade of the N.S.’s of invertebrates and vertebrates When the body is asleep the brain goes through aand thought is given to how the size of these systems process that consists of four different stages calledmay impact on self-organisation. It is illustrated how the R.E.M. (Rapid Eye Movement) sleep cycles. R.E.M.self-organisation can be computationally modelled. sleep is the desirable sleep state characterised byFuture work in this area is also addressed. rapid movements of the eyes. At certain points of the sleep process, the brain is active in different ways.3. SLEEP AND ITS These can be identified using Electroencephalogram (E.E.G.) reader. In the first anROLE IN SELF- stages of sleep, the body starts to relax and the heart rate begins to slow. At this point people often feel asORGANISATION IN if they are falling or feel weightless.THE BRAIN During the second stage of sleep it becomes evident that the brain is not acting in the same way i.e. emitting the same brain waves, as when the body wasIn the past people taught of sleep as a dynamic and awake. This stage is where deep restful sleep occursdormant activity that was part of our every day lives. and the body reenergises itself. The body must goNowadays people are more aware that sleep can through a sufficient amount of R.E.M. cycles or elseactually affect our daily functioning along with our the body will be unable to reenergise itself.physical and mental health. A number of activities Consequently self-deprivation would result. Theoccur within the brain in order to prepare us for sleep. effects of this self-deprivation may include difficultyFirstly, within the brain, neurotransmitters, nerve- concentrating, being in a bad mood, reduced energysignalling chemicals, act on different neurons and and a greater risk of being in or causing an accident,control whether we are asleep or awake. These including fall-asleep crashes. Stage three and four areneurons are located in the brainstem, the part of the much deeper sleep states however, four is moreN.S. that connects the brain with the spinal cord. intense than three. These stages are often referred toHere they produce neurotransmitters that keep as slow-wave sleep or delta sleep. The reason why isdifferent parts of the brain active when a person is evident particularly in stage four where the E.E.G.awake. Other neurons located at the base of the reader records slow waves of high amplitude,brain, begin signalling the relevant neurons with the demonstrating a pattern of deep sleep and rhythmicbody gradually falling into a sleep state. If the later continuity.does not occur sleep deprivation results. Self-
  5. 5. Self Organisation: Inspiring Neural Networks & IT DesignResearch shows that sleep-deprivation has noticeable Scientists are now realising that sleep deprivation cannegative effects on things such as alertness and affect the whole body not only the brain. A study thatcognitive performance. According to Thomas et al was carried out by Dr. Eve Van Cauter from the(2000) this suggests a decrease in brain activity and University of Chicago showed that failing to get thefunction primarily in the thalamus, a subcortical right amount of sleep could affect the chemicalstructure involved in alertness and attention, as well balances in the body as well. The study looked at aas the prefrontal cortex, a region subserving alertness, male after four hours sleep for a total of six nights.attention, and higher-order cognitive processes. It is Results from blood tests showed strikingly similarseen that after extended periods of wakefulness or results to those expected from a person withreduced sleep, neurons can begin to malfunction. diabetes. The male’s ability to process blood sugarThis change in the neurons can have a visible affect on was reduced by a total of thirty percent, this in turna person’s behaviour. caused a drop in insulin levels. It was also reported that the male had specific levels of memoryOrgans such as muscles are able to regenerate impairment.themselves when a person is not asleep so long asthey are resting. In this circumstance the cerebral Scientists strive to come up with an answer to thecortex within the brain is not able to rest but rather question ‘how much sleep does an average personremains alert in a state of ‘quiet readiness’. This need in order for their brain to reenergise itself andsuggests that while some stages of sleep are a ideally self-organise’. This is a difficult one to answer.necessity for the regeneration of neurons others are It is different for everyone and is influenced by factorssuited to creating new memories and the formation of such as age and health. It is generally accepted thatsynaptic connections. babies need an average of nineteen hours a day while teenagers need a total of nine. Adults functionA study was carried out in an attempt to highlight the normally with approximately seven to eight hoursnegative effects of sleep deprivation. Seventeen however, certain individuals can limit themselves tomales over an eighty-five hour period of sleep- five hours while others may require ten hours ofdeprivation were examined. The subjects were sleep.observed four times every twenty-four hours. Duringthis period they were asked to complete a series of It is clear that sleep is absolutely necessary and thataddition and subtraction tasks. Polysomnographic without it our bodies would be unable to survive forexaminations confirmed that the subjects were long. One study involving rats demonstrates thisawake. After twenty-four hours it was reported that point effectively. Rats are seen to live for two tothere was a significant decrease in global C.M.R.glu, three years but because of sleep deprivation and notand dramatic falls in absolute regional C.M.R.glu in going through the R.E.M. cycle, the rats in theseveral cortical and subcortical structures. The main experiment only lived for a total of five weeks. Thechanges occurred primarily in the thalamus and rats developed low blood pressure, sores on their tailsprefrontal and posterior parietal cortices located near and paws as well as an impaired immune system.the front of the brain (See Appendix 1). Researchers in recent years have become a lot more interested in this area of study. They know thatFrom these experiments it was concluded that short- gaining a better understanding of sleep and all that itterm sleep deprivation produces global decreases in encompasses will in turn result in a greater insightbrain activity with larger reductions in activity in the and knowledge of the role that sleep plays in the self-distributed cortico-thalamic network mediating organisation process.attention and higher-order cognitive processes. Itwas also complementary to studies demonstratingdeactivation of these cortical regions during N.R.E.M.and R.E.M. sleep.
  6. 6. Self Organisation: Inspiring Neural Networks & IT Design4. THE NERVOUS After a short period a crease or fold appears in this plate, it begins to grow and a neural groove appears.SYSTEM Folding action continues until the creases meet and fuse together. This fusion results in the neural tube that eventually develops into the nervous system. IfThe human body is made up of trillions of cells that development goes according to plan the neural tubeinteract and work together to achieve certain closes completely. Failure to close could result inoutcomes. Numerous amounts of these cells join abnormalities such as Spina Bifida. Vesicles growforces in order to create complex systems such as the from the front end of the tube. These will eventuallyN.S. This system is found in both animals and people become a part of the C.N.S. During the entire processand is crucial to their survival as it facilitates all the cells in the N.S. comply with a strict set of rules.movement, therefore enabling them to respond to These rules determine exactly where each cell willchanges in their environment and adapt. The N.S. eventually end up and what purpose it will serve.consists of two main parts, the first includes the spinalcord and the brain and is known collectively as the Next, cell differentiation and division occurs. Mitosis,Central Nervous System (C.N.S.). The second part is the process by which cells divide and thus multiplethe Peripheral Nervous System (P.N.S.) and it takes in takes place at the inner part of the wall of the neuralall the bodies’ different nerves. tube. Firstly, the cells move away from the wall to develop further and then return to undergo mitosis.4.1 THE DEVELOPMENT This process of division results in a huge amount of new cells being formed and consequently theOF THE NERVOUS SYSTEM thickening of the neural tube wall. The vesicles also increase in size. The new cells will eventually develop into neurons or glial cells.Self-organisation is undertaken at different stages ofthe N.S. development process. According to Willshaw(2007) it is self-organisation that is responsible for 4.1.2 Cell Migration“Generating nerve cells of the right type, in the rightnumbers, in the right places and with the right The next major step in the development of the N.S. isconnections”. Such a procedure is highly complex and cell migration. Cell migration refers to the movementinvolves “cell division, cell migration, cell death and of cells away from where they first developed, tothe formation and withdrawal of synapses”. where they are needed. This process is an extremely complex one, requiring a high level of organisation. All cells must end up in the exact desired position. “In4.1.1 Cell Division the developing brain, for example, primitive neuronal cells migrate out of the neural tube and take upIn humans after the primitive cell layers are formed, residence in distinct layers, where they sendthe inner cells break into a layer of ectoderm and projections (axons and dendrites) through the layersendoderm. A new layer called the mesoderm grows of developing cells to their final targets with whichbetween these two layers which all then begin to they form specific connections, called synapses thatwork together to produce the notochord. The allow complex functions such as learning andnotochord is a cylindrical shaped structure that is memory” Cell Migration Consortium (2007).responsible for organising the ectoderm layer. Anumber of steps are followed in the achievement ofthis task. Chemicals are released from the notochord;these stimulate the ectoderm so that it begins todivide. This division leads to the creation of theneural plate (See Appendix 2).
  7. 7. Self Organisation: Inspiring Neural Networks & IT Design4.1.3 Cell Death (See Appendix 3). The main task of any synapse is the transformation of electrical impulses into chemicalCell death is a normal occurrence as well as a signals so that they can be transported. The“Fundamental and essential process in development” beginning of this conversion process is sparked byBähr (2006). It is necessary that some cells be what’s known as an action potential, which is insacrificed for the success of the entire process. There essence a nerve impulse. “The end part of an axonare many theories attempting to shed light on the splits into a fine arborisation. Each branch of itreason behind cell destruction. Such theories are terminates in a small end bulb almost touching thehighlighted by Willshaw (2007) and include “Failure of dendrites of neighbouring neurons” Zurada (1992).neurons to find their targets, failure to make the The nerve impulse travels down to the bottom of thecorrect connections, the elimination of entire axon where it stimulates the synaptic vesiclesstructures that may act as transient scaffolds, removal resulting in the release of neurotransmitter. Thisof transient branches of the tree of lineage and lack of flows into the synaptic cleft filling it up. The chemicaladequate innervation”. Such explanations fail to then makes its way towards the dendrites of the cell itaddress all the issues relating to cell death. is trying to communicate with. As a result of this, parts of the membrane open up. Through theseAnother hypothesis known as ‘The Neurotrophic openings ions can flow in and out.Hypothesis’ was constructed and is currently the mostlogical way in which to explain why and how cell This flow of ions results in a change in voltage that isdestruction occurs. “Its principal tenet is that the known as a postsynaptic potential. This potential cansurvival of developing neurons depends on the supply be excitatory or inhibitory. If an excitatory potential isof a neurotrophic factor that is synthesized in limiting created in the case of depolarising currents, thisamounts in their target fields” Davies (1996). If there usually leads to the production of a second actionis not enough neurotrophic factor present, the extra potential. However, inhibitory potentials as withneurons produced will not be able to survive and will hyperpolarising currents, inhibits any further actionsimply die. potential. It is important to note that sometimes impulses will not necessarily travel to another neuron. “The synapses thus help regulate and route the4.2 Neurons And Their constant flow of nerve impulses throughout the N.S.”Connections The World Book Encyclopedia (1991).Once the neurons find their desired position, 5. SELF- ORGANISATION ATconnections have to be established. Such connectionsaccommodate communication between the neurons.Each neuron is made up of an axon and dendrites thatare crucial to the entire communication process. Forexample, when cell A wishes to communicate with cell DIFFERENT LEVELSB, the following sequence of events occur; Cell A using Self-organisation occurs at both a networked cell andthe axon transmits a message. Cell B is able to receive a single cell level. The networked cell level isthis message via the dendrites that act like a receptor concerned with the construction of maps that detailantennae. The meeting point of the two cells is the connections that exist between the nerve cells.known as the synapse (See Appendix 3). On the other hand, at the single cell level, focus is onNeurotransmission is also dependant on chemicals the elimination of superinnervation from developingthat act as a neurotransmitter and the use of electric muscle.signals to get the message across to the other cell.A synapse is made up of three main parts, the axonterminal, the synaptic cleft and the dendrite spine
  8. 8. Self Organisation: Inspiring Neural Networks & IT Design5.1 NETWORKED CELL resemble those maps found namely in the human’s vertebrate visual system. “Topographic maps varyLEVEL: SELF-ORGANISING considerably from one person to another”. They serve their purpose in that “Projections from one area of theMAPS brain to another often preserve neighbour relationships so that an area smoothly andSelf-organising maps are a good example of how self- continuously maps the area which project to it”organisation occurs at a networked level. Such maps Bamford et al (2006).are highly ordered and consist of multiple amounts of Topographic maps have two main distinguishablenerve cells. This conclusion is according to Willshaw characteristics. Take for example, the organism(2007), a consequence of both electrophysiological Xenopus that is made up of recoding positions. Theseand anatomical experiments. The electrophysiological according to Willshaw (2007) “Can be distinguished,experiments are concerned with the identification of all arranged in topographic order. The othera receptive field, “An area in which stimulation leads important attribute of such maps is that they alwaysto response of a particular sensory neuron” Levine & have a specific orientation. All retinotectal maps areShefner (1991). On the other hand the anatomical arranged so that temporal retina projects to rostralexpirements focus on the “Mapping between two tectum and dorsal retina to medial tectum”.points in different structures … using axonal tracers.Tracers placed at one point in one structure typically According to Bamford et al (2006) these maps canlabel a small, circumscribed area in the target, the “Form in the absence of any electrical (spiking) activityspatial layout of points of administration being and mechanisms proposed include varied repulsion toreflected in the layout of points to which the tracers chemicals with graded expression across the targetgo in the target” Willshaw (2007). areas. However maps can be refined in the presence of electrical activity (the spread of connection fields5.1.1 Neural Maps reduced). It can be demonstrated that a combination of spatially correlated input, recurrent connections between target neurons and Hebbian learning canSeiffert & Lakhmi (2001) define neural maps as maps produce ordered projections”.which “Project data from some possibly high-dimensional input space on to a position in some From examination of these maps it has beenoutput space”. These neural maps are made up of concluded that “Connections cannot be made byneuronal groups all of which are connected. “Two means of a simple set of instructions specifying whichfunctionally different neural maps connected by re- cell connects to which other cell, more likely, theentry form a classification couple. Each map populations of cells self-organise their connections soindependently receives signals from other brain maps as to ensure the correct overall pattern” Willshawor from the world. Functions and activities in one map (2007). A better understanding of these connectionsare connected and correlated with those in another that form under a process of self-organisation willmap. For example an input could be vision and the undoubtedly lead to the creation of more biologicallyother from touch” Clancey et al (1994). In basic terms plausible neural networks.neural maps are simply a projection of one twodimensional area onto another. 5.2 SINGLE CELL LEVEL –5.1.2 Topographic Maps ELIMINATION OFThe neural network is capable of being trained SUPERINNERVATIONthrough unsupervised learning. In such circumstancesthe neural network can produce maps that still retain FROM DEVELOPINGtheir topological features (See Appendix 5). Thesemaps find their inspiration from humans and MUSCLE
  9. 9. Self Organisation: Inspiring Neural Networks & IT DesignMany models exist which put forward different Invertebrates are animals sharing one commonarguments that claim to offer a plausible reason for characteristic and that is they do not have a backbonethe elimination of superinnervation during the or spine. On the other hand vertebrates are all thosedevelopment of muscles. The most widely accepted other animals who possess this spinal columnof these is the ‘Dual Constraint Model’ Bennett & structure as part of their anatomy as in the case ofRobinson (1989). This model which “Combines fish, birds, reptiles and of course humans. Freemancompetition for a pre-synaptic resource with (2005) points out that the “Architectures of the C.N.S.competition for a post-synaptic resource, has been of intelligent invertebrate animals differ markedlyshown to be superior to others with only one type of from those in vertebrate animals”. The N.S. of thecompetition” Rasmussen & Willshaw (1993). When invertebrates are fairly simple in construction inmuscle fibres are being developed, they are contrast to the vertebrates. Take for example, an“Superinnervated and this pattern is transformed into invertebrate such as the bee or the octopus that hasone of single innervation after a few weeks” Willshaw “Parallel chains of neurons resembling a ladder(2007). The length of time needed to complete this located ventral to the digestive system, from whichprocess as well as the total amount of elimination that and to which the axons of motor and sensory nervestakes place, differs depending on the section of the extend” Freeman (2005). They may have some typeN.S. where it occurs. of eyes and mouth in which case are “Serviced by large collections of neurons forming the dorsalThis theory basically operates on the tenant that cerebrum. Axons form bi-directional connections withmotor neurons have a particular capacity “For the ventral nerve cords around the gut, so that themaintaining the structure and activity of its terminals, esophagus runs through the brain. Perhaps this is whywhich is shared out among them” Willshaw (1981). all higher invertebrates are restricted to a liquid diet,Each terminal has a survival strength, however this is lest they rupture their brains by swallowing solid food”dynamic and is constantly regulated and fine-tuned. Freeman (2005).Those terminals with a high level of strength are givenprecedent over the feeble terminals. Consequently, The architecture of the vertebrates is such that it doesthe survival strength of the endplate of the stronger not have to deal with a limitation of this nature. “Theterminals is increased to the detriment of the weaker C.N.S. forms by invagination of the dorsal surface andones. creates the neural tube. The posterior part forms the spinal cord while the most anterior part forms theSuch a theory would lead one to conclude that the brain” Freeman (2005). Freeman (2005) also arguesmuscle fibres and the construction of their that despite all the differences that might exist withconnections as well as the pattern that they follow, the architecture of the C.N.S. of both animal typesare the result of a process of self-organisation under they do however, “Share the ladder-like architecturehighly competitive conditions. The patterns are of invertebrates”.therefore not formed by instructions specified in the Numerous studies on invertebrates have beengenome. completed concerning the role of organisation. Research evidence suggests that the Drosophila,6. THE NERVOUS commonly known as the fruit fly displays very “Precise and inflexible organisation” Willshaw (2007). ThisSYSTEM OF would lead one to conclude that self-organisation in this small and simple N.S. of the fruit fly, is in aVERTEBRATES AND position whereby the “Genome can afford to specify precisely all the parameters values needed which haveINVERTEBRATES a smaller number of neurons” Willshaw (2007). Both small and large N.S.’s seem to display the ability to self-organise.
  10. 10. Self Organisation: Inspiring Neural Networks & IT Design7. Computationally As every input node is connected to every output node, adding nodes to the system causes the networkModelling Self- to grow exponentially. The amount of computation required to calculate large systems can quicklyOrganisation become too data-intensive for equations to be solved within a reasonable time-scale. As an understanding of biological systems becomes more detailed, theKohonen created a computational model for self- algorithms for describing such systems require moreorganisation in 1981. Bruske & Sommer (1995) states computation. At its most detailed, if quantum“Kohonen’s self-organizing feature maps, besides back confinement can be proven to exist in the interactionspropagation networks, are now the most popular and within and between neurons, then quantumsuccessful types of artificial neural networks”. mechanics “Represents the ultimate tool to theKohonen’s model, maps every input node to every modelling of bio molecular systems” Chung (2007).output node (See Appendix 6, Figure 1). The first stepin Kohonen’s algorithm is the initialisation the However, Chung writes that a quantum approach issynaptic weights, which can be set to random values. “Formidable and is an extremely time consumingThe next step involves finding the winning neuron by process, even with some simplifying assumptions, itscalculating the Euclidean distance between input and applications are limited to very small systems atoutput neurons. Kohonen (1997) discovered that one present”. As the speed of computational machinescould find the ‘winning’ neuron by using the following increases and we are “Equipped with powerfulformulae: computing techniques and high-performance sensorsBest matching node = mini {||x-mi||} = and actuators, we want to solve much more complex (highly non-linear and high-dimensional) problems”  x  mij  n 2 Kecman (2001). This relates significantly to self- j j 1 organising systems, as the growth of these systems isIn this formula, x represents the input and m exponential.represents the output map. The neuron pair with thesmallest Euclidean distance, the closest output node Kohonen’s (2001) argues that self-organising maps arerepresents the winning neuron on the output map useful for classification. On the other hand, in(See Appendix 6, Figure 2). isolation they are not as good as other methods. Lisboa (1992) compared Kohonen’s network to otherThe value of the closest output node is adjusted so classifiers (See Appendix 7). Kohonen’s self-organisingthat a smaller Euclidean distance results. The nodes in map had the worst performance of the six that werethe closest output node’s neighbourhood are also compared using handwriting digit recognition as a testupdated (See Appendix 6, Figure 3). The weight of the case.winning neuron, as well as the weights of all theneighbouring neurons, is adjusted with the formulae Kohonen’s network was not the first computational(Kohonen 1997). self-organising map. According to Grossberg (1994), in 1976 Grossberg wrote a mathematical model of a self-mi(t+1) = mi(t) + hci(t)[x(t) – mi(t)] organising feature map, where “Neurobiological modelling rules were articulated and restated in theIn this formula, t represents an integer representing a familiar SOFM formalism as an algebraic winner-take-time interval and hci(t) represents a ‘neighbourhood all dot product rule, and a self-normalizing synapticfunction’. The process then returns to the start (See weight change rule whose weights change only if theyAppendix 6, Figure 1) with the updated weights. The are in the neighbourhood of the winner.” Five yearsloop continues until the network sufficiently matches later, Kohonen (2001) sought to “Generalise and atthe target system. the same time ultimately simplify his (Grossberg’s) system description”. With this simplification, further
  11. 11. Self Organisation: Inspiring Neural Networks & IT Designwork can be achieved on self-organisation using above to greatly enhance the computationalKohonen’s network as a baseline. modelling of self-organising maps in Matlab. As new self-organising algorithms are created, further MatlabExamples of this can be found in both hardware and files can be written to continue computationalsoftware implementations of self-organising maps. modelling of biological systems with textual andThe construction of self-organising maps can be a very graphical outputs.time-consuming process because every input must bemapped to every output node. Martinez et al (2002)have found that using systolic arrays with Kohonen’s 8. FUTURE WORKnetwork greatly reduces computational time, as eachnode can be computed in parallel on different Artificial neural networks are modelled on ourprocessors. perception of the way the brain processes information. As technology develops, neurologistsLinaker and Niklasson (2000) used a neighbourhood will be able to find a more definitive understanding tofunction of 0 (no neighbours) and an altered Kohonen self-organisation in the brain while biophysicists find anetwork, called a Resource Allocating Vector better representation of our brain at a molecular andQuantizer (RAVQ), for a robot to successfully learn its atomic level. These findings can then be used toenvironment. The main difference between the self- develop better theories and technologies for artificialorganising map that Kohonen authored and the RAVQ neural networks.is that the latter’s output map is dynamic. The RAVQmore closely mimics biological systems in that output Intelligent systems’ current (third) generation modelsnodes can be created and mapped. While Kohonen’s take past work on neural networks and “Raises theoriginal network is not great for classification, as level of biological realism by using individual spikes”shown in Table 1, enhancements such as the RAVQ Vreeken (2002). A current active area of research incan make self-organisation more realistic and give neural networks that can play a vital role in self-greater performance to Kohonen’s network. organisation is in dynamic synapses. The models above all use static synapses, whose values onlySelf-organising systems can be modelled using any change after the Euclidean distance has been found.development environment and language that has With dynamic synapses, the synaptic weight canaccess to basic mathematical libraries. Matlab has change by up to a few hundred percent dependentlibraries that provide functions not only for upon the inputs to the synapses, which has beenmathematics, but also specifically for self-organising found to be the case in biological synapses.maps. Three such functions include:  newc – returns a new competitive layer 9. CONCLUSION  newsom – returns a new self-organising map This document explored in depth the issue of self- organisation. It looked at sleep and how sleep or a  newlvq – returns a new learning vector lack of it impacts on the body’s ability to self-organise. quantisation network for classification It examined the role of self-organisation in the development of the N.S. as well as the connectionsThese functions can be used in Matlab’s scripting between different neurons. It considered briefly self-language, as well as graphically using Matlab’s Neural organisation at different levels namely the single andNetwork Toolbox graphical user interface. A great networked cell levels.feature of Matlab is the ability to extend itsfunctionality with the use of Matlab *.m files. Anexample of this is the SOM Toolbox, a set of 141Matlab files written by the Helsinki University ofTechnology. These files build upon the functions
  12. 12. Self Organisation: Inspiring Neural Networks & IT DesignThe N.S.’s of invertebrates and vertebrates were during Polyneuronal Innervation of Muscle Cells:analysed to determine whether or not the size of both Atrophic Hypothesis” Royal Society Publishing 235,these systems have any noticeable effect on self- pp. 299-320.organisation. This document illustrated how self-organisation could be modeled computationally. Bruske, J. & Sommer, G. (1995) “Dynamic CellIdeas for future work were also put forward. Structure Learns Perfectly Topology Preserving Map” Neural Computation 7(4), pp. 845-865.The examination of such a process can aid theconstruction of neural networks, especially those that CELL MIGRATION CONSORTIUM (2007) “Overview ofaim to be self-organising or self-modifying. Such a the Migration Process” *Internet+, Date Accessed: 17network would be able to adapt to changes in the March 2007, URL:external environment when required. Shadbolt http://www.cellmigration.org/science/.(2004) argues strongly that the “Insights from onesubject inform the thinking in another . . . The ultimate Chung, S. H. (2007) “Large-Scale Dynamical Modelsambition is an understanding of the C.N.S.”, advances and Estimation for Permeation in Biologicalin the field of science often result in complimentary Membrane Ion Channels” Proceedings of IEEEgains in the area of computing or vice versa. Estimation and Control of Large Scale Systems, 20(20), pp.2-23.There is no doubt that the computing world seeks itsinspiration from the world of biology. “We see Cilliers, P. (1998) “Complexity and Postmodernism:complexity all around us in the natural world – from Understanding Complex Systems” London: Routledge.the cytology and fine structures of cells to theorganization of the nervous system . . . Biological Clancey, W. J, & Smoliar, S.W. & Stefik, M. J. (1994)systems cope with and glory in complexity – they seem “Contemplating minds: A forum for artificialto scale, to be robust and inherently adaptable at the intelligence” London: The M.I.T. Press.system level . . . Nature might provide the most directinspiration” Shadbolt (2004). There is no doubt that DAVIES, A. M. (1996) “The Neurotrophic Hypothesis:“An attempt to imitate a biological phenomenon is Where Does it Stand” Biological Sciences 351(1338),spawning innovative system designs in an emerging pp.389-394.alternative computational paradigm with both specificand yet unexplored potential” Bamford et al (2006). FREEMAN, W. J. (2005) “NDN, Volume Transmission and Self-Organization in Brain Dynamics” Journal ofBIBILOGRAPHY Integrative Neuroscience 4(4), pp. 407-421. Grossberg, S. (1994) “Letter to the editor:bÄhr, M. (2006) “Brain Repair – Advances in Physiological interpretation of the self-organizing mapExperimental Medicine and Biology” American Journal algorithm” *online+, Date Accessed: 22 March 2007,of Neuroradiology 27(9), pp. 2014. URL: http://www.cns.bu.edu/Profiles/ Grossberg/Gro1994KohonenLetter.pdf.Bamford, S. & Murray, A. & Willshaw, D. J. (2006)“Synaptic Rewiring in Neuromorphic VLSI for Kecman, V. (2001) “Learning and Soft Computing”Topographic Map Formation” *Internet+, Date Cambridge: MIT Press.Accessed 15 April 2007, URL:http://www.see.ed.ac.uk/~s0454958/interimreport.p Kohonen, T. (1997) “Self-Organizing Maps” Berlin:df. Springer-Verlag.BENNETT, M. R. & ROBINSON, J. (1989) “Growth and Levine, M. W. & Shefner, J. M. (1991) “FundamentalsElimination of Nerve Terminals at Synaptic Sites of Sensation and Perception” 2nd ed. California: Brooks & Cole.
  13. 13. Self Organisation: Inspiring Neural Networks & IT DesignLinaker, F. & Niklasson, L. (2000) “Time Series THE WORLD BOOK ENCYCLOPEDIA (1991) “TheSegmentation Using an Adaptive Resource Allocating Nervous System” London: World Book Inc.Vector Quantization Network Based on ChangeDetection” IEEE Computer Society, Proceedings of the Thomas, M. & Sing, H. & Belenky, G. & Holcomb, H. &International Joint Conference on Neural Networks. Mayberg, H. & Dannals, R. & Wagner, H. & Thorne, D. & Popp, K. & Rowland, L. & Welsh, A. & Balwinski, S. &Lisboa, P. J. G. (1992) “ Neural Networks – current Redmond, D. (2000) “Neural Basis of Alertness andapplications” London: Chapman & Hall. Cognitive Performance Impairments During Sleepiness - Effects of 24 Hours of Sleep Deprivation on WakingMartinez, P. & Aguilar, P. L. & Perez, R. M. & Plaza, A. Human Regional Brain Activity” Journal of Sleep(2002) “Systolic S.O.M. Neural Network for Research 9(4), pp. 335-352.Hyperspectral Image Classification” in Zhang, D. & Pal,S. K. (2002) “Neural Networks and Systolic Array Vreeken, J. (2002) “Spiking Neural Networks - AnDesign” London: World Scientific Publishing Co. Introduction” Technical Report UU-CS-2003-008, Institute for Information and Computing Sciences,RASMUSSEN, C. E. & WILLSHAW, D. J. (1993) “Pre- Utrecht University.Synaptic and Post-Synaptic Competition in Models for WILLSHAW, D. J. (1981) “The Establishment and thethe Development of Neuromuscular Connections” Subsequent Elimination of Polyneuronal InnervationBiological Cybernetics 68, pp. 409-419. of Developing Muscle: Theoretical Considerations” Biological Sciences 212 (1187), pp. 233-252.Seiffert, U. & Lakhmi, C.J. (2001) “Self-OrganizingNeural Networks: Recent Advances and Applications WILLSHAW, D. J. (2007) “Self-Organisation in the(Studies in Fuzziness and Soft Computing)” New York: Nervous System” Foresight *Internet+, Date Accessed:Physica-Verlag. 17 March 2007, URL: http://www.foresight.gov.uk.SHADBOLT, N. (2004) “From the Editor in Chief: ZURADA, J. M. (1992) “Introduction to Artificial NeuralNature-Inspired Computing” IEEE Intelligent Systems Systems” New York: West Publishing Company.19(1), pp.2-3. www.businesscompany.com

×