SlideShare a Scribd company logo

Neural Network Design: Chapter 18 Grossberg Network

•Download as PPTX, PDF•
•
Jason Tsai
Jason Tsai

Lecture for Neural Networks study group held on January 11, 2020. Reference book: http://hagan.okstate.edu/nnd.html Video: https://youtu.be/H4NKgliTFUw Initiated by Taiwan AI Group (https://www.facebook.com/groups/Taiwan.AI.Group/permalink/2017771298545301/)

Technology
1 of 44
Chapter 18 Grossberg Network Jason Tsai (蔡志順) January 11, 2020 Mozilla Community Space Taipei
*Copyright Notice: Most materials from this presentation are taken from the book “Neural Network Design 2nd edition” authored by Martin T. Hagan, Howard B. Demuth, Mark Hudson Beale and Orlando De Jesús. The other quoted sources are mentioned in the respective slides. This presentation itself adopts Creative Commons license.
A Quote from this Chapter “Although the original inspiration for the field of artificial neural networks came from biology, at times we forget to look back to biology for new ideas. It will be the blending of biology, mathematics, psychology and other disciplines that will provide the maximum growth in our understanding of neural networks.” 「雖然類神經網路這一領域的最初靈感來自生物學， 但有時我們會忘了回顧生物學以啟迪新的想法。融合 生物學、數學、心理學和其他學科將使我們對神經網 路的理解得以最大的發展。」
Vision: Eyeball & Retina
Visual Pathway
Photograph of the Retina
Blind Spot *Figure adopted from https://bit.ly/39iUtNQ
Test for the Blind Spot Look at the blue circle on the left side of above figure with your right eye while covering your left eye and move your head back and forth perpendicularly to the circle to find the point at which the circle on the right will disappear from your field of vision.
Imperfections in Retinal Uptake
Compensatory Processing
Visual Illusions
Neon Color Spreading *Figure adopted from https://bit.ly/37xDyWg
Oriented Receptive Field *Figure adopted from https://bit.ly/2rPXGDF & https://bit.ly/2tmGg1X
Oriented Receptive Field (cont.)
Vision Normalization
Brightness Contrast
Lateral Inhibition *Figure adopted from https://bit.ly/2yaat37
Leaky Integrate-and-Fire *Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
Leaky Integrator
*Solution to 1st order ODE *Credit to Schwartz Lyu
Leaky Integrator Response
Shunting Model Here p+, p-, b+, are b- are nonnegative values Refer to slide 28: Analysis of Normalization
Shunting Model Response
Grossberg Competitive Network
Layer 1
Characteristics of Layer 1 • The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. • The output of Layer 1 is a normalized version of the input pattern. • The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. • The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.
Operation of Layer 1
Analysis of Normalization
Example of Layer 1 Response The response of the network maintains the relative intensities of the inputs, while limiting the total response.
Layer 2
Characteristics of Layer 2 • As in the Hamming and Kohonen networks, the inputs to Layer 2 are the inner products between the prototype patterns (rows of the weight matrix W2) and the output of Layer 1 (normalized input pattern). • The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). • The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated).
Layer 2 Operation
Example of Layer 2 Response
Choice of Transfer Function
Sigmoid Transfer Function • A sigmoid function is faster-than-linear for small signals, approximately linear for intermediate signals and slower-than-linear for large signals. • When a sigmoid transfer function is used in Layer 2, the pattern is contrast enhanced; larger values are amplified, and smaller values are attenuated. • All initial neuron outputs that are less than a certain level decay to 0. • This merges the noise suppression of the faster- than-linear transfer functions with the perfect storage produced by linear transfer functions.
Adaptive Weights W2
Example of W2 Response
Equivalence to Discrete-time Rule
Relation to Kohonen Rule
Relation to Kohonen Rule (Cont.)
Three major differences: • The Grossberg network is a continuous-time network. • Layer 1 of the Grossberg network automatically normalizes the input vectors. • Layer 2 of the Grossberg can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network. This soft competition allows more than one neuron in Layer 2 to learn. This causes the Grossberg network to operate as a feature map. Grossberg vs. Kohonen Networks
Potential Issues • One key problem of the Grossberg network is the stability of learning. As more inputs are applied to the network, there is no guarantee that the weight matrix will eventually converge (form stable clusters / categories). [Refer to Chapter 19: Adaptive Resonance Theory, ART] • Another problem is the stability of the differential equations that implement Grossberg’s continuous- time competitive recurrent network. The output of a recurrent network could converge, oscillate, or even diverge. [Refer to Chapter 20: Stability]
Basic ART Architecture (Ch.19) Add-ons: Layer 2 to Layer 1 expectations The orienting subsystem (reset) Modified gain control Let’s move on…
References

Recommended

Adaptive Resonance Theory
Adaptive Resonance TheoryAdaptive Resonance Theory
Adaptive Resonance TheoryNaveen Kumar
 
Chap 8. Optimization for training deep models
Chap 8. Optimization for training deep modelsChap 8. Optimization for training deep models
Chap 8. Optimization for training deep modelsYoung-Geun Choi
 
Perceptron & Neural Networks
Perceptron & Neural NetworksPerceptron & Neural Networks
Perceptron & Neural NetworksNAGUR SHAREEF SHAIK
 
Chapter 03 cyclic codes
Chapter 03 cyclic codesChapter 03 cyclic codes
Chapter 03 cyclic codesManoj Krishna Yadavalli
 
Mc culloch pitts neuron
Mc culloch pitts neuronMc culloch pitts neuron
Mc culloch pitts neuronSiksha 'O' Anusandhan (Deemed to be University )
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkmustafa aadel
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentMuhammad Rasel
 
Setup and hold time violation in flip-flops
Setup and hold time violation in flip-flopsSetup and hold time violation in flip-flops
Setup and hold time violation in flip-flopsJong Hwan Shin
 

Recommended

Adaptive Resonance Theory
Adaptive Resonance TheoryAdaptive Resonance Theory
Adaptive Resonance TheoryNaveen Kumar
 
Chap 8. Optimization for training deep models
Chap 8. Optimization for training deep modelsChap 8. Optimization for training deep models
Chap 8. Optimization for training deep modelsYoung-Geun Choi
 
Perceptron & Neural Networks
Perceptron & Neural NetworksPerceptron & Neural Networks
Perceptron & Neural NetworksNAGUR SHAREEF SHAIK
 
Chapter 03 cyclic codes
Chapter 03 cyclic codesChapter 03 cyclic codes
Chapter 03 cyclic codesManoj Krishna Yadavalli
 
Mc culloch pitts neuron
Mc culloch pitts neuronMc culloch pitts neuron
Mc culloch pitts neuronSiksha 'O' Anusandhan (Deemed to be University )
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkmustafa aadel
 
Feed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentFeed forward ,back propagation,gradient descent
Feed forward ,back propagation,gradient descentMuhammad Rasel
 
Setup and hold time violation in flip-flops
Setup and hold time violation in flip-flopsSetup and hold time violation in flip-flops
Setup and hold time violation in flip-flopsJong Hwan Shin
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkManasa Mona
 
Fuzzy logic
Fuzzy logicFuzzy logic
Fuzzy logicBabu Appat
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkMohd Arafat Shaikh
 
Counter propagation Network
Counter propagation NetworkCounter propagation Network
Counter propagation NetworkAkshay Dhole
 
Art network
Art networkArt network
Art networkMirzashabana
 
Neural network & its applications
Neural network & its applications Neural network & its applications
Neural network & its applications Ahmed_hashmi
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkPrakash K
 
Adaptive filters
Adaptive filtersAdaptive filters
Adaptive filtersMustafa Khaleel
 
Independent Component Analysis
Independent Component AnalysisIndependent Component Analysis
Independent Component AnalysisTatsuya Yokota
 
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...Simplilearn
 
Data link layer
Data link layer Data link layer
Data link layer Mukesh Chinta
 
Control Units : Microprogrammed and Hardwired:control unit
Control Units : Microprogrammed and Hardwired:control unitControl Units : Microprogrammed and Hardwired:control unit
Control Units : Microprogrammed and Hardwired:control unitabdosaidgkv
 
Artificial neural networks
Artificial neural networksArtificial neural networks
Artificial neural networksstellajoseph
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkKnoldus Inc.
 
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...Mohammed Bennamoun
 
L9 fuzzy implications
L9 fuzzy implicationsL9 fuzzy implications
L9 fuzzy implicationsMohammad Umar Rehman
 
Artifical Neural Network and its applications
Artifical Neural Network and its applicationsArtifical Neural Network and its applications
Artifical Neural Network and its applicationsSangeeta Tiwari
 
Artificial nueral network slideshare
Artificial nueral network slideshareArtificial nueral network slideshare
Artificial nueral network slideshareRed Innovators
 
Semiconductor memory
Semiconductor memorySemiconductor memory
Semiconductor memoryThen Murugeshwari
 
Quantum machine learning basics
Quantum machine learning basicsQuantum machine learning basics
Quantum machine learning basicsKrishna Kumar Sekar
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksAdri Jovin
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithmsaciijournal
 

More Related Content

What's hot

Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkManasa Mona
 
Counter propagation Network
Counter propagation NetworkCounter propagation Network
Counter propagation NetworkAkshay Dhole
 
Neural network & its applications
Neural network & its applications Neural network & its applications
Neural network & its applications Ahmed_hashmi
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural NetworkPrakash K
 
Independent Component Analysis
Independent Component AnalysisIndependent Component Analysis
Independent Component AnalysisTatsuya Yokota
 
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...Simplilearn
 
Control Units : Microprogrammed and Hardwired:control unit
Control Units : Microprogrammed and Hardwired:control unitControl Units : Microprogrammed and Hardwired:control unit
Control Units : Microprogrammed and Hardwired:control unitabdosaidgkv
 
Artificial neural networks
Artificial neural networksArtificial neural networks
Artificial neural networksstellajoseph
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkKnoldus Inc.
 
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...Mohammed Bennamoun
 
Artifical Neural Network and its applications
Artifical Neural Network and its applicationsArtifical Neural Network and its applications
Artifical Neural Network and its applicationsSangeeta Tiwari
 
Artificial nueral network slideshare
Artificial nueral network slideshareArtificial nueral network slideshare
Artificial nueral network slideshareRed Innovators
 

What's hot (20)

Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Fuzzy logic
Fuzzy logicFuzzy logic
Fuzzy logic
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural network
 
Counter propagation Network
Counter propagation NetworkCounter propagation Network
Counter propagation Network
 
Art network
Art networkArt network
Art network
 
Neural network & its applications
Neural network & its applications Neural network & its applications
Neural network & its applications
 
Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Adaptive filters
Adaptive filtersAdaptive filters
Adaptive filters
 
Independent Component Analysis
Independent Component AnalysisIndependent Component Analysis
Independent Component Analysis
 
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
Recurrent Neural Network (RNN) | RNN LSTM Tutorial | Deep Learning Course | S...
 
Data link layer
Data link layer Data link layer
Data link layer
 
Control Units : Microprogrammed and Hardwired:control unit
Control Units : Microprogrammed and Hardwired:control unitControl Units : Microprogrammed and Hardwired:control unit
Control Units : Microprogrammed and Hardwired:control unit
 
Artificial neural networks
Artificial neural networksArtificial neural networks
Artificial neural networks
 
Introduction to Recurrent Neural Network
Introduction to Recurrent Neural NetworkIntroduction to Recurrent Neural Network
Introduction to Recurrent Neural Network
 
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
Artificial Neural Network Lecture 6- Associative Memories & Discrete Hopfield...
 
L9 fuzzy implications
L9 fuzzy implicationsL9 fuzzy implications
L9 fuzzy implications
 
Artifical Neural Network and its applications
Artifical Neural Network and its applicationsArtifical Neural Network and its applications
Artifical Neural Network and its applications
 
Artificial nueral network slideshare
Artificial nueral network slideshareArtificial nueral network slideshare
Artificial nueral network slideshare
 
Semiconductor memory
Semiconductor memorySemiconductor memory
Semiconductor memory
 
Quantum machine learning basics
Quantum machine learning basicsQuantum machine learning basics
Quantum machine learning basics
 

Similar to Neural Network Design: Chapter 18 Grossberg Network

Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksAdri Jovin
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithmsaciijournal
 
Web spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsWeb spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsaciijournal
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classificationCenk Bircanoğlu
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Gaurav Mittal
 
Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Dr.(Mrs).Gethsiyal Augasta
 
Topology ppt
Topology pptTopology ppt
Topology pptboocse11
 
Handwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksHandwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksIRJET Journal
 
DEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONDEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONIRJET Journal
 
Basic Learning Algorithms of ANN
Basic Learning Algorithms of ANNBasic Learning Algorithms of ANN
Basic Learning Algorithms of ANNwaseem khan
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORKESCOM
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural networkDEEPASHRI HK
 
M7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfM7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfArushiKansal3
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningSujit Pal
 

Similar to Neural Network Design: Chapter 18 Grossberg Network (20)

Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural Networks
 
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network AlgorithmsWeb Spam Classification Using Supervised Artificial Neural Network Algorithms
Web Spam Classification Using Supervised Artificial Neural Network Algorithms
 
D028018022
D028018022D028018022
D028018022
 
Web spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithmsWeb spam classification using supervised artificial neural network algorithms
Web spam classification using supervised artificial neural network algorithms
 
Autoencoders for image_classification
Autoencoders for image_classificationAutoencoders for image_classification
Autoencoders for image_classification
 
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
Deep Neural Networks (D1L2 Insight@DCU Machine Learning Workshop 2017)
 
Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)Convolutional Neural Networks (CNN)
Convolutional Neural Networks (CNN)
 
Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”Neural Networks in Data Mining - “An Overview”
Neural Networks in Data Mining - “An Overview”
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
Topology ppt
Topology pptTopology ppt
Topology ppt
 
ai7.ppt
ai7.pptai7.ppt
ai7.ppt
 
Handwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural NetworksHandwritten Digit Recognition using Convolutional Neural Networks
Handwritten Digit Recognition using Convolutional Neural Networks
 
DEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTIONDEEP LEARNING BASED BRAIN STROKE DETECTION
DEEP LEARNING BASED BRAIN STROKE DETECTION
 
Basic Learning Algorithms of ANN
Basic Learning Algorithms of ANNBasic Learning Algorithms of ANN
Basic Learning Algorithms of ANN
 
Counterpropagation NETWORK
Counterpropagation NETWORKCounterpropagation NETWORK
Counterpropagation NETWORK
 
Artificial neural network
Artificial neural networkArtificial neural network
Artificial neural network
 
M7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdfM7 - Neural Networks in machine learning.pdf
M7 - Neural Networks in machine learning.pdf
 
Artificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep LearningArtificial Intelligence, Machine Learning and Deep Learning
Artificial Intelligence, Machine Learning and Deep Learning
 

More from Jason Tsai

基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介Jason Tsai
 
Neural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksNeural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksJason Tsai
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Jason Tsai
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Reinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceReinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceJason Tsai
 
Deep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyDeep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyJason Tsai
 
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsDeep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsJason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路Jason Tsai
 

More from Jason Tsai (13)

基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介基於深度學習的人臉辨識技術簡介
基於深度學習的人臉辨識技術簡介
 
Neural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis NetworksNeural Network Design: Chapter 17 Radial Basis Networks
Neural Network Design: Chapter 17 Radial Basis Networks
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
 
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
Convolutional Neural Networks (CNN) — 卷積神經網路的前世今生
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Reinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 NeuroscienceReinforcement Learning: Chapter 15 Neuroscience
Reinforcement Learning: Chapter 15 Neuroscience
 
Deep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical MethodologyDeep Learning: Chapter 11 Practical Methodology
Deep Learning: Chapter 11 Practical Methodology
 
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning BasicsDeep Learning: Introduction & Chapter 5 Machine Learning Basics
Deep Learning: Introduction & Chapter 5 Machine Learning Basics
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 
漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路漫談人工智慧：啟發自大腦科學的深度學習網路
漫談人工智慧：啟發自大腦科學的深度學習網路
 

Recently uploaded

Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfjimielynbastida
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024BookNet Canada
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Neo4j
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfngoud9212
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Artificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning eraArtificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning eraDeakin University
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 

Recently uploaded (20)

Science&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdfScience&tech:THE INFORMATION AGE STS.pdf
Science&tech:THE INFORMATION AGE STS.pdf
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
 
Hot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort Service
Hot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort ServiceHot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort Service
Hot Sexy call girls in Panjabi Bagh 🔝 9953056974 🔝 Delhi escort Service
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024
 
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdf
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
Artificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning eraArtificial intelligence in the post-deep learning era
Artificial intelligence in the post-deep learning era
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 

Neural Network Design: Chapter 18 Grossberg Network

  • 1. Chapter 18 Grossberg Network Jason Tsai (蔡志順) January 11, 2020 Mozilla Community Space Taipei
  • 2. *Copyright Notice: Most materials from this presentation are taken from the book “Neural Network Design 2nd edition” authored by Martin T. Hagan, Howard B. Demuth, Mark Hudson Beale and Orlando De Jesús. The other quoted sources are mentioned in the respective slides. This presentation itself adopts Creative Commons license.
  • 3. A Quote from this Chapter “Although the original inspiration for the field of artificial neural networks came from biology, at times we forget to look back to biology for new ideas. It will be the blending of biology, mathematics, psychology and other disciplines that will provide the maximum growth in our understanding of neural networks.” 「雖然類神經網路這一領域的最初靈感來自生物學， 但有時我們會忘了回顧生物學以啟迪新的想法。融合 生物學、數學、心理學和其他學科將使我們對神經網 路的理解得以最大的發展。」
  • 7. Blind Spot *Figure adopted from https://bit.ly/39iUtNQ
  • 8. Test for the Blind Spot Look at the blue circle on the left side of above figure with your right eye while covering your left eye and move your head back and forth perpendicularly to the circle to find the point at which the circle on the right will disappear from your field of vision.
  • 12. Neon Color Spreading *Figure adopted from https://bit.ly/37xDyWg
  • 13. Oriented Receptive Field *Figure adopted from https://bit.ly/2rPXGDF & https://bit.ly/2tmGg1X
  • 17. Lateral Inhibition *Figure adopted from https://bit.ly/2yaat37
  • 18. Leaky Integrate-and-Fire *Figure adopted from Wulfram Gerstner, Werner M. Kistler, Richard Naud and Liam Paninski “Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition” Cambridge University Press. 2014. Page 11.
  • 20. *Solution to 1st order ODE *Credit to Schwartz Lyu
  • 22. Shunting Model Here p+, p-, b+, are b- are nonnegative values Refer to slide 28: Analysis of Normalization
  • 26. Characteristics of Layer 1 • The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. • The output of Layer 1 is a normalized version of the input pattern. • The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. • The operation of Layer 1 explains the brightness constancy and brightness contrast characteristics of the human visual system.
  • 29. Example of Layer 1 Response The response of the network maintains the relative intensities of the inputs, while limiting the total response.
  • 31. Characteristics of Layer 2 • As in the Hamming and Kohonen networks, the inputs to Layer 2 are the inner products between the prototype patterns (rows of the weight matrix W2) and the output of Layer 1 (normalized input pattern). • The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). • The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated).
  • 33. Example of Layer 2 Response
  • 34. Choice of Transfer Function
  • 35. Sigmoid Transfer Function • A sigmoid function is faster-than-linear for small signals, approximately linear for intermediate signals and slower-than-linear for large signals. • When a sigmoid transfer function is used in Layer 2, the pattern is contrast enhanced; larger values are amplified, and smaller values are attenuated. • All initial neuron outputs that are less than a certain level decay to 0. • This merges the noise suppression of the faster- than-linear transfer functions with the perfect storage produced by linear transfer functions.
  • 37. Example of W2 Response
  • 40. Relation to Kohonen Rule (Cont.)
  • 41. Three major differences: • The Grossberg network is a continuous-time network. • Layer 1 of the Grossberg network automatically normalizes the input vectors. • Layer 2 of the Grossberg can perform a “soft” competition, rather than the winner-take-all competition of the Kohonen network. This soft competition allows more than one neuron in Layer 2 to learn. This causes the Grossberg network to operate as a feature map. Grossberg vs. Kohonen Networks
  • 42. Potential Issues • One key problem of the Grossberg network is the stability of learning. As more inputs are applied to the network, there is no guarantee that the weight matrix will eventually converge (form stable clusters / categories). [Refer to Chapter 19: Adaptive Resonance Theory, ART] • Another problem is the stability of the differential equations that implement Grossberg’s continuous- time competitive recurrent network. The output of a recurrent network could converge, oscillate, or even diverge. [Refer to Chapter 20: Stability]
  • 43. Basic ART Architecture (Ch.19) Add-ons: Layer 2 to Layer 1 expectations The orienting subsystem (reset) Modified gain control Let’s move on…