This document discusses deep reservoir computing for structured data like time-series and graphs. Recurrent neural networks are naturally suited for sequential data but are difficult to train. Reservoir computing addresses this by only training the output layer, leaving the recurrent hidden layer untrained. Stacking multiple reservoir layers leads to deep reservoir computing, which develops richer dynamics. For graphs, each node is encoded by the fixed point of a dynamical system implemented as a reservoir network. Deep reservoirs can inherently construct rich embeddings for graphs without training recurrent connections. This makes graph neural networks accurate yet fast compared to state-of-the-art methods that require training deep architectures.
https://github.com/telecombcn-dl/lectures-all/
These slides review techniques for interpreting the behavior of deep neural networks. The talk reviews basic techniques such as the display of filters and tensors, as well as more advanced ones that try to interpret which part of the input data is responsible for the predictions, or generate data that maximizes the activation of certain neurons.
https://github.com/telecombcn-dl/dlmm-2017-dcu
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
Invited talk at workshop "Exascale Computing in Astrophysics" held in Ascona, Switzerland, 8-13 September 2013.
http://www.itp.uzh.ch/exastro2013/Home.html
https://telecombcn-dl.github.io/2017-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
https://github.com/telecombcn-dl/lectures-all/
These slides review techniques for interpreting the behavior of deep neural networks. The talk reviews basic techniques such as the display of filters and tensors, as well as more advanced ones that try to interpret which part of the input data is responsible for the predictions, or generate data that maximizes the activation of certain neurons.
https://github.com/telecombcn-dl/dlmm-2017-dcu
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
Invited talk at workshop "Exascale Computing in Astrophysics" held in Ascona, Switzerland, 8-13 September 2013.
http://www.itp.uzh.ch/exastro2013/Home.html
https://telecombcn-dl.github.io/2017-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
Recurrent Neural Networks (RNNs) represent the reference class of Deep Learning models for learning from sequential data. Despite the widespread success, a major downside of RNNs and commonly derived ‘gating’ variants (LSTM, GRU) is given by the high cost of the involved training algorithms. In this context, an increasingly popular alternative is the Reservoir Computing (RC) approach, which enables limiting the training algorithm to operate only on a restricted set of (output) parameters. RC is appealing for several reasons, including the amenability of being implemented in low-powerful edge devices, enabling adaptation and personalization in IoT and cyber-physical systems applications.
This webinar will introduce Reservoir Computing from scratch, covering all the fundamental design topics as well as good practices. It is targeted to both researchers and practitioners that are interested in setting up fastly-trained Deep Learning models for sequential data.
One page summary of master thesis "Mathematical Analysis of Neural Networks"Alina Leidinger
This is a one page summary of my master thesis which I handed in on June 15, 2019 at TUM. The thesis takes the form of a literature review on the existing rigorous analysis on neural networks. It focuses on 3 key aspects: modern and classical results in approximation theory, robustness of neural networks and unique identification of neural network weights. The thesis was supervised by Prof. Dr. Massimo Fornasier at the Chair of Applied Numerical Analysis of the Mathematics Department at TUM.
This lecture reviews methods that allow interpreting the outcomes of a deep convolutional neural network. It presents some of the techniques proposed in the literature.
IJERA (International journal of Engineering Research and Applications) is International online, ... peer reviewed journal. For more detail or submit your article, please visit www.ijera.com
PowerPoint slides from a 2015 Guest Lecture in PSYCH-268A: Computational Neuroscience, Prof. Jeff Krichmar, University of California, Irvine (UCI).
Corresponding publication:
Beyeler*, M., Carlson*, K. D. , Chou*, T-S., Dutt, N., Krichmar, J. L. (2015). CARLsim 3: A user-friendly and highly optimized library for the creation of neurobiologically detailed spiking neural networks. Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland. (*equal contribution)
Presentation of few recent papers on Deep Learning ... in particular Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, Song Han, Huizi Mao, William J. Dally International Conference on Learning Representations ICLR2016
New artificial neural network design for Chua chaotic system prediction usin...IJECEIAES
This study aims to design a new architecture of the artificial neural networks (ANNs) using the Xilinx system generator (XSG) and its hardware co-simulation equivalent model using field programmable gate array (FPGA) to predict the behavior of Chua’s chaotic system and use it in hiding information. The work proposed consists of two main sections. In the first section, MATLAB R2016a was used to build a 3×4×3 feed forward neural network (FFNN). The training results demonstrate that FFNN training in the Bayesian regulation algorithm is sufficiently accurate to directly implement. The second section demonstrates the hardware implementation of the network with the XSG on the Xilinx artix7 xc7a100t-1csg324 chip. Finally, the message was first encrypted using a dynamic Chua system and then decrypted using ANN’s chaotic dynamics. ANN models were developed to implement hardware in the FPGA system using the IEEE 754 Single precision floating-point format. The ANN design method illustrated can be extended to other chaotic systems in general.
Slides for a talk about Graph Neural Networks architectures, overview taken from very good paper by Zonghan Wu et al. (https://arxiv.org/pdf/1901.00596.pdf)
Neuromorphic Computing indicates a broad area of research that aims at achieving means of physical information processing that are inspired by biological brains. As such, this kind of systems is envisaged as being the ideal approach for implementing artificial neural networks concepts. With the rapid pace of development in Deep Learning, the synergy between the development of neuromorphic hardware and neural network concepts is fundamental to obtain intelligent systems that can exploit the full potential of learning efficiently.
This talk aims at giving a broad overview of the possibilities of such synergy. First, we will quickly explore the fundamental differences between neuromorphic and traditional computing, and then we will focus on concepts, algorithms, and neural architectures that are prone to neuromorphic implementation.
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...Numenta
These are slides on a workshop Subutai Ahmad hosted on March 5, 2018 at the Computational and Systems Neuroscience Meeting (Cosyne) 2018.
About:
This workshop on long-range cortical circuits is focused on our peer-reviewed paper, “A Theory of How Columns in the Neocortex Enable Learning the Structure of the World.” Subutai discussed the inference mechanism introduced in the paper, our theory of location information, and how long-range connections allow columns to integrate inputs over space to perform object recognition.
Accelerating Science with Generative Adversarial NetworksMichela Paganini
Presentation at NERSC Data Day 2017 at Lawrence Berkeley National Laboratory on the potential of Generative Adversarial Networks to speed up scientific simulation and empower scientists and researchers.
6th eCAS workshop on Engineering Collective Adaptive SystemsRoberto Casadei
This is the presentation introducing the 6th eCAS workshop on Engineering Collective Adaptive Systems. It recaps its scope, provides data regarding this edition, provides an overview of the program and related initiatives.
International Journal of Computational Engineering Research(IJCER) is an intentional online Journal in English monthly publishing journal. This Journal publish original research work that contributes significantly to further the scientific knowledge in engineering and Technology.
Deep Learning for Hidden Signals - Enabling Real-time Multimessenger Astrophy...Daniel George
Presented at the GPU Technology Conference (GTC17) in San Jose, California on May 10, 2017
-------------------------
We introduce Deep Filtering, a new method for end-to-end time-series signal processing, which combines two deep convolutional neural networks for classification and regression to detect and characterize signals much weaker than the background noise. We applied this method for gravitational wave analysis specifically for mergers of black holes and demonstrated that it significantly outperforms conventional machine learning techniques, is far more efficient than matched-filtering allowing real-time processing of raw big data with minimal resources, and extends the range of gravitational waves that can be detected by advanced LIGO. This initiates a new paradigm for scientific research which uses massively-parallel numerical simulations to train artificial intelligence algorithms that exploit emerging hardware architectures. Our approach offers a unique framework to enable coincident detection campaigns of gravitational wave sources and their multimessenger counterparts.
Recurrent Neural Networks (RNNs) represent the reference class of Deep Learning models for learning from sequential data. Despite the widespread success, a major downside of RNNs and commonly derived ‘gating’ variants (LSTM, GRU) is given by the high cost of the involved training algorithms. In this context, an increasingly popular alternative is the Reservoir Computing (RC) approach, which enables limiting the training algorithm to operate only on a restricted set of (output) parameters. RC is appealing for several reasons, including the amenability of being implemented in low-powerful edge devices, enabling adaptation and personalization in IoT and cyber-physical systems applications.
This webinar will introduce Reservoir Computing from scratch, covering all the fundamental design topics as well as good practices. It is targeted to both researchers and practitioners that are interested in setting up fastly-trained Deep Learning models for sequential data.
One page summary of master thesis "Mathematical Analysis of Neural Networks"Alina Leidinger
This is a one page summary of my master thesis which I handed in on June 15, 2019 at TUM. The thesis takes the form of a literature review on the existing rigorous analysis on neural networks. It focuses on 3 key aspects: modern and classical results in approximation theory, robustness of neural networks and unique identification of neural network weights. The thesis was supervised by Prof. Dr. Massimo Fornasier at the Chair of Applied Numerical Analysis of the Mathematics Department at TUM.
This lecture reviews methods that allow interpreting the outcomes of a deep convolutional neural network. It presents some of the techniques proposed in the literature.
IJERA (International journal of Engineering Research and Applications) is International online, ... peer reviewed journal. For more detail or submit your article, please visit www.ijera.com
PowerPoint slides from a 2015 Guest Lecture in PSYCH-268A: Computational Neuroscience, Prof. Jeff Krichmar, University of California, Irvine (UCI).
Corresponding publication:
Beyeler*, M., Carlson*, K. D. , Chou*, T-S., Dutt, N., Krichmar, J. L. (2015). CARLsim 3: A user-friendly and highly optimized library for the creation of neurobiologically detailed spiking neural networks. Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland. (*equal contribution)
Presentation of few recent papers on Deep Learning ... in particular Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, Song Han, Huizi Mao, William J. Dally International Conference on Learning Representations ICLR2016
New artificial neural network design for Chua chaotic system prediction usin...IJECEIAES
This study aims to design a new architecture of the artificial neural networks (ANNs) using the Xilinx system generator (XSG) and its hardware co-simulation equivalent model using field programmable gate array (FPGA) to predict the behavior of Chua’s chaotic system and use it in hiding information. The work proposed consists of two main sections. In the first section, MATLAB R2016a was used to build a 3×4×3 feed forward neural network (FFNN). The training results demonstrate that FFNN training in the Bayesian regulation algorithm is sufficiently accurate to directly implement. The second section demonstrates the hardware implementation of the network with the XSG on the Xilinx artix7 xc7a100t-1csg324 chip. Finally, the message was first encrypted using a dynamic Chua system and then decrypted using ANN’s chaotic dynamics. ANN models were developed to implement hardware in the FPGA system using the IEEE 754 Single precision floating-point format. The ANN design method illustrated can be extended to other chaotic systems in general.
Slides for a talk about Graph Neural Networks architectures, overview taken from very good paper by Zonghan Wu et al. (https://arxiv.org/pdf/1901.00596.pdf)
Neuromorphic Computing indicates a broad area of research that aims at achieving means of physical information processing that are inspired by biological brains. As such, this kind of systems is envisaged as being the ideal approach for implementing artificial neural networks concepts. With the rapid pace of development in Deep Learning, the synergy between the development of neuromorphic hardware and neural network concepts is fundamental to obtain intelligent systems that can exploit the full potential of learning efficiently.
This talk aims at giving a broad overview of the possibilities of such synergy. First, we will quickly explore the fundamental differences between neuromorphic and traditional computing, and then we will focus on concepts, algorithms, and neural architectures that are prone to neuromorphic implementation.
Could A Model Of Predictive Voting Explain Many Long-Range Connections? by Su...Numenta
These are slides on a workshop Subutai Ahmad hosted on March 5, 2018 at the Computational and Systems Neuroscience Meeting (Cosyne) 2018.
About:
This workshop on long-range cortical circuits is focused on our peer-reviewed paper, “A Theory of How Columns in the Neocortex Enable Learning the Structure of the World.” Subutai discussed the inference mechanism introduced in the paper, our theory of location information, and how long-range connections allow columns to integrate inputs over space to perform object recognition.
Accelerating Science with Generative Adversarial NetworksMichela Paganini
Presentation at NERSC Data Day 2017 at Lawrence Berkeley National Laboratory on the potential of Generative Adversarial Networks to speed up scientific simulation and empower scientists and researchers.
6th eCAS workshop on Engineering Collective Adaptive SystemsRoberto Casadei
This is the presentation introducing the 6th eCAS workshop on Engineering Collective Adaptive Systems. It recaps its scope, provides data regarding this edition, provides an overview of the program and related initiatives.
International Journal of Computational Engineering Research(IJCER) is an intentional online Journal in English monthly publishing journal. This Journal publish original research work that contributes significantly to further the scientific knowledge in engineering and Technology.
Deep Learning for Hidden Signals - Enabling Real-time Multimessenger Astrophy...Daniel George
Presented at the GPU Technology Conference (GTC17) in San Jose, California on May 10, 2017
-------------------------
We introduce Deep Filtering, a new method for end-to-end time-series signal processing, which combines two deep convolutional neural networks for classification and regression to detect and characterize signals much weaker than the background noise. We applied this method for gravitational wave analysis specifically for mergers of black holes and demonstrated that it significantly outperforms conventional machine learning techniques, is far more efficient than matched-filtering allowing real-time processing of raw big data with minimal resources, and extends the range of gravitational waves that can be detected by advanced LIGO. This initiates a new paradigm for scientific research which uses massively-parallel numerical simulations to train artificial intelligence algorithms that exploit emerging hardware architectures. Our approach offers a unique framework to enable coincident detection campaigns of gravitational wave sources and their multimessenger counterparts.
Similar to Claudio Gallicchio - Deep Reservoir Computing for Structured Data (20)
Presentato al sesto WebMeetup del Machine Learning / Data Science Meetup Roma: https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/273089965/
Presentazione per il sesto WebMeetup del Machine Learning / Data Science Meetup Roma: https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/273089965/
Paolo Galeone - Dissecting tf.function to discover auto graph strengths and s...MeetupDataScienceRoma
Original presentation available on GitHub: https://pgaleone.eu/tf-function-talk/
Meetup: https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/264338606/
Multimodal AI Approach to Provide Assistive Services (Francesco Puja)MeetupDataScienceRoma
Presentazione dal Meetup del Machine Learning / Data Science Meetup di Roma - Giugno 2019:
https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/262120815/
Presentazione dal Meetup del Machine Learning / Data Science Meetup di Roma - Giugno 2019:
https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/262120815/
Zero, One, Many - Machine Learning in Produzione (Luca Palmieri)MeetupDataScienceRoma
Talk dal Meetup del Machine Learning / Data Science Meetup di Roma - Giugno 2019:
https://www.meetup.com/it-IT/Machine-Learning-Data-Science-Meetup/events/262120815/
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
4. RECURRENT NEURAL NETWORKS
• DYNAMICAL NEURAL NETWORK MODELS NATURALLY
SUITABLE FOR PROCESSING SEQUENTIAL FORMS OF DATA
(TIME-SERIES)
• INTERNAL DYNAMICS ENABLE TREATING ARBITRARILY LONG
SEQUENCES
input
hidden
readout
𝑥(𝑡)
ℎ(𝑡)
𝑦(𝑡)
Dynamical
Recurrent
Representation
Layer
𝐡 𝑡 = tanh(𝐔 𝐱 𝑡 + 𝐖 𝐡 𝑡 − 1 )
𝐲 𝑡 = fY(𝐕 𝐡 𝑡 )
state input
previous
state
output
tuned parameters
5. TRAINING RECURRENT NEURAL NETS
• GRADIENT MIGHT VANISH OR EXPLODE
THROUGH MANY TRANSFORMATIONS
• DIFFICULT TO TRAIN ON LONG-TERM
DEPENDENCIES
• TRAINING RNN S IS SLOW
Bengio et al, “Learning long-term dependencies with
gradient descent is difficult”, IEEE Transactions on
Neural Networks, 1994
Pascanu et al, “On the difficulty of training recurrent
neural networks”, ICML 2013
6. RESERVOIR COMPUTING
FOCUS ON THE DYNAMICAL SYSTEM:
• THE RECURRENT HIDDEN LAYER IS A (DISCRETE-TIME) NON-
LINEAR & NON-AUTONOMOUS DYNAMICAL SYSTEM
• TRAIN ONLY THE OUTPUT FUNCTION
• MUCH FASTER & LIGHTWEIGHT TO TRAIN
• SPEED-UP ≈ 𝑥100
• SCALABLE FOR EDGE DISTRIBUTED LEARNING
readout
𝑥(𝑡)
ℎ(𝑡)
𝑦(𝑡)
Untrained
Dynamical
System
Trained Output
𝐡 𝑡 = tanh(𝐔 𝐱 𝑡 + 𝐖 𝐡 𝑡 − 1 )
randomized untrained parameters
Reservoir
8. RESERVOIR COMPUTING – INITIALIZATION
𝐡 𝑡 = tanh(𝜔𝐔 𝐱 𝑡 + 𝝆𝐖 𝐡 𝑡 − 1 )
• HOW TO SCALE THE WEIGHT MATRICES?
• FULFILL THE “ECHO STATE PROPERTY”
• GLOBAL ASYMPTOTIC LYAPUNOV STABILITY CONDITION
• SPECTRAL RADIUS < 1
RANDOMLY INITIALIZED + SPARSELY CONNECTED
Yildiz, Izzet B., Herbert Jaeger, and Stefan J. Kiebel. "Re-visiting
the echo state property." Neural networks 35 (2012): 1-9.
9. WHY DOES IT WORK?
Gallicchio, Claudio, and Alessio Micheli. "Architectural
and markovian factors of echo state networks." Neural
Networks 24.5 (2011): 440-456.
Exploit the architectural bias
- Contractive dynamical systems
separate input histories based on
the suffix even without training
- Markovian factor in RNN design
- The separation ability peaks near
the boundary of stability (edge of
chaos)
10. ADVANTAGES
1. FASTER LEARNING
2. CLEAN MATHEMATICAL ANALYSIS
• ARCHITECTURAL BIAS OF RECURRENT NEURAL NETWORKS
3. UNCONVENTIONAL HARDWARE IMPLEMENTATIONS
• E.G., IN PHOTONICS (MORE EFFICIENT, FASTER)
Brunner, Daniel, Miguel C. Soriano, and Guy Van der Sande,
eds. Photonic Reservoir Computing: Optical Recurrent Neural
Networks. Walter de Gruyter GmbH & Co KG, 2019.
Tino, Peter, Michal Cernansky, and Lubica Benuskova.
"Markovian architectural bias of recurrent neural networks." IEEE
Transactions on Neural Networks 15.1 (2004): 6-15.
11. APPLICATIONS
• AMBIENT INTELLIGENCE: DEPLOY EFFICIENTLY TRAINABLE RNNS IN RESOURCE-CONSTRAINED DEVICES
• HUMAN ACTIVITY RECOGNITION
• ROBOT LOCALIZATION (E.G., IN HOSPITAL ENVIRONMENTS)
• EARLY IDENTIFICATION OF EARTHQUAKES
• MEDICAL APPLICATIONS
• ESTIMATION OF CLINICAL EXAMS OUTCOMES (E.G., POSTURE AND BALANCE SKILLS)
• EARLY IDENTIFICATION OF (RARE) HEART DISEASES
• HUMAN-CENTRIC INTERACTIONS IN CYBER-PHYSICAL SYSTEMS OF SYSTEMS
https://www.teaching-h2020.eu
http://fp7rubicon.eu/
14. DEPTH IN RECURRENT NEURAL SYSTEMS
• DEVELOP RICHER DYNAMICS EVEN WITHOUT TRAINING OF THE RECURRENT CONNECTIONS
• MULTIPLE TIME-SCALES
• MULTIPLE FREQUENCIES
• NATURALLY BOOST THE PERFORMANCE OF DYNAMICAL NEURAL SYSTEMS EFFICIENTLY
Gallicchio, Claudio and Alessio Micheli. “Deep
Reservoir Computing” (2020). To appear in
"Reservoir Computing: Theory and Physical
Implementations", K. Nakajima and I. Fischer,
eds., Springer.
15. DESIGN OF DEEP ESNS
- Each reservoir layer cuts part of the
frequency content;
- Idea: stop adding new layers
whenever the filtering effect
(centroid shift) becomes negligible
(independently from the readout
part)
Gallicchio, Claudio, Alessio Micheli, and Luca Pedrelli. "Design of
deep echo state networks." Neural Networks 108 (2018): 33-47.
17. RESERVOIR COMPUTING FOR GRAPHS
• BASIC IDEA: EACH INPUT GRAPH IS ENCODED BY THE FIXED POINT OF A DYNAMICAL SYSTEM
• THE DYNAMICAL SYSTEM IS IMPLEMENTED BY A HIDDEN LAYER OF RECURRENT RESERVOIR
NEURONS
• RESERVOIR COMPUTING (RC):
• THE RESERVOIR NEURONS DO NOT REQUIRE LEARNING
• FAST DEEP NEURAL NETWORKS FOR GRAPHS
Deep Neural
Network ?
18. GRAPH REPRESENTATIONS WITHOUT LEARNING
• EACH VERTEX IN AN INPUT GRAPH IS ENCODED BY THE HIDDEN LAYER
𝑣
𝑣1
𝑣2
𝑣 𝑘
𝑥(𝑣)
ℎ(𝑣)
ℎ 𝑣1 ℎ(𝑣 𝑘)
⋮
⋮
embedding (state)
of vertex 𝑣 input feature
of vertex 𝑣
embedding (state)
of neighbors of vertex
input weight matrix hidden weight matrix
𝐡(𝑣) = tanh(𝐔 𝐱 𝑣 +
𝑣′∈𝑁(𝑣)
𝐖 𝐡(𝑣′))
19. GRAPH REPRESENTATIONS WITHOUT LEARNING
• EQUATIONS CAN BE COLLECTIVELY GROUPED
𝑣
𝑣1
𝑣2
𝑣 𝑘
𝐇 = F X, H = tanh(𝐔 𝐗 + 𝐖 𝐇 𝐀)
state
input feature matrixadjacency matrix
Existence (and uniqueness) of solutions is not guaranteed in case of
mutual dependencies (e.g., cycles, undirected edges)
20. GRAPH EMBEDDING BY LEARNING-FREE NEURONS
• THE ENCODING EQUATION CAN BE SEEN AS A DISCRETE TIME DYNAMICAL SYSTEM
• EXISTENCE UNIQUENESS OF THE SOLUTION IS GUARANTEED BY STUDYING LOCAL ASYMPTOTIC
STABILITY OF THE ABOVE EQUATION
• GRAPH EMBEDDING STABILITY (GES): GLOBAL (LYAPUNOV) ASYMPTOTIC STABILITY OF THE
ENCODING PROCESS
INITIALIZE THE DYNAMICAL LAYER UNDER THE GES CONDITION AND THEN LEAVE IT UNTRAINED
RESERVOIR COMPUTING FOR GRAPHS
𝐇 = F X, H = tanh(𝐔 𝐗 + 𝐖 𝐇 𝐀)
𝑣
𝑣1
𝑣2
𝑣 𝑘
21. DEEP RESERVOIRS FOR GRAPHS
• INITIALIZE EACH LAYER TO CONTROL ITS
EFFECTIVE SPECTRAL RADIUS
𝜌(𝑖)
= 𝜌 𝐖(𝑖)
𝑘
• DRIVE (ITERATE) THE NESTED SET OF
DYNAMICAL RESERVOIR SYSTEMS TOWARDS
THE FIXED POINT FOR EACH INPUT GRAPH𝒉 1 (𝑣)
𝒙(𝑣) 𝒉 𝟏
(𝑣1) 𝒉 𝟏
(𝑣 𝑘)…
…
𝒉 𝒊
(𝑣)
𝒉 𝒊−𝟏
(𝑣) 𝒉 𝒊
(𝑣1) 𝒉 𝒊
(𝑣 𝑘)…
…
vertex
feature
embeddings of neighbors
embeddings of neighbors
embedding in the
previous layer
1-st hidden layer
i-th hidden layer
�
�
�
�
Gallicchio, Claudio, and Alessio Micheli. "Fast
and Deep Graph Neural Networks." AAAI. 2020.
23. IT’S ACCURATE
• HIGHLY COMPETITIVE WITH STATE-OF-
THE-ART
• DEEP GNN ARCHITECTURES WITH
STABLE DYNAMICS CAN INHERENTLY
CONSTRUCT RICH NEURAL
EMBEDDINGS FOR GRAPHS EVEN
WITHOUT TRAINING OF
RECURRENT CONNECTIONS
• TRAINING DEEPER NETWORKS COMES
AT THE SAME COST
Gallicchio, Claudio, and Alessio Micheli. "Fast
and Deep Graph Neural Networks." AAAI. 2020.
24. IT’S FAST
• UNTRAINED EMBEDDINGS, LINEAR COMPLEXITY
IN THE # OF VERTICES
• SPARSE AND DEEP ARCHITECTURE
• A VERY SMALL NUMBER OF TRAINABLE WEIGHTS
(MAX. 1001 IN OUR EXPERIMENTS)
Gallicchio, Claudio, and Alessio Micheli. "Fast
and Deep Graph Neural Networks." AAAI. 2020.
25. CONCLUSIONS
• DEEP RESERVOIR COMPUTING ENABLES FAST YET EFFECTIVE LEARNING IN
STRUCTURED DOMAINS
• SEQUENCES, GRAPH DOMAINS
• THE APPROACH HIGHLIGHTS THE INHERENT POSITIVE ARCHITECTURAL BIAS OF
RECURSIVE NEURAL NETWORKS ON GRAPHS
• STABLE AND DEEP ARCHITECTURE ENABLE RICH UNTRAINED EMBEDDINGS
• IT’S ACCURATE AND FAST