Presentation HC-4016, Heterogeneous Implementation of Neural Network Algorithms, by Dmitri Yudanov and Leon Reznik at the AMD Developer Summit (APU13) November 11-13, 2013.
Neuromorphic Chipsets - Industry Adoption AnalysisNetscribes
The concept of emulating neurons on a chip could enhance complex operations to make business decisions secure and cost-effective. Parallel connected neurons can boost AI verticals compared with the conventional processing systems. Non-stop learning and pattern recognition using this human brain architecture can help compute signals and data in the form of visual, speech, olfactory, etc., to perform real-time operations as well as predict outcomes based on detected patterns. Neuromorphic chipsets can also enhance performance owing to their low-power consumption to process AI algorithms.
Based on patent data, this report analyzes the ongoing R&D and investments in neuromorphic chipsets by major institutions across the globe to reveal the top innovators and technology leaders in this space.
For the full report, contact info@netscribes.com
Visit www.netscribes.com
A Brain computer interface is the direct communication pathway between wired brain and external device.
Brainware University
https://www.brainwareuniversity.ac.in/
Neuromorphic circuits are typically used to emulate cortical structures and to explore principles of computation of the brain. But they can also be used to implement convolutional and deep networks. Here we demonstrate a proof of concept, using our latest multi-core and on-line learning reconfigurable spiking neural network chips.
Neuromorphic Chipsets - Industry Adoption AnalysisNetscribes
The concept of emulating neurons on a chip could enhance complex operations to make business decisions secure and cost-effective. Parallel connected neurons can boost AI verticals compared with the conventional processing systems. Non-stop learning and pattern recognition using this human brain architecture can help compute signals and data in the form of visual, speech, olfactory, etc., to perform real-time operations as well as predict outcomes based on detected patterns. Neuromorphic chipsets can also enhance performance owing to their low-power consumption to process AI algorithms.
Based on patent data, this report analyzes the ongoing R&D and investments in neuromorphic chipsets by major institutions across the globe to reveal the top innovators and technology leaders in this space.
For the full report, contact info@netscribes.com
Visit www.netscribes.com
A Brain computer interface is the direct communication pathway between wired brain and external device.
Brainware University
https://www.brainwareuniversity.ac.in/
Neuromorphic circuits are typically used to emulate cortical structures and to explore principles of computation of the brain. But they can also be used to implement convolutional and deep networks. Here we demonstrate a proof of concept, using our latest multi-core and on-line learning reconfigurable spiking neural network chips.
Foundations of ANNs: Tolstoy’s Genius Explored Using Transformer Architecturegerogepatton
Artificial Narrow Intelligence is in the phase of moving towards the AGN, which will attempt
to decide as a human being. We are getting closer to it by each day, but AI actually is indefinite to
many, although it is no different than any other set of mathematically defined computer operations in its
core. Generating new data from a pre-trained model introduces new challenges to science & technology. In
this work, the design of such an architecture from scratch, solving problems, and introducing alternative
approaches are what has been conducted. Using a deep thinker, Tolstoy, as an object of study is a source
of motivation for the entire research.
Foundations of ANNs: Tolstoy’s Genius Explored using Transformer Architecturegerogepatton
Artificial Narrow Intelligence is in the phase of moving towards the AGN, which will attempt
to decide as a human being. We are getting closer to it by each day, but AI actually is indefinite to
many, although it is no different than any other set of mathematically defined computer operations in its
core. Generating new data from a pre-trained model introduces new challenges to science & technology. In
this work, the design of such an architecture from scratch, solving problems, and introducing alternative
approaches are what has been conducted. Using a deep thinker, Tolstoy, as an object of study is a source
of motivation for the entire research.
Foundations of ANNs: Tolstoy’s Genius Explored Using Transformer Architectureijaia
Artificial Narrow Intelligence is in the phase of moving towards the AGN, which will attempt
to decide as a human being. We are getting closer to it by each day, but AI actually is indefinite to
many, although it is no different than any other set of mathematically defined computer operations in its
core. Generating new data from a pre-trained model introduces new challenges to science & technology. In
this work, the design of such an architecture from scratch, solving problems, and introducing alternative
approaches are what has been conducted. Using a deep thinker, Tolstoy, as an object of study is a source
of motivation for the entire research.
PowerPoint slides from a 2015 Guest Lecture in PSYCH-268A: Computational Neuroscience, Prof. Jeff Krichmar, University of California, Irvine (UCI).
Corresponding publication:
Beyeler*, M., Carlson*, K. D. , Chou*, T-S., Dutt, N., Krichmar, J. L. (2015). CARLsim 3: A user-friendly and highly optimized library for the creation of neurobiologically detailed spiking neural networks. Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland. (*equal contribution)
Foundations of ANNs: Tolstoy’s Genius Explored Using Transformer Architecturegerogepatton
Artificial Narrow Intelligence is in the phase of moving towards the AGN, which will attempt
to decide as a human being. We are getting closer to it by each day, but AI actually is indefinite to
many, although it is no different than any other set of mathematically defined computer operations in its
core. Generating new data from a pre-trained model introduces new challenges to science & technology. In
this work, the design of such an architecture from scratch, solving problems, and introducing alternative
approaches are what has been conducted. Using a deep thinker, Tolstoy, as an object of study is a source
of motivation for the entire research.
Foundations of ANNs: Tolstoy’s Genius Explored using Transformer Architecturegerogepatton
Artificial Narrow Intelligence is in the phase of moving towards the AGN, which will attempt
to decide as a human being. We are getting closer to it by each day, but AI actually is indefinite to
many, although it is no different than any other set of mathematically defined computer operations in its
core. Generating new data from a pre-trained model introduces new challenges to science & technology. In
this work, the design of such an architecture from scratch, solving problems, and introducing alternative
approaches are what has been conducted. Using a deep thinker, Tolstoy, as an object of study is a source
of motivation for the entire research.
Foundations of ANNs: Tolstoy’s Genius Explored Using Transformer Architectureijaia
Artificial Narrow Intelligence is in the phase of moving towards the AGN, which will attempt
to decide as a human being. We are getting closer to it by each day, but AI actually is indefinite to
many, although it is no different than any other set of mathematically defined computer operations in its
core. Generating new data from a pre-trained model introduces new challenges to science & technology. In
this work, the design of such an architecture from scratch, solving problems, and introducing alternative
approaches are what has been conducted. Using a deep thinker, Tolstoy, as an object of study is a source
of motivation for the entire research.
PowerPoint slides from a 2015 Guest Lecture in PSYCH-268A: Computational Neuroscience, Prof. Jeff Krichmar, University of California, Irvine (UCI).
Corresponding publication:
Beyeler*, M., Carlson*, K. D. , Chou*, T-S., Dutt, N., Krichmar, J. L. (2015). CARLsim 3: A user-friendly and highly optimized library for the creation of neurobiologically detailed spiking neural networks. Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland. (*equal contribution)
Introduction to ANN Principles and its Applications in Solar Energy TechnologyAli Al-Waeli
I presented the slides in 2022, at SERI, UKM. The aim of the presentation is to provide an overview of AI, Machine Learning and ANN. Moreover, to introduce their application in Solar energy technologies.
Artificial Neural Network and its Applicationsshritosh kumar
Abstract
This report is an introduction to Artificial Neural
Networks. The various types of neural networks are
explained and demonstrated, applications of neural
networks like ANNs in medicine are described, and a
detailed historical background is provided. The
connection between the artificial and the real thing is
also investigated and explained. Finally, the
mathematical models involved are presented and
demonstrated.
Brain-Inspired Computation based on Spiking Neural Networks ...Jorge Pires
On this live, prof. Kasabov gives us a gentle overview of Spiking Neural Networks, and their current applications
Full live here, with discussion: https://www.youtube.com/watch?v=niAannUB3pc&t=232s
Have fun 😎😂😁😀
Vulkan and DirectX12 share many common concepts, but differ vastly from the APIs most game developers are used to. As a result, developing for DX12 or Vulkan requires a new approach to graphics programming and in many cases a redesign of the Game Engine. This lecture will teach the basic concepts common to Vulkan and DX12 and help developers overcome the main problems that often appear when switching to one of the new APIs. It will explain how those new concepts will help games utilize the hardware more efficiently and discuss best practices for game engine development.
For more, visit http://developer.amd.com/
AMD’s math libraries can support a range of programmers from hobbyists to ninja programmers. Kent Knox from AMD’s library team introduces you to OpenCL libraries for linear algebra, FFT, and BLAS, and shows you how to leverage the speed of OpenCL through the use of these libraries.
Review the material presented in the AMD Math libraries webinar in this deck.
For more:
Visit the AMD Developer Forums:http://devgurus.amd.com/welcome
Watch the replay: www.youtube.com/user/AMDDevCentral
Follow us on Twitter: https://twitter.com/AMDDevCentral
This is the slide deck from the popular "Introduction to Node.js" webinar with AMD and DevelopIntelligence, presented by Joshua McNeese. Watch our AMD Developer Central YouTube channel for the replay at https://www.youtube.com/user/AMDDevCentral.
This presentation accompanies the webinar replay located here: http://bit.ly/1zmvlkL
AMD Media SDK Software Architect Mikhail Mironov shows you how to leverage an AMD platform for multimedia processing using the new Media Software Development Kit. He discusses how to use a new set of C++ interfaces for easy access to AMD hardware blocks, and shows you how to leverage the Media SDK in the development of video conferencing, wireless display, remote desktop, video editing, transcoding, and more.
An Introduction to OpenCL™ Programming with AMD GPUs - AMD & Acceleware WebinarAMD Developer Central
This deck presents highlights from the Introduction to OpenCL™ Programming Webinar presented by Acceleware & AMD on Sept. 17, 2014. Watch a replay of this popular webinar on the AMD Dev Central YouTube channel here: https://www.youtube.com/user/AMDDevCentral or here for the direct link: http://bit.ly/1r3DgfF
Learn more about DirectGMA in this blog post: bit.ly/AMDDirectGMA
AMD has introduced Direct Graphics Memory Access in order to:
‒ Makes a portion of the GPU memory accessible to other devices
‒ Allows devices on the bus to write directly into this area of GPU memory
‒ Allows GPUs to write directly into the memory of remote devices on the bus supporting DirectGMA
‒ Provides a driver interface to allow 3rd party hardware vendors to support data exchange with an AMD GPU using DirectGMA
‒ and more
View the accompanying blog post here: bit.ly/AMDDirectGMA
This Webinar explores a variety of new and updated features in Java 8, and discuss how these changes can positively impact your day-to-day programming.
Watch the video replay here: http://bit.ly/1vStxKN
Your Webinar presenter, Marnie Knue, is an instructor for Develop Intelligence and has taught Sun & Oracle certified Java classes, RedHat JBoss administration, Spring, and Hibernate. Marnie also has spoken at JavaOne.
The Small Batch (and other) solutions in Mantle API, by Guennadi Riguer, Mant...AMD Developer Central
This presentation discusses the Mantle API, what it is, why choose it, and abstraction level, small batch performance and platform efficiency.
Download the presentation from the AMD Developer website here: http://bit.ly/TrEUeC
Inside XBox One by Martin Fuller from the Sweden Game Developers Conference, June 2, 2014, Stockholm, Sweden. View other presentations here: http://bit.ly/TrEUeC
Computer Vision Powered by Heterogeneous System Architecture (HSA) by Dr. Ha...AMD Developer Central
Computer Vision Powered by Heterogeneous System Architecture (HSA) by Dr. Harris Gasparakis, AMD, at the Embedded Vision Alliance Summit, May 2014.
Harris Gasparakis, Ph.D., is AMD’s OpenCV manager. In addition to enhancing OpenCV with OpenCL acceleration, he is engaged in AMD’s Computer Vision strategic planning, ISVs, and AMD Ventures engagements, including technical leadership and oversight in the AMD Gesture product line. He holds a Ph.D. in theoretical high energy physics from YITP at SUNYSB. He is credited with enabling real-time volumetric visualization and analysis in Radiology Information Systems (Terarecon), including the first commercially available virtual colonoscopy system (Vital Images). He was responsible for cutting edge medical technology (Biosense Webster, Stereotaxis, Boston Scientific), incorporating image and signal processing with AI and robotic control.
Productive OpenCL Programming An Introduction to OpenCL Libraries with Array...AMD Developer Central
In this webinar presentation, ArrayFire COO Oded Green demonstrates best practices to help you quickly get started with OpenCL™ programming. Learn how to get the best performance from AMD hardware in various programming languages using ArrayFire. Oded discusses the latest advancements in the OpenCL™ ecosystem, including cutting edge OpenCL™ libraries such as clBLAS, clFFT, clMAGMA and ArrayFire. Examples are shown in real code for common application domains.
Watch the webinar here: http://bit.ly/1obT0M2
For more developer resources, visit:
http://arrayfire.com/
http://developer.amd.com/
Follow us on Twitter: https://twitter.com/AMDDevCentral
See info in the slides for more contact information and resource links!
Rendering Battlefield 4 with Mantle by Johan Andersson - AMD at GDC14AMD Developer Central
Johan Andersson will show how the Frostbite 3 game engine is using the low-level graphics API Mantle to deliver significantly improved performance in Battlefield 4 on PC and future games from Electronic Arts in this presentation from the 2014 Game Developers Conference in San Francisco March 17-21. Also view this and other presentations on our developer website at http://developer.amd.com/resources/documentation-articles/conference-presentations/
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
4. NEURAL
NETWORKS:
ORIGIN,
FEATURES,
APPLICATIONS
OUTLINE
! From
Biological
to
Ar?ficial
Neural
Networks
(ANN)
! ANN
Applica?ons
‒ Applica?on
categories
‒ Examples
! Why
ANN?
! Why
Spiking
Neural
Network
(SNN)?
4
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
5. FROM
BIOLOGICAL
TO
ARTIFICIAL
NEURAL
NETWORK
(ANN)
NEURAL
NETWORKS:
ORIGIN,
FEATURES,
APPLICATIONS
! ANN
is
simplifica?on
of
biological
neural
network
! ANN
consists
of
simple
elements
(neurons)
analogous
to
the
biological
neurons
in
the
brain.
! The
neurons
are
connected
by
weighted
links
and
form
a
network.
! The
links
pass
signals
(numbers)
from
one
neuron
to
another.
Neurons
operate
on
the
weighted
signals
and
retransmit
the
results
! The
network
can
learn
by
adjus?ng
the
weights
(the
behavior
is
encoded
in
weights).
5
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
6. ANN
APPLICATION
CATEGORIES
NEURAL
NETWORKS:
ORIGIN,
FEATURES,
APPLICATIONS
18%
16%
14%
12%
10%
8%
6%
4%
2%
0%
!
Based
on
patent
and
applica?on
search
(US
Patent
and
Trademark
Office,
EU
Patent
Office,
Google
Patent
Search.
Conducted
in
2012
by
students
of
Machine
Learning
class
(Dr.
Leon
Reznik,
RIT)
6
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
7. WHY
ANN?
EXAMPLES
NEURAL
NETWORKS:
ORIGIN,
FEATURES,
APPLICATIONS
! Recogni@on
‒ Character
(e.g.
mail),
speech,
image
(e.g.
image
clustering),
odor
(e.g.
locust
antennal
lobe),
face
and
emo?on
! Gaming
‒ AI
features
in
games
! Robo@cs
‒ Vision,
spa?al
naviga?on
and
planning
(e.g.
mental
maps
with
place
cells),
posi?oning,
decision
making
! Control
‒ Missile
guidance
‒ An?-‐lock
brakes
(Ford)
‒ Self-‐driving
cars,
UAVs
! Crime
preven@on
and
security
‒ Bomb
sniffer
(JFK
airport)
‒ Credit
card
fraud
detec?on
(Visa)
7
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
! Biomedical
‒ Neuroscience:
Brain
modeling
and
simula?on
‒ US
BRAIN
Ini?a?ve
(expected
300
EB/day)
‒ EU
Human
brain
project
‒ Neurology:
(e.g.
disease
modeling
and
forecas?ng,
ModelDB)
‒ Cardiology:
(e.g.
adap?ve
biventricular
pacemaker)
‒ Prosthesis:
BCI
neuromosphic
chips
! Financial
analysis
‒ Mortgage
risk
evalua?on
(AVCO,
Irvine)
‒ Currency
trading
(Ci?bank)
!
Difficul@es
‒ Need
to
compute
fast
but
problem
size
is
large
‒ How
to
get
the
right
ANN
circuit
for
an
applica?on?
8. WHY
ANN?
NEURAL
NETWORKS:
ORIGIN,
FEATURES,
APPLICATIONS
! Novel
algorithms.
‒ Conven?onal
algorithms
performance
is
not
sa?sfactory
in
numerous
problems
with
dynamic
changes
(e.g.
face
recogni?on
may
fail
if
the
view
angle
is
different
or
the
person
is
smiling).
! Learning,
adaptability.
‒ Con?nuously
learn
from
the
available
data
and
adapt
to
new
condi?ons.
! Reliability.
‒ Performance
tends
to
degrade
gracefully
under
par?al
damage.
Parts
of
networks
can
learn
to
perform
func?on
of
damaged
parts.
In
contrast,
most
programs
and
engineered
systems
are
brijle:
if
you
remove
some
arbitrary
parts,
very
likely
the
whole
system
ceases
to
func?on.
8
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
! Low
power.
Neuromorphic
engineering
‒ Switching
speed
of
biological
neurons
is
less
than
1KHz
(CPU
3GHz)
‒ Switching
energy
of
biological
neurons
~
1.0E-‐17
Joules/op
(CPU
1.0E-‐5
joules/op)
‒ Conduc?on
speed
of
biological
neural
network
~
100
m/s
! Parallel.
‒ Brain
performs
massively
parallel
computa?ons
very
efficiently.
Data
and
processing
have
global
impact.
For
example,
complex
visual
percep?on
occurs
within
less
than
100
ms,
that
is,
10
processing
steps.
! AI.
Consciousness.
Intelligence.
Self-‐
awareness.
9. WHY
SNN?
NEURAL
NETWORK
CATEGORIES
NEURAL
NETWORKS:
ORIGIN,
FEATURES,
APPLICATIONS
Learning Ability
Biological
ASNN
iSNN
x ty
SOM
ple
Neural
Gas
m
Co
LVQ
Recurrent
RBF
MLP
Hopfield
ADALINE
Rosenblaj
Time Dynamics
! Which
level
of
abstrac?on
to
choose?
! Which
one
is
the
right
for
the
target
applica?on?
! Point-‐to-‐point
connected
spiking
neural
network
(SNN):
?me
(spikes),
polychroniza?on
(memory
capacity),
unsupervised
learning
(synap?c
plas?city)
9
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
12. HETEROGENEOUS
IMPLEMENTATION:
SIMULATORS
AND
ABSTRACTION
LEVEL
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Popula?on
model
‒ Nengo
! Point-‐neuron
network
models
! Compartmental
neuron
and
membrane
models
‒ NEST
‒ PCSIM
‒ Brian
12
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
‒ NEURON
‒ GENESIS
! Reac?on-‐diffusion
model
of
biochemical
signaling
pathways
‒ STEPS
13. SNN
MODELS:
TRADEOFFS
SNN
SIMULATION
PRINCIPLES
HH
IZ
! Integrate-‐and-‐Fire
(IF):
simple,
but
has
poor
spiking
response
! Hodgkin-‐Huxley
(HH):
has
reach
response,
but
complex
IF
13
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
! Izhikevich
(IZ):
simple,
has
reach
response,
but
phenomenological
15. TIME-‐DRIVEN
(SYNCHRONOUS)
SIMULATION
SNN
SIMULATION
PRINCIPLES
! Events
aligned
to
?me
grid
‒ Can
update
all
neurons
at
the
same
?me
‒ Good
for
parallel
implementa?on
! Time
quan?za?on
error
‒ Delayed
or
missing
events
‒ Can
be
controlled
by
size
of
dt:
the
smaller
the
size
the
smaller
the
error,
but
the
more
computa?on
per
unit
?me
15
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
16. EVENT-‐DRIVEN
(ASYNCHRONOUS)
SIMULATION
SNN
SIMULATION
PRINCIPLES
! Events
are
unique
in
?me:
‒ A
single
event
can
change
the
state
of
the
whole
system
‒ Have
to
update
neurons
sequen?ally
in
the
order
of
events
‒ Minimum
transmission
latency
is
unknown
‒ Assumes
analy?cal
solu?on
for
the
model
equa?ons
‒ …
or
?med
event-‐driven
update
! Time
quan?za?on
error
‒ No
error
caused
by
simula?on
type
‒ Bejer
event
accuracy
‒ Good
for
STDP
16
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
17. TIMED
EVENT-‐DRIVEN
(HYBRID)
SIMULATION
SNN
SIMULATION
PRINCIPLES
! Events
are
unique
in
?me:
‒ A
single
event
can
change
the
state
of
the
whole
system,
but
not
within
the
minimum
transmission
delay
‒ Time
grid:
dt
is
equal
to
the
minimum
delay
‒ Update
all
neurons
at
the
same
?me
every
dt
increment
‒ Also
between
dt
increments
update
every
neuron
in
the
order
of
events
it
receives
within
the
increment.
‒ Good
for
parallel
implementa?on,
but
there
is
computa?on
divergence
across
neurons.
! Time
quan?za?on
error
‒ No
error
caused
by
simula?on
type
‒ Bejer
event
accuracy
‒ Good
for
STDP
17
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
18. NUMERICAL
INTEGRATION
METHODS
SNN
SIMULATION
PRINCIPLES
! Mo@va@on.
Need
to
solve
ini?al
value
problem
(IVP)
! Euler.
Compute
next
y
based
on
tangent
to
current
y.
! Modified
Euler.
Predict
with
Euler,
correct
with
average
slope.
! Runge-‐KuXa
(4th
Order).
Evaluate
and
average.
! Bulirsch–Stoer
‒ Uses
Modified
midpoint
method
with
evalua?on
and
error
tolerance
check
using
extrapola?on
with
ra?onal
func?ons.
Provides
adap?ve
order.
Generally
more
suited
for
smooth
func?ons.
! Parker-‐Sochacki
‒ Uses
expression
of
IVP
in
terms
of
power
series.
Provides
adap?ve
order.
18
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
20. NUMERICAL
INTEGRATION
METHODS:
PARCKER-‐SOCHACKI
SNN
SIMULATION
PRINCIPLES
! A
typical
IVP
! Assume
that
solu?on
func?on
can
be
represented
with
power
series.
! Therefore,
its
deriva?ve
based
on
Maclaurin
series
proper?es
is
! As
a
result:
20
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
21. NUMERICAL
INTEGRATION
METHODS:
PARCKER-‐SOCHACKI
SNN
SIMULATION
PRINCIPLES
! If
is
linear:
! Ship
it
to
eliminate
constant
term:
! As
a
result,
the
equa?on
becomes:
! Benefit:
adap?ve
order
and
error
tolerance
control
‒ Local
Lipschitz
constant
determines
the
number
of
itera?ons
for
achieving
certain
error
tolerance:
! With
finite
order
N:
! Parallelism:
‒ Loop-‐level
parallelism
‒ Parallel
reduc?on
21
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
22. SUMMARY
SNN
SIMULATION
PRINCIPLES
! Result
! Neuron/Synapse
Model
! Simula?on
Type
! Integra?on
Method
! Applica?on
! Requirements
22
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
24. OUTLINE
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Simula?on
Flow
‒ Synchronous
‒ Hybrid
‒ Combined
! Implementa?on
of
Hybrid
Simula?on
Type
‒ Simula?on
Flow
‒ Simula?on
Phases
‒ Update
‒ Expand
‒ Sort
‒ Results
! Heterogeneous
Implementa?on
of
Synchronous
Simula?on
Type
‒ NEST
Simulator
‒ Sopware
Architecture
24
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
25. SYNCHRONOUS
SIMULATION
FLOW
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Simula?on
step
(dt)
has
two
phases:
‒ Update:
‒ Compute
new
state
for
all
neurons.
‒ Detect
spiked
neurons
and
process
them
separately
to
update
spike
history
(divergence
reduc?on).
‒ Propaga?on:
‒ Expand
spikes
to
arriving
events.
25
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
26. HYBRID
SIMULATION
FLOW
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Simula?on
step
(dt)
has
two
phases:
‒ Update:
‒ Compute
new
state
for
all
neurons
at
the
?mes
of
arriving
spikes
(event-‐driven).
‒ Detect
spiked
neurons
and
process
them
separately
to
compute
spike
?me
and
update
spike
history
(divergence
reduc?on).
‒ Propaga?on:
‒ Expand
spikes
to
arriving
events.
‒ Sort
the
events
that
are
due
for
delivery
in
the
current
?me
step
by
arrival
?me
for
each
neuron.
‒ Create
a
pointer
array
that
maps
neurons
to
their
sorted
events.
26
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
27. COMBINED
SIMULATION
FLOW
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Exchange
spikes
between
compute
nodes
(MPI)
‒ Spike
is
(?me
stamp,
source
neuron
ID)
! Store
spikes
in
the
spike
ring
buffer
‒ How
many
ring
segments?
int(max
delay
/
min
delay)
‒ The
ring
‘rotates’
every
step
by
one
segment
! Expand
spikes
‒ Spike
segments
are
matched
with
relevant
delay
segments
(synap?c
connec?vity
matrix)
‒ Arrival
?me
is
computed
‒ Synap?c
events
due
filtered
! Sort
synap?c
events
by
arrival
?me
for
each
target
neuron
(event-‐driven
only)
! Update
neurons
! Update
synapses
! Gather
new
spikes
27
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
28. IMPLEMENTATION
OF
HYBRID
SIMULATION:
UPDATE
PHASE
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Wave-‐fronts
(WFs)
work
on
their
segments
of
neurons
represented
by
parameters
and
state
stored
in
global
memory
(GM)
! A
work-‐item
(WI)
takes
a
neuron
and
updates
its
state
at
every
arriving
event
! The
state
is
stored
back
to
GM
! Spike
data
is
accumulated
in
local
data
store
(LDS)
and
flushed
to
GM
periodically.
! Spiked
neurons
are
processed
in
a
separate
kernel
(divergence
reduc?on)
‒ Spike
?me
is
computed
with
Newton
Raphson
method
(NR)
‒ Spiked
neurons
are
updated
for
the
rest
of
arriving
events.
28
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
29. IMPLEMENTATION
OF
HYBRID
SIMULATION:
EXPAND
PHASE
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Load
source
spike
packets
from
GM
and
stored
them
in
con?guous
array
in
LDS.
! Load
synap?c
pointer
to
LDS.
‒ Each
neuron
is
connected
to
100s
or
even
1000s
of
other
neurons.
Synap?c
pointer
describes
where
to
get
synap?c
data
for
target
neurons
for
known
spike
source
neuron.
! Main
loop
‒ A
WF
picks
a
source
spike
(?me
stamp,
source
neuron
ID)
and
the
pointer
‒ A
WI
loads
synap?c
data
for
a
target
neuron,
computes
arrival
?me
and
stores
synap?c
event
in
the
ring
buffer
in
GM.
! Alone
the
way
the
sort
histogram
(required
in
radix
sort)
is
loaded
and
stored
in
LDS.
It
is
updated
reflec?ng
the
newly
created
synap?c
events.
29
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
30. IMPLEMENTATION
OF
HYBRID
SIMULATION:
SORT
PHASE
SNN:
HETEROGENEOUS
IMPLEMENTATION
Radix
sort
example:
1
bit
radix.
LSD
sort.
! We
need
to
order
synap?c
events
by
arrival
?me
and
by
target
ID
! Radix
sort:
select
next
radix
from
LSD
to
MSD
and
group
numbers
based
on
radix
value
from
smallest
to
largest
‒ Group
numbers
based
on
current
radix
and
compute
histogram
(count
of
numbers
with
the
same
radix
value)
‒ Scan
histogram:
compute
prefix
sum
(global
offset
for
the
next
grouping).
! 8
passes
for
32-‐bit
addressing
and
4-‐bit
radix.
30
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
31. IMPLEMENTATION
OF
HYBRID
SIMULATION:
PERFORMANCE
SNN:
HETEROGENEOUS
IMPLEMENTATION
Network
Size
(neurons)
Average
Synapses
per
Neuron
Average
Events
per
Step
Average
Spikes
per
Step
Total
Synapse
Count
(millions)
“Tahi@”
GPU
Time
2,100,000
90
230,000
2,522
190
13.5
131,000
1,458
370,000
257
191
5.7
16,000
11,677
300,000
25
191
3.2
! Size-‐connec?on
scalability
in
mul?-‐precision
networks
with
per-‐WF
precision
alloca?on
! 1000
itera?ons,
250
us
step
! Randomly-‐connected
SNN
with
only
AMPA
synapses
! Speedups
up
to
100
depending
on
configura?on
and
compared
devices
31
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
per
Step,
(ms)
32. HETEROGENEOUS
IMPLEMENTATION:
SIMULATOR
ARCHITECTURE
SNN:
HETEROGENEOUS
IMPLEMENTATION
!
!
!
!
Interface:
Python
–
SLI
–
Network
class
(C++)
Object-‐oriented:
Nodes
–
Connec?ons
–
Events
Network:
administrates
node
connec?ons
Scheduler:
orchestrates
simula?on
‒ Node
management:
update,
prepare,
finalize
‒ Execu?on
type
selec?on:
serial,
p-‐threads,
OpenMP
‒ Step
scheduling
‒ Event
transmission
via
Communicator
! Communicator
‒ Inter-‐process
communica?on
‒ MPI
! Features
‒ Primarily
used
as
a
vehicle
for
neuroscience
research
‒ Generic,
suitable
for
SNN
applica?ons
‒ Both
?me-‐
and
event-‐driven
simula?on
types
‒ Flexible
node
dynamics,
a
variety
of
built-‐in
models
‒ Communica?on
infrastructure
to
deliver
both
discrete
and
con?nuous
events
at
the
same
?me.
‒ Emphasis
on
correctness,
performance
and
scalability
32
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
33. HETEROGENEOUS
IMPLEMENTATION:
SOFTWARE
ARCHITECTURE
SNN:
HETEROGENEOUS
IMPLEMENTATION
! Simplified
UML
diagram
for
heterogeneous
part
of
implementa?on
! Neuron
model
templates
(single
and
double
precision)
with
OpenCL™
update
phase
! Object-‐oriented
design
with
shared
vector
members
(data
redundancy
reduc?on)
! STL-‐like
containers
with
OpenCL™
memory
/
buffer
types
underneath
! On-‐a-‐fly
CPU-‐GPU
execu?on
steering:
adaptability
! Data
structure
size
stability:
sta?s?cal
monitoring,
steering,
error
repor?ng
33
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
34. CONCLUSION
HETEROGENEOUS
IMPLEMENTATION
OF
NEURAL
NETWORK
ALGORITHMS
! Thank
You!
34
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
35. LITERATURE
HETEROGENEOUS
IMPLEMENTATION
OF
NEURAL
NETWORK
ALGORITHMS
!
R.
Breje,
et
al.,
"Simula?on
of
networks
of
spiking
neurons:
A
review
of
tools
and
strategies,"
Journal
of
Computa0onal
Neurscience,
vol.
23,
no.
3,
pp.
349-‐398,
2007.
!
B
Gaster,
D
R
Kaeli,
L
Howes,
and
P
Mistry,
Heterogeneous
Compu?ng
with
OpenCL
™
:
Morgan
Kaufmann
Pub,
2011.
!
T
Harada
and
L
Howes.
(2011,
Dec.)
“Introduc?on
to
GPU
Radix
Sort.”
Heterogeneous
Compute.
[Online].
!
E.
M.
Izhikevich,
"Simple
model
of
spiking
neurons,"
Neural
Networks,
IEEE
Transac?ons
on,
vol.
14,
pp.
1569-‐-‐1572,
2003.
!
R
Stewart
and
W
Bair,
"Spiking
neural
network
simula?on:
numerical
integra?on
with
the
Parker-‐Sochacki
method,"
Journal
of
Computa?onal
Neuroscience,
vol.
27,
no.
1,
pp.
115-‐133,
August
2009.
!
D
Yudanov,
L
Reznik,
"Scalable
mul?-‐precision
simula?on
of
spiking
neural
networks
on
GPU
with
OpenCL
™."
Neural
Networks
(IJCNN),
The
2012
Interna?onal
Joint
Conference
on.
IEEE,
2012.
35
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL
36. THANKS
HETEROGENEOUS
IMPLEMENTATION
OF
NEURAL
NETWORK
ALGORITHMS
!
Wayne
Burleson
!
Mayank
Daga
!
Markus
Diesmann
!
Joseph
Dinh
!
Tan
Ho
!
Aus?n
Hung
!
Jeremy
Johnson
!
John
Keaty
!
Bingley
Li
!
Gewal?g
Marc-‐Oliver
!
Saul
Mar?nez
!
Haibin
Niu
!
Kyle
Pour
!
Jason
Shantz
!
Jason
Tang
!
Yury
Zaytsev
36
|
Heterogeneous
implementa?on
of
Neural
network
algorithms
|
NOVEMBER
2013
|
CONFIDENTIAL