In this deck from GTC 2019, Seongchan Kim, Ph.D. presents: How Deep Learning Could Predict Weather Events.
"How do meteorologists predict weather or weather events such as hurricanes, typhoons, and heavy rain? Predicting weather events were done based on supercomputer (HPC) simulations using numerical models such as WRF, UM, and MPAS. But recently, many deep learning-based researches have been showing various kinds of outstanding results. We'll introduce several case studies related to meteorological researches. We'll also describe how the meteorological tasks are different from general deep learning tasks, their detailed approaches, and their input data such as weather radar images and satellite images. We'll also cover typhoon detection and tracking, rainfall amount prediction, forecasting future cloud figure, and more."
Watch the video: https://wp.me/p3RLHQ-k2T
Learn more: http://en.kisti.re.kr/
and
https://www.nvidia.com/en-us/gtc/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
2020 vision - the journey from research lab to real-world productKTN
This presentation, delivered by Jag Minhas, CEO and Founder, Sensing Feeling, was the first presentation of the Implementing AI: Vision Systems Webinar.
Density-Based Multi feature Background Subtraction with Support Vector MachineNAZNEEN BEGUM
itz a Low cost monitoring self adaptive method using background subraction.
Our project is based on security that is
used to monitor the moving objects and
store the images …
notify the owner about the slight changes by sending a message to the owner on his/her mobile phone …
For this we are making use of BACKGROUND SUBTRACTION METHOD
FINken eYe: Visual SLAM-Based Position EstimationMichael Mera
Development of a system, hardware and software, to implement Visual SLAM-based Position Estimation on the FINken Copters in the SwarmLab at The Otto-von-Guericke University Magdeburg. We use Semi-Direct Visual Odometry SVO.
Preview delle ultime novità di prodotto Sourcefire IPS Entriamo in dettaglio delle novità di prodotto annunciate da Sourcefire nell\’ultimo mese, incluso:
New 3D8000 Series Sensors with FirePOWER
New Defense Center Models
New IPSx Solution
Early Benchmarking Results for Neuromorphic ComputingDESMOND YUEN
An update on the Intel Neuromorphic Research Community’s growth and benchmark results, including the addition of new corporate members and numerous new benchmarking updates computed on Intel’s neuromorphic test chip, Loihi.
The problem of scene classification in surveillance footage is of great importance for ensuring security in public areas. With challenges such as low quality feeds, occlusion, viewpoint variations, background clutter etc. The task is both challenging and error-prone. Therefore it is important to keep the false positives low to maintain a high accuracy of detection. In this paper, we adapt high performing CNN architectures to identify abandoned luggage in a surveillance feed. We explore several CNN based approaches, from Transfer Learning on the Imagenet dataset to one-shot detection using architectures such as YOLOv3. Using network visualization techniques, we gain insight into what the neural network sees and the basis of classification decision. The experiments have been conducted on real world datasets, and highlights the complexity in such classifications. Obtained results indicate that a combination of proposed techniques outperforms the individual approaches.
Author: Utkarsh Contractor
Similar to SSII2020TS: Event-Based Camera の基礎と ニューラルネットワークによる信号処理 〜 生き物のように「変化」を捉えるビジョンセンサーと 「変化」を処理するニューラルネットワーク 〜 (20)
SSII2021 [SS2] Deepfake Generation and Detection – An Overview (ディープフェイクの生成と検出)SSII
SSII2021 [SS2] Deepfake Generation and Detection – An Overview (ディープフェイクの生成と検出)
6/10 (木) 14:30~15:00
講師:Huy H. Nguyen 氏(総合研究大学院大学/国立情報学研究所)
概要: Advances in machine learning and their interference with computer graphics allow us to easily generate high-quality images and videos. State-of-the-art manipulation methods enable the real-time manipulation of videos obtained from social networks. It is also possible to generate videos from a single portrait image. By combining these methods with speech synthesis, attackers can create a realistic video of some person saying something that they never said and distribute it on the internet. This results in loosing social trust, making confusion, and harming people’s reputation. Several countermeasures have been proposed to tackle this problem, from using hand-crafted features to using convolutional neural network. Some countermeasures use images as input and other leverage temporal information in videos. Their output could be binary (bona fide or fake) or muti-class (deepfake detection), or segmentation masks (manipulation localization). Since deepfake methods evolve rapidly, dealing with unseen ones is still a challenging problem. Some solutions have been proposed, however, this problem is not completely solved. In this talk, I will provide an overview on both deepfake generation and deepfake detection/localization. I will mainly focus on image and video domain and also introduce some audiovisual-based methods on both sides. Some open discussions and future directions are also included.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
21. ニューラルネットワークによる処理, SSII2020
Biological Vision
“Retina is sensitive to temporal brightness gradients”
“Retina is blind to static scenes in absence of eye movements ”
22
Receptive fields of single neurons in the cat’s striate cortex
David H Hubel et.al.,1959, Nobel prize 1981
23. ニューラルネットワークによる処理, SSII2020
History
24
2010
・ 1991 Mahowald et.al
1990 20202000
・2020 Gen.4
1280 x 720 w/ SONY
・2018 Celexl V
1280 x 960
Y.Sekikawa, イベントカメラの研究動向と,
2015-
2014-
・2020 DVXplorer
640 x 480
2012-
・2018 Samsung Gen.3
Event-Based Sencing Device
2017- ・2020 GaAI One
・2018 DyNap CNN2017-
・2014 IBM TrueNorth
・2018 Intel Loihi
・2018 Stanford Braindrop
Event-Based Processing Device
・2009 Lichtsteiner et.al
128x128
24. ニューラルネットワークによる処理, SSII2020
Comparison between different event cameras
25
Prophesee(Chronocam) iniVation(iniLabs) Samsung Celepixel(Hillhouse)
Latest version ATIS-Gen4 DAVIS346 DVS Gen.4 CeleX-V
Resolution CD : 1280 x 720 CD+EM : ? 346x260 1280 x 960 1280 x 800
Pixel pitch CD : 4.86μm CD+EM : ? 18.5μm 4.95μm 9.8μm
Intensity information
EM: Exposure Measurement
130dB
Event resets a capacitor to a
high voltage. Brighter →faster
discharges
APS: Active pixel sensor
56.7dB
Similar to standard frame
N/A
Intensity at event-rate
Other feature
/ Info
IMARGO industrial Camera
Joint dev. with SONY
Sony acquires Insightness
DVXplorer
(640 x 480 no intensity)
In-home monitoring camera
Event-wise optical flow
Commercial product for
DSM in China
Y.Sekikawa, イベントカメラの研究動向と,
31. ニューラルネットワークによる処理, SSII2020
Event generation Model
§Each pixel asynchronously report intensity changes
Image from Kim et.al., et.al, Simultaneous mosaicing and tracking with an event camera
32Y.Sekikawa, イベントカメラの研究動向と,
33. 34
Δ" = $!% ≈ −("/(* ⋅ ,Δ-
v Temporal relation
v Spatial relation (optical flow constraint)
) #$, %$ − ) #$, %$%& ≈ ,
$∈(
('$
Note: No event when image gradient is perpecdicular to motion −("/(*
,
Time
LogIntensity
!
""""#$
34
34. ニューラルネットワークによる処理, SSII2020
Event-Based Camera
35
• High speed (1µs)
• Low data rate/Sparse (0-30Mbps)
• No motion blur
• High dynamic range (130dB)
✔
✔
Fram
e-based
cam
era
Speed[fps]
EnergyComsumption[W]
Event-based camera
Datarate[bps]
Price[$]
✔
Y.Sekikawa, イベントカメラの研究動向と,
35. ニューラルネットワークによる処理, SSII2020
Difficulties when dealing with event data
§Sparse data representation: Frame-based alg. cannot be applied
Image from Gehrig et.al, Asynchronous, Photometric Feature Tracking using Events and Frames
36Y.Sekikawa, イベントカメラの研究動向と,
§Motion dependent data: Association in SLAM / Generalization in ML
Frame:
Motion Independent
Event (Histogram):
Motion Dependent
36. ニューラルネットワークによる処理, SSII2020
Wide Range of Usage
Algorithm
§Tracking
§Optical Flow
§Visual odometry
§SLAM
§Image Reconstruction
§Stereo depth estimation
§3D measurement with SL
§Object Recognition
§Etc..
37
Applications
§Surveillance at Home
§Obstacle avoidance
§UAV, automotive
§Bin-picking
§Gesture recognition
§Etc..
Y.Sekikawa, イベントカメラの研究動向と,
38. ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
39Y.Sekikawa, イベントカメラの研究動向と,
Algorithm
• Feature Tracking
• Optical Flow (OF)
• Visual Odometry (VO)
• Simultaneous Localization and Mapping (SLAM)
• 3D Reconstruction
• Intensity Reconstruction (IR)
Model-based Processing
Setup
Image from Gallego et.al., Event-based, 6-DOF Camera Tracking from Photometric Depth Maps
Geometry
Planer/Known/3D? Texture
Known/Estimate?
Known/Estimate?
Rotation/SE(2)/SE(3)?
Environment
Static/Dynamic
Use image as
Proxy/Direct?
Ext. sensor
/Reconstruct ?
Algorithm
Input
39. ニューラルネットワークによる処理, SSII2020
Speed Invariant Time Surface
for Learning to Detect Corner Points with Event-Based Cameras
Manderscheid et.al., CVPR2019
Very fast/robust corner tracking in challenging illumination conditions
Corner detetion using SI time surface
Event to Time Surface
SI:Speed Invariant
40
Corner tracking
Event only
Y.Sekikawa, イベントカメラの研究動向と,
Image (on the bottom) from Alzugaray, et.al., Asynchronous Corner Detection and Tracking for Event Cameras in Real Time
Video from Alzugaray et.al., Asynchronous Corner Detection and Tracking for Event Cameras
Tracking by simple Nearest Neighbor association
40. ニューラルネットワークによる処理, SSII2020
EKLT: Asynchronous Photometric Feature Tracking Using Events and Frames
Gehrig et.al., Depts. Informatics and Neuro informatics@ETH, ECCV2018 IJCV2019
159
! = arg min
!
( − )
Very fast/rubust feature tracking in challenging illumination conditions
Compare intensity increment from event with prediction from frame
41
Feature tracking
Event + Frame
Y.Sekikawa, イベントカメラの研究動向と,
42. ニューラルネットワークによる処理, SSII2020
Simultaneous Optical Flow and Intensity Estimation from an Event Camera
Bardow et.al., Imperial College Dyson Lab., CVPR2016
43
32
Data term
"#
Optical flow
Intensity & OF estimation at high rate in challenging illumination conditions
Joint optimization using optical flow constrains
IR+OF
Event
Y.Sekikawa, イベントカメラの研究動向と,
43. ニューラルネットワークによる処理, SSII2020
Continuous-time Intensity Estimation Using Event Cameras
Scheerlinck et.al., ACCV2018
37
Real-time & high rate intensity estimation in challenging inllumination conditions
Complementary fusion of frame and event
44
IR
Event+Frame
Y.Sekikawa, イベントカメラの研究動向と,
45. ニューラルネットワークによる処理, SSII2020
Simultaneous Mosaicing and Tracking with an Event Camera
§Kim et.al, Imperial College London, BMVC2014
46
Localization (PF)
39
Mapping (EKF based IR)
SLAM (rotation only) in challenging illumination conditions
Mapping by intensity reconstructing (Pioneering work for event-based SLAM)
Intencity MAP
SLAM(SO(3))
Event
Y.Sekikawa, イベントカメラの研究動向と,
47. ニューラルネットワークによる処理, SSII2020
Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera
Kim et.al, Imperial College London, ECCV2016 (Best Paper)
48
47
Full 6DOF SLAM in challenging illumination conditions
Extension of Kim2016 to SE(3) by incorporation depth estimation
SLAM(SO(3))
Event
Y.Sekikawa, イベントカメラの研究動向と,
48. ニューラルネットワークによる処理, SSII2020
Focus Is All You Need: Loss Functions For Event-based Vision
Guillermo et.al., UZH@ETH, CVPR2018, CVPP2019
49
120
Efficient motion (OF) estimation in challenging illumination conditions
OF estimation w/o intensity. Novel focus-based loss
OF
Event
Y.Sekikawa, イベントカメラの研究動向と,
53. ニューラルネットワークによる処理, SSII2020
EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking
and Mapping in Real-time
Rebecq et.al., UZH@ETH, RAL2016
54
48
Very fast SLAM (500Hz on CPU) in challenging illumination conditions
Utilize DSI (No IR, edge-map alignment suffice)
SLAM(SE(3))
Event
Y.Sekikawa, イベントカメラの研究動向と,
55. ニューラルネットワークによる処理, SSII2020
Ultimate SLAM? combining events, images, and IMU
for robust visual SLAM in HDR and high speed scenarios
Vidal et.al., RPG@ETH, ROBOTICS AND AUTOMATION LETTERS 2017
56
50
Efficient SLAM in challenging illumination conditions
Utilize all available sensors for computational efficiently and robustness
SLAM(SE(3))
Event+Frame+Gyro
Y.Sekikawa, イベントカメラの研究動向と,
62. ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
63
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing
Pros: Efficient (Process only for changed pixels)
Cons: No established method like CNN
Event to Frame "(⋅)
fS
(e) = y
63. ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
64
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing
Pros: Efficient (Process only for changed pixels)
Cons: No established method like CNN
Event to Frame "(⋅)
fS
(e) = y
64. ニューラルネットワークによる処理, SSII2020
Event-based Vision meets Deep Learning
on Steering Prediction for Self-driving Cars
Maqueda et.al., Dept. of Informatics and Neuroinformatics@ETH, CVPR2018
65
126
Sophisticated CNN can be used
Convert sparse events to dense frame
66. ニューラルネットワークによる処理, SSII2020
Learning an event sequence embedding for dense event-based deep stereo
Tulyakov et.al., EPLF, ICCV2019
67
Event camera Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event to Frame "(⋅) &
Better than hand crafted conversion
Learn to convert (temporal kernel ) sparse events to dense frame
'&
72. ニューラルネットワークによる処理, SSII2020
High Speed and High Dynamic Range Video with an Event Camera
Rebecq et.al., UZH@ETH, CVPR 2019 PAMI 2019
73
Existing alg. can be readily applicable for challenging applications
Leaning to convert sparse event to intensity frame
74. ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
75
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing
Pros: Efficient (Process only for changed pixels)
Cons: No established method like CNN
Event to Frame "(⋅)
fS
(e) = y
75. ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
76
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing (Spike)
Pros: Efficient (Process only for changed pixels)
Cons: Requires Neuromorphic H/W (less neurons), Difficult to train
Event to Frame "(⋅)
Event-based processing (Continuous)
fS
(e) = y
76. ニューラルネットワークによる処理, SSII2020
SNN (Spikingk Neural Network):
3rd generation of neural network: Spiking Neural Network
77
Leaky and Integrate and Fire (LIF)
Charge → Fire(10ms)→ Refractory(100ms)→
"
#
Activation: Non-differentiable Spike (ANN: Relu, Sigmoid)
Asynchronous: MP※ > threshold → Fire (Similar to Event Camera)
MP%(')
Non-differentiable
*+
*,!
=
*+
*,"
⋅ …
*,#
*,!
Chain Rule
※MP: membrane potential
77. ニューラルネットワークによる処理, SSII2020
SNN Hardware
TrueNorth DYNAP Loihi Braindrop
Manufacture IBM aiCTX Intel Stanford
Type of neurons Digital LIF Analog LIF Digital LIF Analog
Neurons per chip 1,0000,000 4096x4 130,000x8 4096
Year 2014 2017 2018 2018
Programing Corelet, Eedn libcaer /cAER in C/C++ Nengo/Brain/PyNN Nengo
Training Outside chip On chip On chip Outside chip
For more detailed review see Young et.al. A Review of Spiking Neuromorphic Hardware Communication Systems, IEEE Access 2019
78
78. ニューラルネットワークによる処理, SSII2020
Categorization of training SNN
Supervised
Rewarded-STDP ANN (Back-propagation) to SNN
Unsupervised
STDP
Back-propagation
• Continuous relaxation (Approximate gradient / Inefficient)
• Temporal Coding (Exact / Dead neuron)
• Random back propagation
79
79. ニューラルネットワークによる処理, SSII2020
Synaptic Modifications in Cultured Hippocampal Neurons:
Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type
Bi et.al., Journal of Neuroscience, 1998
Images from arc-instruments
Weight update
Simple unsupervised training for non-differentiable spike. Neuroplausible
Hebb rule: “who fire together, wire together”
80
80. 81
Image Diehl et.al, l. Unsupervised learning of digit recognition using spike-timing-dependent plasticity
Simple mapping yealds 95% accuracy
81
81. ニューラルネットワークによる処理, SSII2020
A Low Power, Fully Event-Based Gesture Recognition System
Amir et.al., IBM Research+UZH-ETH, CVPR2017
Realized efficient gesture recognition using real SNN H/W (TrueNorth)
Convert trained ANN to SNN
82
83. ニューラルネットワークによる処理, SSII2020
Training Deep Spiking Neural Networks Using Backpropagation
Lee et.al, Institute of Neuroinformatics@ETH, Frontiers in Neuroscience 2016
84
Events by emulation saccade
"
# 0
9
Non-differentiable
0
9
Error
Est Ref
44
E2E training of SNN
Approximate non-differentiable spike using differentiable low-passed spike
85. ニューラルネットワークによる処理, SSII2020
Random synaptic feedback weights support error backpropagation
for deep learning
Lillicrap et.al., Univ.Oxford, Nature2016
Symmetric Backpropagation
(Chain Rule on ANN)
Random- Backpropagation
Direct feedback
For more detail see http://www.cs.toronto.edu/~tingwuwang/2546.pdf
Asymmetric Backpropagation
(Mammal neuron)
Neuroplausible training
DNN can be trained using Random matrix $ instead of symmetric weight
Direct error feedback
86
86. ニューラルネットワークによる処理, SSII2020
Event-Driven Random Back-Propagation:
Enabling Neuromorphic Deep Learning Machines
Neftci et.al., Univ.California+Intel, Frontiers in Neuroscience 2016
Weight update
Enables on-chip training & layer-by-layer parallelization
Apply RBP to SNN.
87
87. ニューラルネットワークによる処理, SSII2020
ML-Based (End-to-end neural-network for event processing)
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing (Spike)
Pros: Efficient (Process only for changed pixels)
Cons: Requires Neuromorphic H/W (less neurons), Difficult to train
Event to Frame "(⋅)
Event-based processing (Continuous)
88
fS
(e) = y
88. ニューラルネットワークによる処理, SSII2020
EventNet: Asynchronous recursive event processing
Sekikawa et.al, CVPR 2019
89
Event camera
Pros: Can be utilize exiting architecture !! (e.g., CNN)
Cons: Inefficient, Slow
Frame-based processing
!! ≔ x, y, p, t "
fD
(g(e)) = y
Event-based processing (Continuous)
Pros: Efficient (Process only for changed pixels)
Cons: Requires Neuromorphic H/W (less neurons), Difficult to train
Event to Frame "(⋅)
Event-based processing (Spike)
133
Real-time event-wise inference on CPU
Recursive formulation & LUT to drastically reduce computational complexity
fS
(e) = y