Slide di Danilo Pau presentate all'evento "Reti neurali e media"

1. Danilo Pau, Claudio Marchisio
System Research and Applications
Agrate Brianza
Components for
Neural Networks
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

2. This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/
Turing Test
• “I propose to consider the question: can machines think ?”
• Because thinking is difficult to define, "Are there imaginable digital
computers which would do well in the imitation game?"
• “The imitation game could then be played with the machine in
question … mimicking digital computer and the interrogator would
be unable to distinguish them”
2
Turing Test
Rosenblatt’s
Perceptron Winter of AI
Universal
Aproximation
Theorem
The rise of AI
Google AI
Assistant
Robots
powered by
Artificial
Intelligence ?
1950 1957 1989 1990 2006 2018 20501936
Turing
Machine

3. This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/
Turing Test
• “I propose to consider the question: can machines think ?”
• Because thinking is difficult to define, "Are there imaginable digital
computers which would do well in the imitation game?"
• “The imitation game could then be played with the machine in
question … mimicking digital computer and the interrogator would
be unable to distinguish them”
2
Turing Test
Rosenblatt’s
Perceptron Winter of AI
Universal
Aproximation
Theorem
The rise of AI
Google AI
Assistant
Robots
powered by
Artificial
Intelligence ?
1950 1957 1989 1990 2006 2018 20501936
Turing
Machine

4. This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/
Turing Test
• “I propose to consider the question: can machines think ?”
• Because thinking is difficult to define, "Are there imaginable digital
computers which would do well in the imitation game?"
• “The imitation game could then be played with the machine in
question … mimicking digital computer and the interrogator would
be unable to distinguish them”
2
Turing Test
Rosenblatt’s
Perceptron Winter of AI
Universal
Aproximation
Theorem
The rise of AI
Google AI
Assistant
Robots
powered by
Artificial
Intelligence ?
1950 1957 1989 1990 2006 2018 20501936
Turing
Machine

5. The effects on the Applications
Centralized
Intelligence
Systems
Performance reduction,
cascade effects
on Applications
Faults, Errors,
Uncertainty,
Malfunctioning,
Intrusions
Changes:
Nonstationary,
seasonality, periodicity
5
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

6. • Definition of Cyber Physical Systems
6
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

7. Cyber-Physical SystemsPhysical
Domain
Cyber
Domain
Object
Domain
Information
Transmission
Actuation
and Control
7
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

8. • Definition of Cyber Physical Systems
• Current Limitations
8
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

9. Voice Recognition: Challenging the Could 9
Average Person
• 16000 utterances */day
@163 words/minute +
;
• ≈ 98 minutes speech/day.
Natural
Language
Processing
Audio coding@
128Kbps
≈94 MB/day
≈94 TB/day
≈1 PB/day
≈9 PB/day
1 Million People
10 Million
People
100 Million
People
Cloud based Voice
Recognition means
huge data bandwidth
and computation
capabilities
Source:
1. Are Women Really More Talkative Than Men? ResearchGate
2. What is the Average Speaking Rate? SixMinutes
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

10. 10
Courtesy Dday.it
Courtesy Tecnoandroid.it
….
Source: Wired, Aug 2017
Voice Recognition: Challenging the Could
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

11. 11
https://www.theguardian.com/environment/2017/dec/11/tsunami-of-data-could-consume-fifth-global-electricity-by-2025
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

12. Object domain under attack
DDoS, screwing Dyn, Oct 2016
• A group called “New World Hackers“ has
claimed responsibility for the attack.
• It was based on Mirai code
• Attackers were refrigerators, thermostats,
and toasters.
• From 09:30 to 18:00 ET, Dyn’s servers
were attacked in three DDoS waves.
• Cyberattack, affected Twitter, Amazon,
Reddit, Netflix, and more since they used
Dyn DNS provider.
12
https://readwrite.com/2016/10/22/the-internet-of-things-was-used-in-fridays-ddos-attack-pl4/
New World Hackers, Mirai code
Refrigerators,
Thermostats,
toasters
Dyn Servers
Twitter, Amazon,
Reddit, Netflix
… pinging …. … pinging ….
… denial ….
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

13. 13
https://www.bloomberg.com/news/articles/2019-04-10/is-anyone-listening-to-you-on-alexa-a-global-team-reviews-audio
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

14. • Definition of Cyber Physical Systems
• Current Limitations
• Opportunities
14
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

15. Computers vs Embedded µControllers 15
Olivetti M24 23
• Intel 8086
• 8 MHz
• 128 KB RAM
• 16 KB ROM
• 1.84 W
• 360 $
• STM32 MCU L4
• 80 MHz
• 128 KB RAM
• 1 MB Flash
• < 20mW
• < 4 €
STM32 L4+
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

16. STM 6-axis IMU evolution
SiP 3D
Digital Accelerometer and Gyroscope
Embedded Sensors
Credit http://semieurope.omnibooksonline.com/2014/semicon_europa/International_MEMS_Forum/13_Romain_Fraux_System_Plus_Consulting.pdf
16
• First accelerometer (1923)
• Credits: McCollum and Peters
• Resistance Bridge type, with
carbon rings in a tension-
compression Wheatstone
• Dimensions: ~ 28 c63
• Price: $420 ($6,275 today)
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

17. Digital Camera Kodak’s Steven Sasson 73 17
• 50ms to capture an image
• 10K pixels, black and white
• 23s to record it on tape
• 3.6 Kg
• Electronic still camera,
• US patent 4131919 A
OpenMV Cam
STM32H743VI ARM Cortex M7
400 MHz, 1MB RAM, 2 MB Flash
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

18. Local Processing
• STM32F446 32-bit ARM Cortex-M4F MCU
• 180MHz, 2MB FLASH
• STM32F446 product line provides from 256-
Kbyte to 512-Kbyte Flash, 128-Kbyte SRAM
• DSP Instructions
Inertial and Environmental
• LSM6DSM iNEMO 6DoF Gyro + Acc
• LSM303AGR e-Compass
• 4 X MP34DT06J Microphones
• LPS22HB MEMS Pressure sensor
• HTS221 Humidity & Temperature Sensor
Sensing
Processing
Wireless
• BlueNRG-MS, Bluetooth Low Energy
Network Processor supporting Bluetooth 4.2
core specification
Connectivity
Intelligent Acoustic Sensing Unit
STEVAL-BCNKT01V1
18
https://www.st.com/en/evaluation-tools/steval-bcnkt01v1.html
STEVAL-BCNST01V1
CoinStation
130mAh LiPo Battery
(UN38.3 Certified)
ST-Link SWD
Programming Cable
STEVAL-BCNCS01V1
Core System
Plastic Case
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

19. 100s Billions of Sensors 19
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/
https://www.slideshare.net/KNIMESlides/advanced-analytics-for-the-internet-of-things-restocking-rental-bike-stations?from_action=save

20. Physical
Domain
Unit
SensorsSensors
Comm
Unit
SensorsSensors
Comm
Unit
SensorsSensors
Comm
Gate
way
Unit
SensorsSensors
Comm
Gate
way
Gate
way
Server
Cloud
Application
Artificial IntelligenceScalability
Responsiveness
Intelligent units
Designing Intelligent and distributed CPS 20
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

21. • Definition of Cyber Physical Systems
• Current Limitations
• Opportunities
• STM32CubeAI
21
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

22. Can STM32 run Artificial Neural Network ? 22
88 9:6;<=>?@ABCDEF
GHIBCDEF/F
K
LM6;<=L ?N M O?NP:O
QRSTRUV RWXYZ[T/R
1,034,722BCDEF
13,6GBCDEF/F
K
16,384FbcECdF
16,000FbcECdF/R
efGcgh i f9@?jM@?:NLkSlmUVn
e0Gcgh i o=?pq@LkSlmUVn
r*stu i
K vwtu
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

23. Key Steps Behind AI Applications 23
2
Collect, clean, label Data
5
Run Field trials
1
Define AI application
4
Convert ANNs into
optimized code
for STM32 MCU
Neural Network (ANN) Model Creation Deployment Mode
3
Build ANN Topologies
Train ANN Models
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

24. Sound Recognition
Requires Distributed Sensing Nodes
24
Doppler effect of cars
approaching/leaving
1
1
2
2
Installed at the side of streets
1
Define AI application
2
Collect, clean, label Data
Managed
by
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

25. Deep Learning Frameworks
Popularity
25
https://www.kdnuggets.com/2018/09/deep-learning-framework-power-scores-2018.html
3
Build ANN Topologies
Train ANN Models
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

26. 26Keras Model Example
3
Build ANN Topologies
Train ANN Models
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

27. Key Steps Behind AI Applications 27
2
Collect, clean, label Data
1
Define AI application
Neural Network (ANN) Model Creation Deployment Mode
4
Convert ANNs into
optimized code
for STM32 MCU
3
Build ANN Topologies
Train ANN Models
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

28. Neural Networks
Available Now for STM32
28
Embedded Solution
Optimized Neural Network
Code generated for STM32
Deep
Learning
SW Solution
Choose your IDE
Compiler and Debugger
Framework Independent
Sensors and OS Agnostic
Interoperability
Pre-trained Neural Network models
Deep Learning framework dependent
TodayTomorrow
AI Hub
Multiple Neural
Networks
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

29. Making AI Accessible Now 29
World 1st
Cortex-M MCU
World 1st
Cortex-M
Ultra-low-power
1st High Perf.
120 MHz, 90nm
1st High Perf.
Cortex-M4
168 MHz
Entry Cost
STM32F0
Cortex-M0
1st Mixed Signal
DSP + Analog
STM32F3 Cortex-M4
Entry Cost
Ultra-low-power
World 1st
Cortex-M7
Leadership
Ultra-low-power
Cortex-M4
273 ULPBench™
#1 ULP
#1
Performance
2020 CoreMark
Ultra-low-power
Excellence
Dual-core,
multi-protocol
and open radio
Introduction of M33
Excellence in ULP
with more security
Mainstream
Cortex-M0+ MCUs
Efficiency at its best!
20182007 2009 2010 2011 2012 2013 2014 2015 2016 20192017
First STM32 MPU
Dual Cortex-A7 + Cortex-M4
STM32 meets Linux
Compatible with
STM32Cube.AI ecosystem
Compatible with Partner
Machine Learning ecosystems
More than 60,000 customers Over 4 Billion STM32 shipped since 2008
Leader in Arm® Cortex®-M 32-bit General Purpose MCU
• Based on single or dual ARM Cortex M core
• Broadest global portfolio
• > 1000 products available
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

30. STM32Cube.AI: select STM32
- Loading a pre-trained
NN model
- Possibility to compress
the weights to gain Flash
space
- Analyze the NN to
provide RAM/Flash
usage
30
After analysis:
list of compatible
STM32
Minimum RAM /
Flash for the
selected NN
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

31. - Loading several pre-
trained NN model
- Compress weights
- Analyze the NNs to
provide RAM/Flash
usage
- Validate on desktop or
on target
31
Validate each NN
layer
Configure
STM32 pin
(standard
CubeMX feature)
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

32. STM32Cube.AI: project generation 32
Generate project
for your favorite
IDE
Run on target
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

33. STM32Cube.AI Architecture
This optimized STM32 neural network model can be included into the user project
(using any user IDE) and can be compiled and ported onto the final device for field trials
Neural
Network
Exporter
DL Framework
Independent
Neural Network
Representation
NN Layers
Library
for STM32
Neural
Networks
API’s
Code
Generator
33
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

34. X-Cube-AI Generated files 34
Customer application file
Artificial Neural Network
topology and API
Artificial Neural Network data
file (weights/bias parameters)
Once the code is generated we
have:
Validation application
Always generated/needed
Needed only if NN validation
on target is required
Needed in case we start from
a blank project.
Neural network
computing kernel
provided as a
static library
precompiled for
the given STM32
P/N selected
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

35. We brought Demo here
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

36. ST and Bluewind @ Embedded World 2019 36
https://www.youtube.com/watch?v=94tT9nlv5B8&list=PLnMKNibPkDnFpi3vFzkbH32yblwN3XVwv&index=6
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

37. Speech to Intent on MCU 37https://www.youtube.com/watch?v=WadKhfLyqTQ
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

38. Audio Scene Classification (ASC)
Audio Example in FP-AI-SENSING1 Package
38
Embedded audio Data stored on the device
SD card for future learning
Inference result
displayed on mobile app
Inferences running
on the microcontroller
Embedded audio
pre-processing
NN & example
dataset provided
Indoor, Outdoor, In vehicle
3 classes
Labelling controlled
by smartphone application
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

39. STM32Cube.AI Roadmap 39
2019 Apr Jun
• Floating Point
Support
• Quantization
• TensorFlow for MCU
• Command line interface
• UI Improvements
• Additional layers
2020
• Introduction
• Additional layers
• Debug improvements
+microcontrollers
+
IoTNode
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/

40. www.st.com/STM32CubeAI
This work is licensed under the Creative Commons Attribution- NonCommercial-NoDerivs 3.0 Unported License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/