Successfully reported this slideshow.
Your SlideShare is downloading. ×

Intelligent Edge - Getting started with TinyML for industrial applications

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Intelligent Edge - Getting started with TinyML for industrial applications

  1. 1. Copyright © 2019 EdgeImpulse Inc. Getting started with TinyML for industrial applications
  2. 2. 2 Jan Jongboom jan@edgeimpulse.com Hello! Raphael Apfeldorfer raphael.apfeldorfer@st.com
  3. 3. 3 Typical industrial sensor in 2020 Vibration sensor (up to 1,000 times per second) Temperature sensor Water & explosion proof Can send data >10km using 25 mW power Processor capable of running >20 million instructions per second
  4. 4. 4 But... what does it actually do? Once an hour: • Average motion (RMS) • Peak motion • Current temperature
  5. 5. 5 99% of sensor data is discarded due to cost, bandwidth or power constraints. https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/ The%20Internet%20of%20Things%20The%20value%20of%20digitizing%20the%20physical%20world/The- Internet-of-things-Mapping-the-value-beyond-the-hype.ashx
  6. 6. 6 Lots of interesting events get lost Peak
  7. 7. 7 Single numbers can be misleading updown circle avg. RMS 3.3650 3.3515
  8. 8. 8 On-device intelligence is the only solution Vibra&on pa+ern heard that lead to fault state in a weekTemperature varies in a way that I've never seen before Machine oscillates different than all other machines in the factory
  9. 9. 9 On-device intelligence is the only solution Temperature varies in a way that I've never seen before (0x1)
  10. 10. Machine learning is great at finding patterns in messy data (anything you can't reason about in Excel)
  11. 11. 11 Machine learning?
  12. 12. 12 TinyML Inspired by "OK Google" Focus on inferencing, not training Machine learning model is just a mathematical function with lots of parameters Accuracy vs. speed, reducing parameters, hardware- optimized paths Targeting battery-powered microcontrollers Pete Warden Neil Tan
  13. 13. Predictive maintenance https://www.flickr.com/photos/uwnews/26143149238
  14. 14. Audible monitoring
  15. 15. Lone worker monitoring Check new Person Presence Detection example in FP-AI-VISION1 ! https://www.st.com/en/embedded-software/fp-ai-vision1.html
  16. 16. Endpoint Machine level Decisions Distributed AI from Edge to Cloud Edge Actions Data Gateways Room or building-level Decisions HomeIndustrial Cloud Cities or factory-level decisions • Fleet monitoring • Macro Decisions (AI) • Macro Actuation Sensor • Sensing Smart sensor • Sensing • Preprocessing • Micro classification Data MCU • Real time • Data privacy • Micro Decisions (AI) • Micro Actuation Actions Event detection Event detection Sensor Data (Option) Event MCU / MPU • Data or Event integration • Data Privacy • Meso Decisions (AI) • Meso Actuation 16
  17. 17. GatewaysEndpoint Edge Actions Data Room or building-level Decisions Cloud Cities or factory-level decisions • Fleet monitoring • Macro Decisions (AI) • Macro ActuationActions Event detection Connectivity Security Sensing Accelerometer Gyrometer Magnetometer Vibration PressureAcoustic Temperature ML Core Extensive MCU portfolio Processing & Actuating Machine level Decisions STM32MP1 STM32H7 STM32WB Connectivity Security Processing & Actuating Event detection Sensor Data (Option) Event Distributed AI from Edge to Cloud 17
  18. 18. IIS3DWB Ultra Wide Bandwidth Accelerometer ISM330DLC ISM330DHCX Wide Bandwidth Accelerometer + Gyroscope IIS2DH Wide Bandwidth, Ultra-low-power Accelerometer IIS2MDC Low-Noise, Low Power Magnetometer 18 ST portfolio for condition monitoring Vibration MP23ABS1TR Analog Differential Microphone IMP34DT05-A Digital Top Port Microphone Acoustic LPS22HH High Accuracy – Compact Size Absolute Pressure Sensor LPS27HHW LPS33HW Water Resistant Absolute Pressure Sensor STTS22H Digital Temperature Sensor STLM20 Analog Temperature Sensor Environmental High Perf MCUs Ultra-low Power MCUs Wireless MCUs Mainstream MCUs MPU MEMS & Sensors
  19. 19. Wireless Industrial node to capture data at industrial grade 19www.st.com/stwin Industrial-grade sensors • Industrial scale 9-DoF motion sensors including accelerometer, gyrometer and an ultra wide-band vibrometer with ultra low noise • Very high frequency audio and ultrasound microphone • High precision temperature and environmental monitoring • Micro SD card for standalone data logging • BLE connectivity and WiFi expansion board • USART STWIN Process & analyze new data using trained NNCapture data 9
  20. 20. 20 v From 0 to model https://www.flickr.com/photos/120586634@N05/14491303478
  21. 21. 21 1. Everything starts with raw data Get data at the highest resolu/on possible - e.g. using serial or directly over WiFi
  22. 22. 22 2. Extracting meaningful features Very dependent on your use case Raw data can be notoriously hard to deal with (3s. accelerometer data = 900 data points, 1s. audio data = 16,000 data points) Use proven DSP algorithms -1.1200, 0.5300, 10.6300, -0.5200, 1.1600, 13.1600, -0.1600, 1.4200, 13.5900, -0.2400, 1.3200, 10.8500, -0.3700, 0.5100, 7.7800, -0.1900, 0.0500, 8.8400, -0.1900, 0.0500, 8.8400, 0.2400, 0.7300, 11.1900, 0.5200, 1.6500, 12.5200, 0.6400, 1.5000, 10.8200, 0.2900, 0.3300, 7.4000, -0.3100, -0.5600, 8.5900, -0.8800, 0.6600, 11.9200, -0.8800, 0.6600, 11.9200, -0.4500, 1.3900, 11.9100, -0.0800, 1.6000, 12.7800, -0.1100, 0.3600, 9.1700, -0.0200, 0.0700, 9.8200, 0.0600, 0.9600, 12.0000, 0.2800, 1.7700, 13.2000, 0.2800, 1.7700, 13.2000, 0.2300, 1.5100, 12.6000, 0.2700, 1.0800, 11.4300, 0.0900, 0.9300, 10.9400, 0.1700, 1.2100, 11.0400, 0.4100, 1.8500, 11.5000, 0.3900, 1.9600, 11.4300, 0.3900, 1.9600, 11.4300, 0.2400, 1.4400, 10.7200, 0.0300, 1.1900, 10.3600, -0.0300, 1.3000, 10.6100, 0.4800, 1.7900, 11.7200, 1.0400, 2.6300, 13.3300, 1.0400, 2.6300, 13.3300, 1.0600, 2.3700, 13.5800, 0.3600, 1.9600, 13.3800, -0.1000, 2.1200, 13.9600, -0.3600, 2.0200, 14.5300, 0.0000, 2.1500, 14.6900, 0.0600, 2.1400, 14.3700, 0.0600, 2.1400, 14.3700, -0.3000, 1.8800, 13.7900, 0.0500, 1.7000, 13.5900, 0.1300, 1.6700, 13.1500, -0.0100, 1.7700, 12.9000, 0.4000, 1.8900, 12.2300, 0.5300, 2.3300, 12.2600, 0.5300, 2.3300, 12.2600, 0.0400, 1.9500, 11.8100, -0.2300, 1.9600, 11.2400, -0.0600, 2.1100, 10.2200, -0.1100, 2.4100, 9.7800, -0.3500, 2.7100, 9.7500, -0.7800, 3.1000, 10.1100, -0.7800, 3.1000, 10.1100, -1.0700, 3.1100, 9.8000, -1.2100, 2.9400, 9.1900, -1.1500, 3.2100, 8.6400, -0.7300, 3.6500, 8.4600, -0.5000, 3.9500, 8.6300, -0.4300, 3.9100, 8.7400, -0.4300, 3.9100, 8.7400, -0.6400, 3.7800, 8.8700, -1.2000, 3.9200, 8.9300, -1.0800, 4.4400, 8.8100, -0.7800, 4.1900, 8.1200, -0.4400, 4.1000, 7.6400, -0.5400, 4.2000, 7.5600, -0.5400, 4.2000, 7.5600, -1.0700, 4.2600, 7.2700, -1.3000, 4.5100, 7.2300, -1.2600, 4.4600, 6.6900, -1.2800, 4.4100, 6.6000, -1.7000, 4.6800, 7.0800, -2.3400, 5.1100, 7.5900, -2.3400, 5.1100, 7.5900, -2.8300, 4.8700, 6.8700, -2.9700, 4.7600, 6.2700, -3.2500, 4.6000, 6.1500, -3.4900, 4.5900, 6.2600, -3.3000, 4.9200, 6.3400, -2.7000, 4.9300, 5.8600, -2.7000, 4.9300, 5.8600, -2.9000, 4.5100, 4.9900, -3.5200, 4.3200, 4.9900, -4.1400, 4.2100, 5.7800, -3.7600, 4.1600, 5.7700, -3.0200, 4.2200, 5.4900, -3.0000, 3.8900, 3.9100, -3.0000, 3.8900, 3.9100, -3.3500, 3.5800, 3.4400, -3.1100, 3.2000, 3.6000, -3.0900, 3.6000, 5.9400, -3.0800, 3.0900, 5.1800, -2.8000, 2.9400, 4.5600, -2.4000, 2.4900, 3.9200, -2.4000, 2.4900, 3.9200, -1.8500, 2.6300, 4.4900, -1.4300, 3.9900, 7.3400, -1.4900, 3.4900, 6.1600, -1.5300, 3.2100, 5.4500, -0.9900, 2.8600, 6.2400, -0.7900, 3.2200, 8.4700, -0.7900, 3.2200, 8.4700, -0.9300, 3.5700, 9.0300, -1.6600, 2.9600, 7.0700, -1.7600, 2.1000, 6.9000, -1.4600, 2.1100, 9.0100, -1.3000, 2.5400, 10.3000, -1.3000, 2.5400, 10.3000, -1.4500, 2.6500, 9.7800, -1.5300, 1.8900, 8.4100, -1.1400, 1.3000, 9.4400, -0.7500, 1.6100, 10.3600, -0.9400, 1.5300, 9.7800, -1.0700, 1.0100, 9.2700, -1.0700, 1.0100, 9.2700, -0.8600, 1.0200, 10.1100, 0.3900, 1.6800, 11.3200, 1.2900, 1.8500, 11.4700, 1.0700, 1.3200, 11.2500, -0.2100, 1.5800, 12.4300, -1.8100, 1.3500, 12.3300, -1.8100, 1.3500, 12.3300, -2.1800, 1.0400, 11.1600, -1.5400, 0.3000, 10.3100, -0.4700, 0.2700, 11.0900, 0.7900, 1.4100, 12.9800, 1.1600, 1.7100, 12.4700, 0.5800, 0.8700, 9.6300, 0.5800, 0.8700, 9.6300, 0.1300, 0.3100, 9.5600, 0.2800, 0.3600, 10.5600, 0.7400, 0.8500, 11.4200, 0.9300, 1.0200, 11.4300, 0.6700, 0.5600, 10.5300, 0.8500, 0.3500, 9.4100, 0.8500, 0.3500, 9.4100, 1.6600, 1.2800, 10.9200, 2.0500, 0.9900, 9.7000, 2.1300, 0.8800, 10.1900, 2.0500, 0.9100, 11.3300, 1.7700, 1.4100, 12.2700, 1.4800, 1.7600, 12.1000, 1.4800, 1.7600, 12.1000, 0.9400, 1.1300, 10.8500, 0.2000, 0.8000, 10.1200, 0.2600, 1.1600, 10.5800, 0.5100, 1.6500, 10.7800, 0.4600, 1.2000, 9.9900, 0.9100, 0.8400, 9.6400, 0.9100, 0.8400, 9.6400, 1.4500, 0.7400, 10.2500, 2.0200, 1.3000, 11.4500, 1.8100, 1.8700, 12.1300, 1.0500, 1.5300, 12.0200, 0.6200, 0.6700, 11.3100, 0.7100, 0.8500, 12.0000, 0.7100, 0.8500, 12.0000, 0.6400, 1.2200, 13.1400, 1.1300, 2.0400, 14.6200, 0.8300, 2.0200, 15.5100, -0.1400, 1.4800, 15.6500, -0.6300, 1.5900, 16.0500, -1.3100, 1.7100, 16.3900, -1.3100, 1.7100, 16.3900, -1.7300, 1.5800, 16.6300, -1.1500, 1.4400, 16.0300, -0.5300, 1.1700, 15.1000, -0.1800, 0.9900, 14.4600, -0.3300, 1.0100, 13.5100, -0.3300, 1.0100, 13.5100, -0.4400, 0.9100, 12.6700, 0.0400, 1.2300, 12.5400, 0.6900, 2.0500, 13.1600, 0.3100, 1.7700, 12.8600, 0.0300, 1.3800, 11.1100, -0.4400, 1.2200, 9.4900, -0.4400, 1.2200, 9.4900, 0.1100, 1.1400, 7.3100, 0.8500, 2.2500, 8.4600, 0.8600, 3.3700, 11.2200, -0.1100, 2.2800, 8.4400, -1.3800, 1.5300, 7.1700, -1.0600, 1.5400, 6.9500, -1.0600, 1.5400, 6.9500, -0.5200, 2.8300, 8.7100, -0.2100, 2.3500, 8.1800, -0.3400, 2.7000, 8.9200, -0.3000, 2.3100, 8.7500, -0.4800, 1.4700, 7.8700, -0.3600, 0.9400, 6.9700, -0.3600, 0.9400, 6.9700, -0.2300, 1.4700, 7.6100, -0.3300, 2.2300, 8.5000, 0.3000, 1.9200, 7.8600, -0.2300, 1.5700, 6.8700, -1.4900, 1.5600, 6.3700, -2.8200, 1.6200, 7.2000, -2.8200, 1.6200, 7.2000, -3.1600, 1.8800, 7.1500, -2.7600, 2.2900, 6.8500, -2.6000, 2.2200, 6.2600, -2.9000, 1.9900, 5.8900, -3.3800, 2.2200, 6.2600, -3.9000, 2.1700, 6.0300, -3.9000, 2.1700, 6.0300, -3.8600, 2.3800, 5.6600, -3.5300, 2.5200, 5.6700, -3.2400, 2.3700, 5.8200, -3.2800, 2.1800, 5.5200, -3.1500, 2.1800, 5.6500, -3.0900, 2.0700, 5.1600, -3.0900, 2.0700, 5.1600, -2.4300, 2.1000, 5.3800, -2.0200, 2.3600, 6.0800, -2.0000, 2.5200, 6.4500, -2.2400, 2.4500, 6.0000, -2.0500, 1.8400, 4.6500, -1.3800, 1.3000, 4.6400, -1.3800, 1.3000, 4.6400, -1.2800, 1.8600, 6.9400, -1.3000, 2.5600, 9.0300, -1.5400, 2.7600, 8.5000, -1.7700, 1.6400, 6.1400, -1.6800, 1.4200, 7.5900, -1.3200, 2.0800, 9.8300, -1.3200, 2.0800, 9.8300, -0.8200, 2.1600, 10.3900, -0.7800, 1.7300, 9.8300, -1.1300, 1.3400, 9.7100, -1.3600, 1.6800, 10.2400, -1.5200, 1.6000, 9.3200, -1.8700, 1.4900, 9.1900, -1.8700, 1.4900, 9.1900, -1.9300, 1.0600, 9.9500, -1.3100, 0.8100, 10.6900, 0.0200, 2.0400, 11.0600, 0.2700, 2.5800, 9.3900, -0.0500, 2.2800, 7.3200, -0.3000, 0.4400, 7.6300, -0.3000, 0.4400, 7.6300, -1.4600, 1.0800, 12.3700, -1.9600, 1.7500, 15.3800, -0.7100, 2.1500, 14.0700, 0.7400, 1.7800, 10.4700, 0.6800, 0.8900, 9.9500, 0.0400, 1.5200, 12.0800, 0.0400, 1.5200, 12.0800, -0.4900, 1.7900, 12.7500
  23. 23. 23 Example of a signal processing pipeline 32,000 => 240
  24. 24. 24 Before and after feature extraction
  25. 25. Classification What's happening right now? Anomaly detection Is this behavior out of the ordinary? Forecasting What will happen in the future? 25 3. Letting the computers figure it out
  26. 26. 26 Picking the right algorithm Classification Neural network Anomaly detection K-means clustering Forecasting Regression
  27. 27. 27 4. Deploying Signal processing, neural network and anomaly detec&on
  28. 28. Seamless integration to STM32Cube ecosystem STM32CubeMX, GUI Builders Configure & generate code 28 Configure clocks, peripherals and generate low-level code Add application code and compile for any STM32 target with the IDE of your choice Deploy, debug and monitor Generate model, optimize with STM32Cube.AI and export as CMSIS-PACK And commercial IDEs CMSIS-PACK Import in CubeMX and benefit from STM32 ecosystem CMSIS-PACK running in Edge Impulse’s Cloud
  29. 29. STM32CubeL1 Hardware Abstraction Layer CMSI S STM32CubeF0 Hardware Abstraction Layer CMSI S STM32CubeF2 Hardware Abstraction Layer CMSI S STM32CubeF4 Hardware Abstraction Layer CMSI S STM32CubeF1 Hardware Abstraction Layer CMSI S STM32Cube Hardware abstraction layer CMSIS STM32Cube Middleware User code FirmwareHardware Making it easy to start development, develop Proof of Concept STM32 Ecosystem foundations Configuration Development Programming Monitoring Tools macOS® And more
  30. 30. STM32CubeMX expansion STM32Cube.AI tool STM32Cube.AI offers interoperability with state-of-the-art Deep Learning design frameworks Any framework that can export models in ONNX open format can be imported, including quantized models Automatic and fast generation of an STM32-optimized library Train NN Model Convert NN into optimized code for MCU Process & analyze new data using trained NN 30 On-device validation enable fast comparison of model accuracy vs different memory / quantization options And more
  31. 31. STM32Cube.AI main features 31 Dimension Optimize Fine Tune And quickly iterate thanks to on-target validation STM32Cube.AI is available both as graphical and command line interface
  32. 32. Making AI Accessible Now High Perf MCUs Ultra-low Power MCUs Wireless MCUs Mainstream MCUs STM32F0 106 CoreMark 48 MHz STM32G0 142 CoreMark 64 MHz STM32F1 177 CoreMark 72 MHz STM32F3 245 CoreMark 72 MHz STM32F2 398 CoreMark 120 MHz STM32F4 608 CoreMark 180 MHz STM32L0 75 CoreMark 32 MHz STM32L5 424 CoreMark 110 MHz STM32L1 93 CoreMark 32 MHz STM32L4 273 CoreMark 80 MHz STM32WL 161 CoreMark 48 MHz STM32L4+ 409 CoreMark 120 MHz STM32G4 550 CoreMark 170 MHz STM32MP1 4158 CoreMark 650 MHz Cortex -A7 209 MHz Cortex -M4 MPU Arm® Cortex® core -M0 -M3 -M33 -M4 -M7 dual -A7& -M4-M0+ STM32F7 1082 CoreMark 216 MHz STM32H7 3224 CoreMark 240 MHz Cortex -M4 480 MHz Cortex -M7 STM32WB 216 CoreMark 64 MHz Compatible with Deep Learning STM32Cube.AI ecosystem Compatible with Machine Learning Partner ecosystems Leader in Arm® Cortex®-M 32-bit General Purpose MCU More than 40,000 customers Over 4 Billion STM32 shipped since 2007 32
  33. 33. Making AI Accessible Now High Perf MCUs Ultra-low Power MCUs Wireless MCUs Mainstream MCUs STM32F0 106 CoreMark 48 MHz STM32G0 142 CoreMark 64 MHz STM32F1 177 CoreMark 72 MHz STM32F3 245 CoreMark 72 MHz STM32F2 398 CoreMark 120 MHz STM32F4 608 CoreMark 180 MHz STM32L0 75 CoreMark 32 MHz STM32L5 424 CoreMark 110 MHz STM32L1 93 CoreMark 32 MHz STM32L4 273 CoreMark 80 MHz STM32WL 161 CoreMark 48 MHz STM32L4+ 409 CoreMark 120 MHz STM32G4 550 CoreMark 170 MHz STM32MP1 4158 CoreMark 650 MHz Cortex -A7 209 MHz Cortex -M4 MPU Arm® Cortex® core -M0 -M3 -M33 -M4 -M7 dual -A7& -M4-M0+ STM32F7 1082 CoreMark 216 MHz STM32H7 3224 CoreMark 240 MHz Cortex -M4 480 MHz Cortex -M7 STM32WB 216 CoreMark 64 MHz Compatible with Deep Learning STM32Cube.AI ecosystem Compatible with Machine Learning Partner ecosystems Leader in Arm® Cortex®-M 32-bit General Purpose MCU More than 40,000 customers Over 4 Billion STM32 shipped since 2007 33 Vibration Motion Environmental Bio signals Acoustic Condition monitoring Industrial gateways Thermal and RGB cameras
  34. 34. Getting started 🚀
  35. 35. 35 Get some hardware ST IoT Discovery Kit 80MHz, 128K RAM, $50 Any smartphone Any other device With our SDKs
  36. 36. 36 Edge Impulse - TinyML as a service Embedded or edge compute deployment options Test Edge Device Impulse Dataset Acquire valuable training data securely Test impulse with real-time device data flows Enrich data and generate ML process Real sensors in real time Open source SDK Free for developers: edgeimpulse.com
  37. 37. End-to-end tutorials on vibra/on, audio, and vision: docs.edgeimpulse.com
  38. 38. Demo 🚀
  39. 39. https://www.innovationworldcup.com/categories/stwin-challenge/ Join the STWIN CHALLENGE and compete to win overall prizes worth over $500,000
  40. 40. Recap The ML hype is real ML + sensors = perfect fit Start using the remaining 99% of sensor data edgeimpulse.com / st.com
  41. 41. Slides: http://janjongboom.com (will be emailed afterwards)

×