Download as PDF, PPTX
![What happens on a CPU
int main() {
int a[5], b[5], output[5];
for(int i=0;i<5;i++){
output[i]=a[i]+b[i];
}
}
main:
str fp, [sp, #-4]!
add fp, sp, #0
sub sp, sp, #68
mov r3, #0
str r3, [fp, #-8]
.L3:
ldr r3, [fp, #-8]
cmp r3, #4
bgt .L2
ldr r3, [fp, #-8]
lsl r3, r3, #2
sub r2, fp, #4
add r3, r2, r3
ldr r2, [r3, #-24]
ldr r3, [fp, #-8]
lsl r3, r3, #2
sub r1, fp, #4
add r3, r1, r3
ldr r3, [r3, #-44]
add r2, r2, r3
ldr r3, [fp, #-8]
lsl r3, r3, #2
sub r1, fp, #4
add r3, r1, r3
str r2, [r3, #-64]
ldr r3, [fp, #-8]
add r3, r3, #1
str r3, [fp, #-8]
b .L3
.L2:
mov r3, #0
mov r0, r3
add sp, fp, #0
ldr fp, [sp], #4
bx lr
• Variable latency](https://image.slidesharecdn.com/implementingai-runningaiattheedge-200616142707/85/Implementing-AI-Running-AI-at-the-Edge-44-320.jpg)
![What happens on a FPGA
int main() {
int a[5], b[5], output[5];
for(int i=0;i<5;i++){
output[i]=a[i]+b[i];
}
} + +
a[0] b[0] a[1] b[1] a[2] b[2] a[3] b[3] a[4] b[4]
+ + +
output[0] output[1] output[2] output[3] output[4]
• Control over the design
• Control over the latency
• Control over the clock (power consumption)](https://image.slidesharecdn.com/implementingai-runningaiattheedge-200616142707/85/Implementing-AI-Running-AI-at-the-Edge-45-320.jpg)
Uploaded byKTN
Implementing AI: Running AI at the Edge
- Dynamic neural networks (DNNs) can adapt to varying resource availability on edge devices through techniques like incremental training and group convolution pruning. This allows meeting requirements for timing, power/energy, and accuracy. - Experiments on two embedded platforms showed that dynamic DNNs combined with DVFS and task mapping can reduce energy consumption while maintaining classification accuracy compared to static DNNs. - Runtime power management is needed to coordinate heterogeneous processors, respond to environmental factors, balance power consumption and battery life, and meet requirements for concurrently executing tasks and applications under varying conditions on edge devices.
More Related Content
Similar to Implementing AI: Running AI at the Edge
Implementing AI: Running AI at the Edge
- 1.
- 2. • Innovate UKdrives productivity and economic growth by supporting businesses to develop and realise the potential of new ideas, including those from the UK’s world-class research base. • Knowledge Transfer Network (KTN) is Innovate UK’s Network partner • We help business to grow the economy and improve people’s lives by capturing maximum value from innovative ideas, scientific research and creativity. • KTN combines in-depth expertise in all sectors with the ability to cross boundaries. • Nigel Rix, Head of Enabling Technology: nigel.rix@ktn-uk.org
- 3. eFutures aims tostrengthen and support a network of people working in electronic systems across the UK • Building new links and increasing involvement with industry • Mapping the national electronics research, to ensure the work across the UK is known and noted • Encouraging and funding innovative multi-disciplinary/multi-university proposals • Working to improve, encourage and support equality, diversity and inclusion across our sector • Communicating with our network via a monthly magazine & social media • Running regular events that support our network & strategy • Launching a Big Ideas Challenge Twitter @efuturesuk Sign up to our mailing list: efutures@qub.ac.uk
- 4. Next webinar: Friday3rd July Vision & Imaging Systems AI: Vision Systems Speakers include Xilinx; University of Edinburgh; Sensing Feeling and AAEON Technology
- 5. EMBEDDING LOW-COST INTELLIGENCE WITHXCORE.AI 12 JUNE 2020
- 6. 22 TECHNOLOGY IS WOVEN THOUGHOUTOUR LIVES
- 7. 3 THE AIOT ISAPPLICABLE ACROSS MARKETS ENABLING HIGH PERFORMANCE, ACROSS VERTICALS, ECONOMICALLY Smart speaker Audio visual Appliances Lighting Security Fitness Care Diagnostics & monitoring MHealth Traffic & parking Environmental Utilities Public safety & security TAM Operations Tracking Safety Maintenance Energy management Asset tracking & predictive maintenance In car people tracking Autonomous L1 driving & safety 500M UNITS 500M UNITS 650M UNITS 450M UNITS 90M UNITS
- 8. 44 CHALLENGES OF THEAIoT REVOLUTION 45% DATA SECURITY AND AUTONOMY 38% BANDWIDTH 32% LATENCY 24% SCALABILITY 24% CLOUD INFRASTRUCTURE LIMITATIONS BASED ON PRIMARY RESEARCH WITH ELECTRONICS ENGINEERS
- 9. WHAT’S NEEDED? AIoT devicesdemand a processor with high-performance compute, efficient energy usage and a low eBOM.
- 10. A NEW KINDOF PROCESSOR Fast, flexible and economical, xcore.ai puts intelligence at the core of smart products, combining AI, DSP, control and IO compute in a one dollar device.
- 11. 77 FAST, FLEXIBLE ANDECONOMICAL 32 x 16 x 15 x 21 x ARM Cortex M7 @ 600MHzxcore.ai AI performance faster I/O processing DSP performance more 16-bit MACs Benchmarked 18 Nov 2019. Preliminary information subject to change without notice DELIVERING STANDOUT PERFORMANCE
- 12. 88 FLEXIBLE & SCALABLEARCHITECTURE DRIVING FAST TIME TO MARKET, ENABLING COST EFFECTIVE SOLUTIONS xcore device families xcore Tools xcore Libraries 3rd Party Libraries xcore LibrariesFreeRTOS Custom platform solutions xcore Libraries USB Audio Voice Human Presence Smart Home Connect Health Smart Mobility IndustryIoT SmartCities Solutions
- 13. 99 STATE OF THEART ARCHITECTURE HIGH PERFORMANCE AND ENERGY EFFICIENCY CONVERGE IN A LOW eBOM CLASS LEADER c hardware ports IO pins switch xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xtime scheduler hardware ports xtime scheduler IO pins SRAMSRAM ALU (FP + int) vector unit ALU (FP + int) vector unit High-Speed USB PHY MIPI D-PHY external LPDDR interface JTAG core PLL app PLL OTP OTP oscillator reset16 real-time logical cores, with support for scalar / float / vector instructions Vector processing unit, supports 8-bit and binarised neural network inferences Extended memory support for large applications Flexible IO ports with nano-second latency; create interfaces in software High performance instruction set for DSP, ML and cryptographic functions Integrated MIPI interface for imaging support Example software tasks
- 14. 1010 MAPPING REAL-TIME TASKS,APP TASKS, AND INFERENCING TASKS Neuralnetmodel c Hardware Ports IO pins Switch xTIME scheduler Hardware Ports xTIME scheduler IO pins High speed USB PHY MIDI D-PHY External LPDDR interface JTAG Core PLL App PLL Oscillator Reset FreeRTOS and app tasks dynamically share fixed number of thread contexts Inferencing and real time tasks allocated fixed threads at compile time I2SLEDdrivers PDMPDM c Far-fieldmicrophone processing Applicationtask Applicationtask … Applicationtask Keyworddetection FreeRTOS I2C xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core xcore logical core Internal SRAM Internal SRAM ALU (FP + int) Vector unit ALU (FP + int) Vector unit OTP OTP PDM Far-field microphone processing Keyword detection Free RTOS I2S, I2C, LED drivers Apptask PDM Apptask Apptask Apptask Neural net model
- 15. 1111 FOUR CLASSES OFCOMPUTE, ONE DEVELOPMENT PLATFORM “USING XMOS WE WERE ABLE TO REPLACE THREE SEPARATE DEVELOPMENT SYSTEMS” Richard Hollinshead, Meridian Embedded code DSP code NN Model Cortex-M DSP core NPU Hardware gates IO & accelerators Cortex SoC development Embedded code DSP code NN Model xcore IO & accelerators xcore development
- 16. 1212 PROGRAMMABLE USING INDUSTRYSTANDARD TOOLS ENABLING RAPID DEPLOYMENT AND SHORTENING TIME TO MARKET Example software tasks • Industry-standard TensorFlow Lite workflow • Automatic model translation • Community support Applicationtask Applicationtask … Applicationtask FreeRTOS • Familiar, real-time, industry-standard development environment • Community support • Wide variety of third party applications FFT FFT QSPI Filter Filter • High performance, predictable DSP • Accessed using industry standard tools • Highly optimised library kernels access xcore.ai processing CONTROL AI DSP
- 17. 1313 AI USER WORKFLOW Trainedfloating point network Lite convertor (python API) Run TFL to xcore.ai convertor Key TensorFlow component XMOS component User component Key TensorFlow component XMOS component User component ONNX componentAlternative framework flow trained network my_model.tflite to TensorFlow convertor xcore.ai micro Runtime my_model.tflite lib_xs3_ai
- 18. 1414 PROGRAMMING – PULLINGIT ALL TOGETHER xmos compiler 3rd party Libraries Executable Control source code Neural net model Dataflow source code XMOS Libraries TensorFlowLite to xcore.ai convertor Applicationtask Applicationtask … Applicationtask FreeRTOS FFT FFT QSPI Filter Filter
- 19. 1515 IN SUMMARY • TheAIoT industry has reached a tipping point that will radically transform our way of life • Success depends on being able to drive one of the most impressive feats of electronics engineering • xcore.ai is that feat
- 20.
- 21. 1 Adapting AI toAvailable Resource in Mobile/Embedded Devices Geoff Merrett Implementing AI: Running AI at the Edge 12 June 2020 | KTN & eFutures Online Webinar spatialml.net
- 22. 2 WHY AI ATTHE EDGE? Data Privacy • Increased privacy if data never leaves the edge Sending data to a central location consumes energy. Once there, the temptation is great to keep crunching them 1 Network Latency/Bandwidth/Connectivity • Cloud AI requires good networking Self-driving cars need very fast-reacting connections and cannot risk being disconnected; computing needs to happen in the car itself 1 Traffic lights in Las Vegas generate 60 terabytes a day (10% of the amount Facebook collects in a day) 1 • (the edge must fulfil requirements instead though!) 1 https://www.economist.com/special-report/2020/02/20/should-data-be-crunched-at-the-centre-or-at-the-edge “ “ “ ”
- 23. 3 WHY AI ATTHE EDGE? Power Consumption of AI • Cloud AI consumes considerable natural resource. The carbon footprint of training a single AI is up to 284 tonnes of CO2 equivalent – 5x the lifetime emissions of an average car 2 An estimate puts the energy used to train the model at over 3x the yearly consumption of the average American 3 From the earliest days, the amount of computing power required by the technology doubled every two years. But from 2012 onwards, the computing power required for today’s most-vaunted machine-learning systems has been doubling every 3.4 months 3 • An indirect benefit of moving computation to the edge, is that it has to be more efficient 2 https://www.newscientist.com/article/2205779-creating-an-ai-can-be-five-times-worse-for-the-planet-than-a-car/ 3 https://www.theguardian.com/commentisfree/2019/nov/16/can-planet-afford-exorbitant-power-demands-of-machine-learning “ “ ” “
- 24. 4 PERFORMANCE METRICS Inference atthe Edge (/End) • Connectivity, latency; privacy… • …but constrained platforms Inference Test Data result Inference Trained model Servers Servers Training Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Optimising Resource Management for Embedded Machine Learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20).
- 25. 5 EMBEDDED AI ACCELERATION •General/specialist compute units for AI rapidly increasing • Some mobile/embedded AI systems are reasonably static… • …however, others aren’t – General purpose systems – Multi-tenant systems – ‘Adaptive’ AI/event-driven operation – etc
- 26. 6 • Complexity ofhardware-software interaction has grown • Managing resources is no longer trivial, yet is increasingly needed 1 CPU Core SYSTEM RESOURCE MANAGEMENT n CPU1 Cores n GPU Cores n FPGA Cores n CPU2 Cores n T/NPU Cores n Device Variants n Workloads Samsung Exynos 5422 Xilinx Zynq Ultrascale+ HiSiliconKirin9905G NVIDIA Xavier NX
- 27. 7 DESIGN-TIME CHALLENGES PlatformDiversity How canwe develop DNN models that can: 1. operate across a wide range of different heterogeneous platforms, and 2. meet diverse application requirements? • Existing design-time approaches such as static model pruning compress the model to approximately the ‘right size’. Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Optimising Resource Management for Embedded Machine Learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20).
- 28. 8 RUN-TIME CHALLENGES WorkloadDiversity How canwe perform inference while: 1. meeting timing requirements? 2. meeting power/energy requirements? 3. meeting accuracy requirements? How can we do this: • while executing another DNN model at the same time? • while executing other foreground/ background tasks at the same time? We need dynamic DNNs… Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Optimising Resource Management for Embedded Machine Learning. In Design, Automation and Test in Europe Conference 2020 (DATE'20). CPU Type1 CPU Type2 GPU CPU Type1 CPU Type1 CPU Type1 CPU Type2 CPU Type2 CPU Type2 NPU DNN 1 DNN 2 VR/AR
- 29. 9 DYNAMIC DNNs IncrementalTrainingwithGroupConvolutionPruning L. Xunet al. Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
- 30. 10 DYNAMIC DNNs ExperimentalSetup Model: ModifiedAlexNet (~320kB) Dataset: CIFAR10 – 32*32*3 images in 10 classes – 50,000 training and 10,000 testing images Framework: Caffe Hardware: • Odroid XU3 – CPU: 4x Arm A15 ( f = 0.2–2 GHz ) + 4x Arm A7 ( f = 0.2–1.4 GHz ) – GPU: Mali-T628 ( not used in these experiments ) • Nvidia Jetson Nano – CPU: 4x Arm A57 ( f = 0.9, 1.4 GHz ) – GPU: 128x CUDA core Maxwell ( f = 0.6, 0.9 GHz ) Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
- 31. 11 DYNAMIC DNNs Results:DVFSandTaskMapping(OdroidXU3) Energy ConsumptionTop-1 Accuracy Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
- 32.
- 33. 13 DYNAMIC DNNs Results:DVFSandTaskMapping(JetsonNano) Energy ConsumptionTop-1 AccuracyEnergy Consumption Xun, Lei, Tran-Thanh, Long, Al-Hashimi, Bashir and Merrett, Geoff (2020) Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms. In Workshop on Machine Learning for CAD (MLCAD’19).
- 34. 14 RUNTIME POWER MANAGEMENT www.prime-project.org RuntimeManagement (RTM) • System software to react and predict • Controls/’knobs’ • ‘Monitors’/sensors RTM to coordinate/balance… • Mapping to heterogeneous PEs • Response to environmental factors • Power consumption/battery life • (concurrently) Executing tasks • Application(s) requirements • User requirements/QoE Bragg, Graeme McLachlan, Leech, Charles R., Balsamo, Domenico, Davis, James J., Weber Wachter, Eduardo, Merrett, Geoff, Constantinides, George A. and Al-Hashimi, Bashir (2018) An application- and platform-agnostic control and monitoring framework for multicore systems. 3rd International Conference on Pervasive and Embedded Computing, Portugal. 29 - 30 Jul 2018.
- 35. 15 CONCLUSIONS • AI ismoving to the edge… If machine learning is going to be deployed at a global scale, most of the computation will have to be done in users’ hands, ie in their smartphones 3 • …but available resources on edge platforms are typically both constrained and time-varying • We need improved approaches to manage resources in systems while providing acceptable performance Companies will learn to make trade-offs between accuracy and computational efficiency, though that will have unintended, and antisocial, consequences too 3 3 https://www.theguardian.com/commentisfree/2019/nov/16/can-planet-afford-exorbitant-power-demands-of-machine-learning Photo by Patrick Schneider on Unsplash “ “ ”
- 36. 16 ACKNOWLEDGEMENTS Lei Xun (PhDstudent) w https://www.ecs.soton.ac.uk/people/lx2u16 @XunLei_CHN spatialml.net International Centre for Spatial Computational Learning (EPSRC) w https://spatialml.net/ @spatialmlnet Power and Reliability in Many-Core Embedded Systems (EPSRC) w https://www.prime-project.org @prime_programme
- 37. 17 YOUR QUESTIONS Professor GeoffMerrett Head of Centre for IoT and Pervasive Systems e: gvm@ecs.soton.ac.uk w: www.geoffmerrett.co.uk @g_merrett
- 38.
- 39. Andrew Swirski, Founder andExecutive Director, a.swirski@beetlebox.org Real Time Low Latency Computer Vision Unit 1. 10, Chester House, Kennington Park, 1-3 Brixton Road, London, SW9 6DE
- 40. • Meeting theincreasing demands of Computer Vision • What are FPGAs and why do they perform better than CPUs • Why in previous years FPGAs have been restricted to hardware engineers • The new generation of Xilinx software development tools: • Vitis Unified Software Development platform • Vitis Vision Library • Vitis AI • ClickCV: Beetlebox’s computer vision library • Electronic Image Stabilisation with Sundance • Getting involved with our Early Access Introduction
- 41. Our vision ofthe future puts enormous demand on embedded devices to understand the world around them Autonomous delivery drone Our vision of the future
- 42. What do wemean by high performance? 1. Throughput: • Achievable Resolution • Achievable Frames Per Second (FPS) 2. Latency: • Consistent end-to-end latency 3. Power Consumption: • Battery Life • Heat Production What is high performance?
- 43. What are FPGAs Whatare FPGAs Inputs/Outputs Memory Logic DSP Engines
- 44. What happens ona CPU int main() { int a[5], b[5], output[5]; for(int i=0;i<5;i++){ output[i]=a[i]+b[i]; } } main: str fp, [sp, #-4]! add fp, sp, #0 sub sp, sp, #68 mov r3, #0 str r3, [fp, #-8] .L3: ldr r3, [fp, #-8] cmp r3, #4 bgt .L2 ldr r3, [fp, #-8] lsl r3, r3, #2 sub r2, fp, #4 add r3, r2, r3 ldr r2, [r3, #-24] ldr r3, [fp, #-8] lsl r3, r3, #2 sub r1, fp, #4 add r3, r1, r3 ldr r3, [r3, #-44] add r2, r2, r3 ldr r3, [fp, #-8] lsl r3, r3, #2 sub r1, fp, #4 add r3, r1, r3 str r2, [r3, #-64] ldr r3, [fp, #-8] add r3, r3, #1 str r3, [fp, #-8] b .L3 .L2: mov r3, #0 mov r0, r3 add sp, fp, #0 ldr fp, [sp], #4 bx lr • Variable latency
- 45. What happens ona FPGA int main() { int a[5], b[5], output[5]; for(int i=0;i<5;i++){ output[i]=a[i]+b[i]; } } + + a[0] b[0] a[1] b[1] a[2] b[2] a[3] b[3] a[4] b[4] + + + output[0] output[1] output[2] output[3] output[4] • Control over the design • Control over the latency • Control over the clock (power consumption)
- 46. With great controlcomes great responsibility • Need hardware specialists: • Hardware Description Languages: Verilog or VHDL • Design and Verification is a time-consuming process • Newer tools (High Level Synthesis) do allow for the uses of C/C++, but: • Limited subset of C • No standard libraries • Need to still understand the hardware Restricted to Hardware Engineers
- 47. The new generationof tools are now focused on System-on- Chip (SoC) • Xilinx’s Zynq series and the Zynq Ultrascale+ series contain ARM cores • Benefits of a CPU host code + FPGA accelerated code The new generation of tools CPU FPGA
- 48. Providing a familiarsoftware development environment • Develop, compile and debug code using C, C++ or OpenCL either through the provided IDE or using a makefile build flow • Embedded platforms run a version of Linux known as PetaLinux • Use Standard Libraries and Tools such as GStreamer, OpenCV and FFMPEG • Gain access to open-source Xilinx Vitis Accelerated Libraries Xilinx Vitis Unified Software Platform
- 49. Vitis Vision libraryprovides a library of low- level computer vision kernels • Provides a strong video pipeline framework • Great for forming basic pipelines: e.g. colour thresholding Xilinx Vitis Vision Library CPU FPGA CPU Collect Data Computer Vision Pipeline Output Data 30 FPSVideo In
- 50. What we cando with Vitis Vision Xilinx Vitis Vision Library Edge Detection Colour Detection
- 51. Allows the deploymentof models from deep learning frameworks such as Tensorflow and Caffe on to a specialized processor • Optimizes your model to run on FPGAs: • Prune • Quantisation Xilinx Vitis AI Development Environment CPU FPGA CPU Collect Data Pre-process Output Data 30 FPSVitis AI DPU
- 52. What we cando with Vitis AI Xilinx Vitis AI Object prediction Bounding Boxes Semantic Segmentation
- 53. High performance, highlevel FPGA kernels without the need for hardware expertise • Provides high-level, out-of-the-box functionality using software only to get computer vision developed faster • Also provides the low-level kernels needed to build custom systems or for building test systems • Integrates with industry standard software such as OpenCV and GStreamer Beetlebox ClickCV: Accelerated Computer Vision
- 54. Partnered with SundanceMicroprocessor technologies • Robotics and vision hardware experts • Robots and drones often suffer from shaky camera, especially when there is no room or power for a gimbal • Cameras with in-built stabilization, such as GoPros are often impractical • Stabilise video purely through the video itself • No sensor data Electronic Image Stabilisation (EIS) VCS-1
- 55.
- 56. Currently running theClickCV Early Access Programme • High Performance with no need for computer vision hardware specialists: • Out-of-the-box functionality • We can develop bespoke hardware for a specific solution • Provide custom systems to get testing on FPGAs up and running fast • Let your software engineers use industry-standard tools, such as OpenCV, GStreamer and FFMPEG • Areas where we are looking next: • Super Resolution • Template Matching • SLAM • Deep Learning ClickCV Early Access
- 57. Our Team Andrew Swirski MD •MEng Electronic Engineering at Imperial College London • Design Engineer at Intel Matthew Simpson Head of Systems Dr Christos-Savvas Bouganis Consultant • Reader and Director of the Integrated Digital Systems Lab at Imperial College: • Computer Vision, Image Processing, Machine Learning and SLAM on FPGAs • MSc Electronic Engineering at Imperial College London • CMOS Engineer at NXP Ashley Unitt Advisor • CTO and CSO of NewVoiceMedia, a SaaS contact centre tech provider • Acquired by Vonage Peter Collins Advisor • CEO of Permasense Ltd- sensor systems; acquired by Emerson • Chairman Inflowmatix Ltd • Chariman Guided Ultrasonics Ltd Dr Marlon Wijeyasinghe Head of HLS • PhD in Electronic Engineering at Imperial College London
- 58. Contribute back andgrow the community • Technical tutorials: Vitis Vision Library tutorial: • https://beetlebox.org/getting-started-with-computer-vision-for- vitis-embedded-systems/ • Explainer articles: What is Computer Vision and why are Neural Networks so important: • https://beetlebox.org/what-is-computer-vision-and-why-are- neural-networks-important/ • Soon launching a tutorial on Vitis AI, where we explore sign language recognition • Open Source • Soon launching our github.io page: • beetlebox.github.io Free resources
- 59. • Previously FPGAshave always been a powerful but obscure chip that only a few hardware specialists can use • With Xilinx’s new focus on SoCs and Vitis software development environment, embedded FPGAs have never been more accessible • ClickCV Computer Vision library provides out-of-the-box high performance functionality allowing the fast development of systems without the need for hardware expertise • Currently running an Early Access Programme. Come talk to us! • Check out our website Conclusion
- 60. Andrew Swirski, Founder andExecutive Director, a.swirski@beetlebox.org Any Questions? Unit 1. 10, Chester House, Kennington Park, 1-3 Brixton Road, London, SW9 6DE
- 61. EDGE AI CASE-STUDY: RADARGESTURES W W W . I M A G I M O B . C O M ALEXANDER SAMUELSSON, CTO/CO-FOUNDER JUNE 2020
- 62. • Specialized inEdge AI • Experience from 20+ Edge AI customer projects • We offer • Imagimob AI – Software-tools-as-a-Service • Edge AI expertise • Based in Stockholm, Sweden W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep learning GestureRecognitiononTheEdge|2020 Introduction Imagimob
- 63. • What isEdge AI? • Imagimob AI • Case study – radar gestures in earphones • Edge AI opportunities for the future W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep learning GestureRecognitiononTheEdge|2020 What I will talk about
- 64. So, WHAT IS EDGEAI? W W W . I M A G I M O B . C O M COMPANYPRESENTATION||JUNE2019 Placing AI at the very Edge of the network • Where data is collected • Autonomous • Real time • Low power • Strong privacy • We are democratizing Edge AI Cloud aggregated raw data • Needs connection • High latency • High communication cost • High power consumption • Weak Privacy EDGE AI Cloud aggregated curated data • Autonomous • Real time • Low communication cost • Low power consumption • Strong privacy CLOUD AI RAW DATA
- 65. EDGE AI APPLICATIONDEVELOPMENT IMAGIMOB AI SUPPORTS THE FULL CYCLE FROM DATA COLLECTION TO FINISHED EDGE AI APPLICATION W W W . I M A G I M O B . C O M COMPANYPRESENTATION||2019 DATA COLLECTION & LABELLING DATA MANAGEMENT MODEL BUILDING PREPARED FOR EDGE MODEL EVALUATION EDGE OPTIMIZATION AND VERIFICATION APPLICATION PACKAGING Imagimob Capture (Android + Sensor) MCUTraining Service (Cloud) Imagimob Studio (PC) Imagimob Studio (PC) Imagimob Studio (PC) Train Validation Test
- 66. Lot’s of applicationsdepending on sensors • Predictive maintenance • Anomaly detection • Human activity recognition • Wake-word detection (Hi Alexa) • … Radar + Edge AI • Material or surface recognition • Detect deviations/defects in manufacturing • Object detection • Detect hand gestures in headphones W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 What can you do with Edge AI?
- 67. • Working proofof concept shown at CES with Acconeer • Application running in real time on the actual radar module • ARM-Cortex M4 processor with only 256KB shared with BLE stack, firmware, other applications. • Impossible without Edge AI, the data generated by the sensor and the almost infinite variations in how a gesture can be performed demands an AI solution. • Sending the data off device would drain the battery and be impossible over bluetooth W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 Radar gestures in headphones
- 69. W W W. I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 Challenge #1 – Data collection USB WIFI Acconeer radar
- 70. W W W. I M A G I M O B . C O M GestureRecognitiononTheEdge|2020
- 71. Radar output: 30KB data per second Model predictions: 14.3 Hz W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 Challenge #2 - Preprocessing Learned Preprocessing Manualpreprocessing Avg FFT Hanning Window Abs Sum Sliding Window
- 72. W W W. I M A G I M O B . C O M GestureRecognitiononTheEdge|2020
- 73. • Testing andverifying Edge AI models is a REAL pain • To test on device you would have to go all the way to C code • Moreover you would have to reflash the firmware of your headphones • To get a really good test we would have to do this on several headphones in different locations multiple times each week W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 Challenge #3 – Testing and verifying
- 74. W W W. I M A G I M O B . C O M GestureRecognitiononTheEdge|2020
- 75. W W W. I M A G I M O B . C O M GestureRecognitiononTheEdge|2020
- 76. • End-to-end • Wehave solutions to the major problems • Data collection • Model evaluation/testing • Computation designed for the Edge all the way W W W . I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 How are we different?
- 77. W W W. I M A G I M O B . C O M Imagimob AI Edge AI | Software-tools-as-a-Service | Deep Learning GestureRecognitiononTheEdge|2020 The future (opportunities)
- 78. THANK YOU W WW . I M A G I M O B . C O M FOLLOW US @imagimob @samuelsson_al SIGN UP FOR OUR IMAGIMOB AI EARLY ACCESS PROGRAM GET IMAGIMOB AI FOR A FULL MONTH + EDUCATION LIMITED SEATS JOIN THE WORKSHOP LATER TODAY