“Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Presentation from Cadence
•
0 likes•18 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/06/tensilica-processor-cores-enable-sensor-fusion-for-robust-perception-a-presentation-from-cadence/ Amol Borkar, Product Marketing Director at Cadence, presents the “Tensilica Processor Cores Enable Sensor Fusion for Robust Perception” tutorial at the May 2023 Embedded Vision Summit. Until recently, the majority of sensor-based AI processing used vision and speech inputs. Recently, we have begun to see radar, LiDAR, event-based image sensors and other types of sensors used in new AI applications. And, increasingly, system developers are incorporating multiple, heterogeneous sensors in their designs and utilizing sensor fusion techniques to enable more robust machine perception. In this presentation, Borkar explores some of the heterogeneous sensor combinations and sensor fusion approaches that are gaining adoption in applications such as driver assistance and mobile robots. He also shows how the Cadence Tensilica ConnX and Vision processor IP core families and their associated software tools and libraries support sensor fusion applications with high performance, efficiency and ease of development.
Recommended
Recommended
More Related Content
Similar to “Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Presentation from Cadence
Similar to “Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Presentation from Cadence (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
“Tensilica Processor Cores Enable Sensor Fusion for Robust Perception,” a Presentation from Cadence
- 1. Tensilica Processor Cores Enable Sensor Fusion for Robust Perception Amol Borkar Product Marketing Director Cadence
- 2. TENSILICA® CUSTOMERS >50B Processors SHIPPED DSP LICENSING REVENUE #1 DSP IP LICENSING REVENUE SEMICONDUCTORS 19 19 of the Top 20 SEMICONDUCTOR VENDORS USE TENSILICA IP Cadence Tensilica Processor and DSP IP Business GLOBAL ECOSYSTEM 200+ ECOSYSTEM PARTNERS TENSILICA LICENSEES 300 1999 2002 2005 2014 WORLDWIDE LICENSEES 2008 2011 5 0 100 15 0 20 0 2020 350+ 2017 250 PROCESSOR LICENSING REVENUE #2 Processor IP LICENSING REVENUE 2 © 2023 Cadence Design Systems, Inc.
- 3. Target Markets for Cadence Tensilica DSPs 3 © 2023 Cadence Design Systems, Inc.
- 4. • Better quality • Have multiple sensors of same kinds • Two different type of sensors to compensate for the error generated by one sensor • Better reliability • Redundancy: if one fails other works • Measuring what is not possible with one sensor • Image sensor + Radar: may not work well at nighttime so add a radar • Image Sensor + IR Sensor • Short distance, Mid distance, Long distance • Image Sensor, Lidar, Radar • Utilize each sensor’s strength and minimize their weakness Why Sensor Fusion? 4 © 2023 Cadence Design Systems, Inc.
- 5. Types of Sensor Fusion Sensor Fusion Image Sensor Image Sensor Sensor Fusion Image Sensor Radar or Lidar Sensor Fusion Image Sensor Event-Based Sensor Sensor Fusion Image Sensor IMU Sensor Fusion Gyro Magnetometer Accelerometer Sensor Fusion Sensors V2X (location) or GPS © 2023 Cadence Design Systems, Inc. 5
- 6. Types of Sensor Fusion Early Fusion Mid Fusion Late Fusion Fusing at point of data Example: Stereo sensors Feature level Example: Image and radar doing feature extraction At result level Example: Image and radar both identifying object © 2023 Cadence Design Systems, Inc. 6
- 7. Stereo Sensors Single (Mono) visible camera can not measure the distance to an object Add a second sensor (Stereo) and use sensor fusion to measure distance Sensor Fusion Image Sensor Image Sensor One such algorithm, Semi Global Matching, assumes left and right images are rectified • PixelWise cost compute uses Birchfield Tomasi and later box filter of BT for 3x3 window around point (x,y) • Disparity refinements: Involves uniqueness find, quadratic interpolation, disparity of right image and disparity validation • Classical image processing algorithm • Requires processing on each pixel © 2023 Cadence Design Systems, Inc. 7
- 8. Sensor + IMU: Classical Sensor Fusion © 2023 Cadence Design Systems, Inc. https://pub.mdpi-res.com/sensors/sensors-19-03747/article_deploy/html/images/sensors-19-03747-g001.png?1568249905 Benefits • Implemented in conjunction with one or more cameras • IMU provides refresh rates of 1kHz+, camera at 30-60 fps • SLAM calculation and pose estimates at refresh rate faster than camera • Able to track movements more accurately • Can compensate if camera is unable to find good features to track f1 f2 f2 f3 Features IMU & Camera States Visual Measurements IMU Measurements x1 x3 x4 x5 https://www.mdpi.com/sensors/sensors-19-01624/article_deploy/html/images/sensors-19-01624-g001.png IMU Camera Pre-Integration Feature Detection and Tracking Initialization Sliding Window Visual-Inertial Optimization 6-DOF Pose Loop Closure 30-60Hz IMU only 8
- 9. Radar + Camera: Using Kalman Filter and GNN • Inputs: • Multiple camera and radar sensors mounted on Ego vehicle would provide multiple detections clusters received from numerous surrounding objects • Outputs • Assigned tracks for the detection clusters along with internal state of those tracks for updates for next frame Radar #1 Radar #2 Radar #6 …. Camera #1 Camera #2 Clustered Detections from All Sensors Multi-Object Tracker Using Kalman Filter and GNN Association Track #1 Track #2 Track #3 …. 9 © 2023 Cadence Design Systems, Inc.
- 10. • A lot of sensor fusion relies on classical approaches: Kalman filter, etc. • For large and complex systems, scalability is a big problem • Inefficient to manually code “rules” for each corner case • Over time, these rules will become difficult to maintain or improve • AI: • Achieve higher levels of automation • Scalability • Past decade, majority of speech and image/video processing has transitioned to neural networks for better performance/accuracy • Now radar and lidar-based classification and object detection is moving to AI, also • For AI to work well, we need data, lots of it • Image + radar + lidar data is limited at the moment, short-term problem AI and Sensor Fusion? © 2023 Cadence Design Systems, Inc. 10
- 11. • Using Single Shot object / pedestrian detection with only RGB or only depth data can have limitations • Example: Detecting objects in group, occluded objects • Remarkable detection accuracy improvements can be obtained by fusing features from subnets processing RGB and depth data – followed by a single network for fused data RGB + Depth Fusion with AI for Robust Object Detection 11 © 2023 Cadence Design Systems, Inc.
- 12. • Sensor fusion-based 3D object detection • Has 2 subnets • RPN (Region Proposal Network) • PointRCNN • Some additional processing (pre and post) • Fusion of features from pointcloud and image is done in RPN • RPN generates bounding box (BBOX) data which is further fine-tuned by PointRCNN Lidar + Camera: Using EPNet BBox Img RPN Proposal Layer Processing for PointRCNN PointRCNN Confidence of BBox 12 © 2023 Cadence Design Systems, Inc.
- 13. Used in various markets: consumer, automotive, … Definition depends on type of sensors being used Different sensors require different processing Traditional digital signal processing algorithms are still being used Various AI-based algorithms are being experimented Amount of processing depends on size of sensors and type of sensors Your solution still needs both traditional digital signal processing and AI processing Sensor Fusion Summary 13 © 2023 Cadence Design Systems, Inc.
- 14. Vision • Image/Vision Processing • On Device AI • AR/VR • ADAS • Mobile • Sensor Fusion Floating Point • AI/ML • Motor Control • Sensor Fusion • Object Tracking • AR/VR • HPC Xtensa NX Tensilica DSPs Xtensa LX Performance Vision Q7, Q8 Vision P1, P6 Floating Point KQ8, KQ7 Floating Point KP6, KP1 High Performance Low Power Radar / Lidar / Comms • Automotive Radar/Lidar • Sensor Fusion • V2X, 5G, LTE, Wireless • WiFi, Smart Grid • Infrastructure and Terminals ConnX B10, B20 ConnX 120, 110 14 © 2023 Cadence Design Systems, Inc.
- 15. Cadence Tensilica: Comprehensive Software Solutions Cadence® Compiler / Tool Cadence SW library / Runtime User Code Cadence Tensilica® DSP and Accelerators OS Layer (XTOS, XOS, ThreadX, FreeRTOS) Embedded C/C++ Halide OpenCL ONNX/TensorFlow/ PyTorch Xtensa C/C++ Compiler (LLVM) OpenCL Compiler (LLVM) Halide Compiler XNNC XAF TensorFlow Micro Lite ANN CV Lib / SLAM Lib / DSP Lib / Eigen Lib / Simulink Lib/ Radar Lib OpenCL Runtime OpenCL BIFL Library NN Library Audio Lib / NN Library XRP E c o s y s t e m iDMA Memory Manager XIPC Xtensa & TIE Vision Radar / Lidar / Comms Audio / Voice Cadence Low level SW Components HAL Tensilica Xtensa Xplorer IDE AI Processor 15 © 2023 Cadence Design Systems, Inc.
- 16. Cadence DSPs for Sensor Fusion 16 Processing Capacity 400GOPS to 3.2TOPs processing capacity Sensor Fusion Need Cadence Offering Domain-Specific Sensor Processing Vision, Radar, Audio/Voice DSPs Different Data Types (8,16,32 bit) fixed point, complex, (16,32,64 bit) FP data type support Traditional Digital Signal Processing + AI Traditional DSPs with optimized instruction set >2TOPS AI processing SW Tools and Library Various optimized library, NN complier, NN-lib © 2023 Cadence Design Systems, Inc.
- 17. Tensilica DSP Customer Success GW5400, Automotive Smart Viewing Camera Processor Kneron KL720 Black Sesame Technologies' A1000 (HS2) Renesas RH850/V1R-M RH850/V1R-M Data Acquisition Tensilica BBE32 DSP RAM Flash CPU 1 CPU 2 Target detection Object Classification Conflicts & escapes Tracking RF ADC MMIC #1 MMIC #2 High Speed High Speed )))) CAN Ethernet Flexray Radar Sensor Architecture NXP S32R45/41 4D Imaging Radar Andes Automotive Radar SOC X9: Automotive Applications Processor V9: Automotive Processor SemiDrive Visconti 17 © 2023 Cadence Design Systems, Inc.
- 18. Summary Cadence Tensilica Group is a leading supplier of IP for edge device sensor processing with on-device AI Cadence® Tensilica® DSPs are well-suited for sensor fusion Tensilica DSPs and AI solutions for automotive-grade products are already in production Rich environment of third-party solution providers and partners 18 © 2023 Cadence Design Systems, Inc.
- 19. One Last Thing… Come visit our booth #117 • See demonstrations of our customers’ products in real- world automotive, smart camera, and IoT applications 19 © 2023 Cadence Design Systems, Inc. Cadence® Tensilica® Vision Q8 and Vision P1 DSPs www.cadence.com/go/VisionQ8P1 AI-Based Sensor Fusion https://cariad.technology/de/en/news/stories/sen sor-fusion-introduction.html Vision DSP Video https://www.youtube.com/watch?v=eXegAFLqz-g