Downloaded 76 times
Uploaded byYury Gorbachev
Building ADAS system from scratch
This document discusses the development of an ADAS system from scratch by Itseez, which was acquired by IoTG/Intel in 2016. It outlines their strategy of creating highly portable computer vision algorithms using OpenCV and developing their own datasets. It details their timeline and challenges faced with hardware issues, testing paradigms, and limited datasets. Recent proofs of concept using semantic segmentation and obstacle detection with deep learning are also mentioned.
More Related Content
Similar to Building ADAS system from scratch
Building ADAS system from scratch
- 1. Building ADAS SYSTEM fromscratch Alex Myakov, Chief CV Advocate Yury Gorbachev, CV Integration & Optimization lead September, 2016
- 2. Who we are? Itseez was acquired by IoTG/Intel in July, 2016 Itseez was founded in 2005: • 3 ex- Intel co-Founders + 1 Principal Engineer • OpenCV development and support (2005-present) • OpenVX initiative leaders: v1.0 and v1.1 were published in October, 2014 and in May, 2016 SW Products: ADAS, Facense, AcceleratedCV (ACV) Skills: CV algorithms, HW specific optimization, 3+ years of deep learning (DL) Industries: automotive, security, robotics, wearables,etc
- 3. Building ADAS Systemfrom Scratch The Dream and Ambition: • Create state-of-the-art software based front camera ADAS algos • License such algos to Tier-1s and OEMs Starting point (late 2013): • Strong knowledge of CV • Strong knowledge of embedded/optimization • Good knowledge of cameras/sensors/optics • No ADAS/automotive specific knowledge
- 4. Building ADAS solutionfrom scratch Strategy: • Highly portable CV algos (pure ARM optimized code) • Open SW platform (Android, Linux) • Simplest system possible • Cheapest COTS components • Camera (optics, sensor, packaging) • HW platform • Easy/fast installation in any car with no dependence on car parameters • Automatic or simple calibration
- 5. ADAS Project Timeline 2013 •TSR • Demo platform: Nexus 4 2014 • +LDW • +FCW • Demo Platform: Asus Transformer Tablet + Android OS + Standalone Camera 2015 • +PD • Demo Platform: TK1+ Linux OS + Standalone Camera • Demos with QNX: • CES 2015 • TU Update 2015 2016 • PoCs: • +Semantic Road Segmentation • +Obstacle Detection • +Driver Monitoring • Demos with QNX: • CES 2016
- 6. Demo Setup Camera: • 1M •1280H x 800V • HDR/WDR • 30 fps Embedded Platform USB 3.0 Snap-and-go concept: • Simple and fast installation in any car • No dependence on car parameters • Automatic calibration or simple calibration
- 7. Lessons Learned We expectedADAS to be just another CV application ! • we ended up running into and solving lots of issues • SW development/testing paradigm • HW issues • Datasets
- 8. ADAS Solution SWArchitecture TSR LDW FCW PD Common Image Processing Pipeline + Autocalibration OpenCV IPP AcceleratedCV (ACV) x86 ARM CV algo prototyping on desktop Lab testing/CI on server Execution on target Live test/benchmarking
- 9. SW design approach ADASalgorithms are purely software based: • Possible to design and test on desktops • Purely based on OpenCV • No special software skills are required (GPU, DSP, etc) • Flexible and upgradable Solved platform compatibility issues • No vendor provides cross-platform CV framework • OpenCV is limited in supporting this Created AcceleratedCV (ACV) library to address platform compatibility issue
- 10. Continuous integration Any changein ADAS algorithms or processing pipeline requires complete re-evaluation • Detection/processing quality on the entire dataset • Performance figures for each ADAS algorithm Benefits from pure SW based approach • Quality evaluation on servers/cloud for entire dataset • Performance benchmarking on multiple HW targets • Reduces test time from days to hours!
- 11. Datasets CV algorithms requiredatasets for design and testing: • No available commercial datasets • Research datasets cannot be used for products Our own datasets for each ADAS algo were created: • Different conditions (rain, snow, sun) • Geographical locations • Dataset annotation and management tools • Many days of driving + many months of annotation The market offering of quality annotated datasets is still very limited !
- 12. Datasets stats TSR: 2.5Kgood unique signs PD: 83K+ pedestrian bounding boxes FCW: 5K+ different cars and ca 1K trucks LDW: 0.5M+ boundaries
- 13. HW Issues HW issuesare caused by consumer “gradeness” of components Temperature issues: • Camera overheating -> skipped or corrupted frames • HW platform overheating -> throttling System issues: • throttling and unpredicted system behavior under heavy processing loads Mechanical issues: • USB 3.0 cable connectors get loose and break
- 14. Where is DeepLearning in our algos? Original ADAS algos were based on classical CV • Embedded platforms were too weak and not able to provide required performance Gflops • Datasets were too small to yield quality DL results • DL technology was not fully there Conventional CV + small DL networks: • PD: validation of PD results based on classical CV – increase DR and reduce FA rate • Driver monitoring: conventional face detector + DL based headpose estimation
- 15.
- 16. Obstacle Detection usingSfM - PoC •We estimate 3D coordinates using points tracking and vehicle speed. •Obstacles are calculated as clusters of points above the road plane.
- 17. Semantic segmentation +Obstacle Detection
- 18. Deep Learning -Silver Bullet ? • Great new CV tool • Large datasets • A few orders of magnitude more compute than classic CV • FuSa implications
- 19. Questions ? Contact Details: AlexMyakov, Chief CV Advocate Email: alex.myakov@intel.com