This document discusses the development of an ADAS system from scratch by Itseez, which was acquired by IoTG/Intel in 2016. It outlines their strategy of creating highly portable computer vision algorithms using OpenCV and developing their own datasets. It details their timeline and challenges faced with hardware issues, testing paradigms, and limited datasets. Recent proofs of concept using semantic segmentation and obstacle detection with deep learning are also mentioned.

1. Building ADAS
SYSTEM from scratch
Alex Myakov, Chief CV Advocate
Yury Gorbachev, CV Integration & Optimization lead
September, 2016

2. Who we are ?
Itseez was acquired by IoTG/Intel in July, 2016
Itseez was founded in 2005:
• 3 ex- Intel co-Founders + 1 Principal Engineer
• OpenCV development and support (2005-present)
• OpenVX initiative leaders: v1.0 and v1.1 were published in
October, 2014 and in May, 2016
SW Products: ADAS, Facense, AcceleratedCV (ACV)
Skills: CV algorithms, HW specific optimization, 3+
years of deep learning (DL)
Industries: automotive, security, robotics, wearables,etc

3. Building ADAS System from Scratch
The Dream and Ambition:
• Create state-of-the-art software based front
camera ADAS algos
• License such algos to Tier-1s and OEMs
Starting point (late 2013):
• Strong knowledge of CV
• Strong knowledge of embedded/optimization
• Good knowledge of cameras/sensors/optics
• No ADAS/automotive specific knowledge

4. Building ADAS solution from scratch
Strategy:
• Highly portable CV algos (pure ARM optimized code)
• Open SW platform (Android, Linux)
• Simplest system possible
• Cheapest COTS components
• Camera (optics, sensor, packaging)
• HW platform
• Easy/fast installation in any car with no dependence
on car parameters
• Automatic or simple calibration

5. ADAS Project Timeline
2013
• TSR
• Demo platform:
Nexus 4
2014
• +LDW
• +FCW
• Demo Platform:
Asus Transformer
Tablet + Android OS
+ Standalone
Camera
2015
• +PD
• Demo Platform:
TK1+ Linux OS +
Standalone Camera
• Demos with QNX:
• CES 2015
• TU Update 2015
2016
• PoCs:
• +Semantic Road
Segmentation
• +Obstacle Detection
• +Driver Monitoring
• Demos with QNX:
• CES 2016

6. Demo Setup
Camera:
• 1M
• 1280H x 800V
• HDR/WDR
• 30 fps
Embedded
Platform
USB 3.0
Snap-and-go concept:
• Simple and fast installation in any car
• No dependence on car parameters
• Automatic calibration or simple
calibration

7. Lessons Learned
We expected ADAS to be just another CV application !
• we ended up running into and solving lots of issues
• SW development/testing paradigm
• HW issues
• Datasets

8. ADAS Solution SW Architecture
TSR LDW FCW PD
Common Image Processing Pipeline + Autocalibration
OpenCV
IPP AcceleratedCV (ACV)
x86 ARM
CV algo prototyping on desktop
Lab testing/CI on server
Execution on target
Live test/benchmarking

9. SW design approach
ADAS algorithms are purely software based:
• Possible to design and test on desktops
• Purely based on OpenCV
• No special software skills are required (GPU, DSP,
etc)
• Flexible and upgradable
Solved platform compatibility issues
• No vendor provides cross-platform CV framework
• OpenCV is limited in supporting this
Created AcceleratedCV (ACV) library to address
platform compatibility issue

10. Continuous integration
Any change in ADAS algorithms or processing
pipeline requires complete re-evaluation
• Detection/processing quality on the entire dataset
• Performance figures for each ADAS algorithm
Benefits from pure SW based approach
• Quality evaluation on servers/cloud for entire dataset
• Performance benchmarking on multiple HW targets
• Reduces test time from days to hours!

11. Datasets
CV algorithms require datasets for design and
testing:
• No available commercial datasets
• Research datasets cannot be used for products
Our own datasets for each ADAS algo were created:
• Different conditions (rain, snow, sun)
• Geographical locations
• Dataset annotation and management tools
• Many days of driving + many months of annotation
The market offering of quality annotated datasets is
still very limited !

12. Datasets stats
TSR: 2.5K good unique signs
PD: 83K+ pedestrian bounding boxes
FCW: 5K+ different cars and ca 1K trucks
LDW: 0.5M+ boundaries

13. HW Issues
HW issues are caused by consumer “gradeness” of
components
Temperature issues:
• Camera overheating -> skipped or corrupted frames
• HW platform overheating -> throttling
System issues:
• throttling and unpredicted system behavior under
heavy processing loads
Mechanical issues:
• USB 3.0 cable connectors get loose and break

14. Where is Deep Learning in our algos?
Original ADAS algos were based on classical CV
• Embedded platforms were too weak and not able to
provide required performance Gflops
• Datasets were too small to yield quality DL results
• DL technology was not fully there
Conventional CV + small DL networks:
• PD: validation of PD results based on classical CV –
increase DR and reduce FA rate
• Driver monitoring: conventional face detector + DL
based headpose estimation

15. Semantic Road Segmentation - PoC

16. Obstacle Detection using SfM - PoC
•We estimate 3D coordinates using
points tracking and vehicle speed.
•Obstacles are calculated as
clusters of points above the road
plane.

17. Semantic segmentation + Obstacle Detection

18. Deep Learning - Silver Bullet ?
• Great new CV tool
• Large datasets
• A few orders of magnitude more compute
than classic CV
• FuSa implications

19. Questions ?
Contact Details:
Alex Myakov, Chief CV Advocate
Email: alex.myakov@intel.com