"Approaches for Vision-based Driver Monitoring," a Presentation from PathPartner Technology

Copyright © 2017 PATHPARTNER TECHNOLOGY 1
Approaches for Vision-based
Driver Monitoring
Jayachandra Dakala
May 2017

• Inattention – Distraction and Drowsiness
• Vision-based DMS – Overview and Scope
• Detection vs. Classification – CNN vs. Boosted Cascades
• Drowsiness from Eye
• Optimization on PC
• Summary
Outline

Distraction & Drowsiness
Comprehending
Environment
Decision
Making
Taking
Action
Driving
Symptoms of Fatigue
Repeated yawning / Lazy steering
Slower reaction and responses
Nodding off at the wheel
Day dreaming
Difficulty keeping eyes open and
burning sensation in the eye
Vehicle wandering from the road or
into another lane
Visual
distraction
Auditory
distraction
Cognitive
distraction
Biomechanical
distraction
Types of Distraction (acc. To NHTSA)

• Visual cues
• Eyes-off-road duration
• Head-off-road duration
• Texting / Talking over phone
• Eye closure, eye blink, PERCLOS
• Yawning, nodding
• Standard deviation of lane position
Distraction and Drowsiness

Vision-based DMS
Driver Monitoring System
Drowsiness Distraction Driving behaviourEmotions Personalization
YawningEye closure,
Eye openness,
Pupil dilation,
Eye blink
frequency
Nodding off
&
Head down
Looking
elsewhere!
Talking over
phone,
Texting, etc.
Anger,
Stress,
Pain, etc.
Steering angle,
Acceleration &
Break,
(OBD data in
general)
Recognition &
Personalized
learning

Core vision technologies needed!
Eye
Measurements
Face and
Landmark
Detector
Eye Gaze
3D Face Pose
Facial Expression
Face Recognition

Cascaded detector
All
image
patches
Detected
faces
Rejected non-face
patches
Weak classifier
Classification score

Choice of weak classifier
Decision stumps on
• Multiscale Haar-like features
• Multiscale Local Binary Patterns
Decision trees on
• Pixel differences: (p1-p2)
• Normalized pixel differences (NPD): (p1-p2)/(p1+p2)
• These features use
averaging over an area of
the image!
• Susceptible to occlusions
and alignment!
• Needs many cascades to
cover all face poses!
Single cascade could handle
multiple poses, better at
handling occlusions!

Multi-task Cascaded Detector
All
image
patches
Detected
faces with
landmarks
& pose
Rejected non-face
patches
Incremental detection score, shape update, pose update!
Multi-task Classification / Regression tree

CNNs for Detection
Image Multiscale linear search Selective search
• RCNN-like methods
• Region Proposal + Region evaluation = Region Inference
• Region Proposal
• Multiscale Linear search
• Selective search
• CNN based Region Proposal network

• Motivation: Let the CNN do the search
• Pros: Fast, One-pass inference
• Cons:
• Chance of mis-aligned face bbox, sensitive to Ground truth preparation
• High false positive rate
• Needs effective search and NMS method
• Model: AlexNet trained as a face/no-face classifier
• Method:
• Fine-tune AlexNet with Face/No face patches
• Perform Network surgery, convert fc6, fc7 and fc8 to convolutional layers
• Pass multiscale images to network, obtain heatmap for each scale
• Obtain candidate Bounding boxes by thresholding heatmap
• Perform NMS to obtain final bbox
Deep Dense CNNs
Required “Depth”
of CNNs can still
make it difficult on
embedded
systems!

• For RCNN-like models and Dense detection, AlexNet was fine-tuned on
AFLW dataset (~24 k images, 65 k faces)
• Obtained 100 k face images, 240 k non-face images by GT bounding box
jittering
• Model evaluation on FDDB dataset (~2800 images, ~5 k faces)
• Evaluation metric: IoU > 50%
Experiments / Results

Experiments / Results (on TitanX Pascal) on FDDB
* For one set of scale factor (1.3) and step size (32)!
** Selective search with 2400 regions
*** For faces of size > 64x64
FD Method Accuracy False
positives
Time
RCNN Linear search* 66.03% 1806 14.04 sec
Selective
search**
64.55% 1100 7.52 sec
Deep
dense
NMS-Max 74.03% 2808 1.3 sec
NMS-Avg 81.23% 1564 1.67 sec
NPD
Cascade
Linear search 72.24%,
95.68%***
500 ~40 ms on PC

Hence, we say…
Eye
Measurements
Face and
Landmark
Detector
Eye Gaze
3D Face Pose
Facial Expression
Face Recognition
• Use boosted multi-task cascades for
detecting correlated tasks
Ex: Face, landmarks, and face pose detection
• Use CNNs for one-shot inference of a
given patch
Ex: Eye state analysis, facial expression, etc.,
on detected faces

• Drowsiness is reflected in PERCLOS, eye blink variations
• Eye openness is specific to each person
• Challenging to differentiate partially open eyes, closed eyes, and the
person specific eye openness
Drowsiness from Eye

• Trained as a patch classifier, i.e., Open-eye vs. Closed-eye vs. No-Eye inference
• Total 0.75 million parameters, 80 times smaller than AlexNet
EyeNet

• Motivation
• Make model robust to mis-aligned query patch
• Extract meta-data from class-conditioned filters learned in EyeNet
• Perform Network surgery on EyeNet
• Transform weights of FC5 into 9x5 Convolution, FC6 and FC7 to 1x1
convolutions
• Perform Hole Transform (Atrous algorithm) on Conv2, Conv3 and Conv4
• Model desensitized to Facial effects, eyebrows, and spectacles by Hard-negative
mining during training
• Takes ~15 ms on CPU for a patch of 60x40
EyeNet for Dense Prediction

EyeNet for Dense prediction
Sample heatmaps

• Model works well on a conservative eye crop, independent of
template size
• The saliency maps, “heatmaps” generated indicate the model’s
confidence that each pixel belongs to the target classes
• Meta-data useful to drowsiness detection encapsulated in the
distribution of the dense confidence scores
EyeNet for Dense prediction

Optimization of Multi-task Cascaded Detector
Parallelize “one” tree
for “n” patches!
Parallelize “n” trees
for “one” patch!
Parallelize “n” trees
for “m” patches!
0
20
40
60
80
100
120
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
379
397
415
433
451
469
487
505
523
Cumulative rejection rate after every tree!
• Using SIMD implementation and 3 cores of an Intel i7, gives ~25 fps for
VGA (640x480)
Tree number
%ofrejections

• With minimal system delay, parallelize multi-task Cascade Detector and
the Dense Eyenet
• We could get real time performance on 4-core intel i7 for VGA
• Parallelizing between DSP and EVEs, and with SIMD on EVE, we
forecast real-time performance on TI’s TDA2x
Optimization of Overall System

• Group correlated tasks together to learn multi-task cascades
• Use multi-task cascades for computationally intensive detection problems
• Use negative patch rejection statistics to optimize multi-task cascades
• Use CNNs for dense prediction from a coarsely cropped patches
Summary

• Y. Dong, Z. Hu, K. Uchimura and N. Murayama, “Driver Inattention Monitoring System for Intelligent
Vehicles: A Review,” in IEEE Transactions on Intelligent Transportation Systems, vol. 12, no. 2, pp. 596-
614, June 2011.
• S. Liao, A. K. Jain and S. Z. Li, “A Fast and Accurate Unconstrained Face Detector,” in IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 211-223, Feb. 1 2016.
• D. Chen, S. Ren, Y. Wei, X. Cao, and J. Sun, “Joint cascade face detection and alignment,” in European
Conference on Computer Vision, 2014.
• Cao, X., Wei, Y., Wen, F., Sun, J.: Face Alignment by Explicit Shape Regression. In: Computer Vision and
Pattern Recognition (2012).
• Sachin Sudhakar Farfade, Mohammad J. Saberian, and Li-Jia Li. 2015. Multi-view Face Detection Using
Deep Convolutional Neural Networks. In ICMR 2015.
• L.C. Chen, G Papandreou, I Kokkinos, K. Murphy, A. Yuille, “Semantic Segmentation with Deep
Convolutional Neural Networks, atrous Convolutions and Fully Connected CRFs,” ICLR 2015.
• F. Yu, V. Koltun, “Multiscale Context Aggregation by Dilated Convolutions,” ICLR 2016.
Resource Slide

PathPartner is a top notch design engineering services company and works with several
semiconductor companies, OEMs and ODMs in embedded media-centric devices and
intelligent systems
About PathPartner
• Incorporated in July 2006 HQ in
Bangalore, India
• R&D Centers: Bangalore, India and
California, USA
• Marketing representatives in USA,
Japan and India
• PathPartner is a member of
Embedded Vision Alliance, GENIVI
and CARCONNECTIVITY consortium
• Semiconductor Companies
• OEMs and ODMs
• Camera Sensor Integration &
Image Quality Tuning
• Device Drivers, BSP & OS
integration
• Embedded Vision
• Multimedia Codecs & Middleware
• Algorithm Optimization
• Present company strength is ~250
• Grown at 40% in 2015-16

PathPartner Technology Pvt.Ltd.
#16, PSS Plaza, 1st and 2nd Floor,
New Thippasandra Main Road,
HAL III Stage, Bangalore-560075
Contact Us
+91 80 6772 2000 | +1 408 242 7411 | +81 9055466819
sales.india@pathpartnertech.com | sales.usa@pathpartnertech.com | sales.japan@pathpartnertech.com
THANK YOU

Questions?

"Approaches for Vision-based Driver Monitoring," a Presentation from PathPartner Technology

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to "Approaches for Vision-based Driver Monitoring," a Presentation from PathPartner Technology

Similar to "Approaches for Vision-based Driver Monitoring," a Presentation from PathPartner Technology (20)

More from Edge AI and Vision Alliance

More from Edge AI and Vision Alliance (20)

Recently uploaded

Recently uploaded (20)

"Approaches for Vision-based Driver Monitoring," a Presentation from PathPartner Technology