Sentiance's Bertrand Fontaine gives a few concrete use-cases for deep learning on sensor data: transport classification, driver/passenger classification, and driver behavior characterization.

1. From mobile data to smart life.

2. • Which data do we collect and what information do we extract
• Transport classifier
• Driving behaviour characterisation
• Driver passenger classification
Overview

3. • Our clients integrate our SDK in their app
• SDK detects stationaries: sends GPS fix, start/end timestamps
• SDK detects transports: one GPS fix per minute, accelerometer and gyroscope
sensor data
• All processing done on AWS cloud
Data collection

4. Low-level DATA (GPS+sensors)

5. Driving behaviour profiling
Aggregate
User profile
For each trip

6. Transport Classification

7. • One waypoint per minute
• Timestamps
• Coordinates (GPS+cell tower+wifi)
• Speed
• Accuracy
Waypoints
Sensors
segment trip and classify
Accelerometer
Gyroscope
BIKING WALKING IDLE
Transport data

8. Transport Classification

9. Sensor processing
for driving behaviour

10. Phone orientation alignment
• What we want: longitudinal and lateral car accelerations, angular velocity
x
z
x
z
x
z
y
z
xy
Straight up Mounted In pocket On passenger seat
• What we have: a phone with axes in arbitrary direction
• Align the referential of the phone
with the referential of the car = find
rotation matrix
z
Gravity pull
x
y
x
Gravity pull
x
z
y
z
xyKalman filter
Gravity pull
x
y
x
y
Gradient descent

11. Phone orientation alignment
z
rotation
matrix
longitudinal
lateral
• Compared externally with high-end inertial measurement units
• As good as 100k $ hardware
accelerations
brakes
left turn
right turn

12. Driver passenger
classification

13. Driver vs Passenger - Context based
Home Station Station
Hotel
Office
car
car
car
car
train
train
car
DRIVER
PASSENGER
• Time line chunks with transports between two home stationaries
• Find consistent car cycles = closed loops where each trip starts where
the previous one ended
• If home in cycle -> likely driver, if not passenger

14. Driver specific model
• Unsupervised approach utilising the derived car signals
• Patterns in signals relate to how one drives
• Learn a model specific for each user characterising her driving style
• Feature extractor must characterise driving style but invariant to road
type, traffic, weather …
• One feature extractor for all user
Sensors from many trips Feature vectors
User-specific
model
Feature
extraction

15. Feature extraction: transfer learning
• We don’t want to engineer the features -> deep learning approach
• No big labelled car data set annotated with driver or passenger
• Transfer learning: train a network for a task, use if for another
• Classify trips from 1000 users (1000 classes)
• take the output of a dense layer as feature vector = projection to
metric space

16. Feature extraction: deep learning model
User finger print:
• Acceleration, brake, turn events: short time scale
• Temporal relationship between events
Beyond 30 seconds: road layout and traffic more prominent
Data augmentation (sign, additive and multiplicative noise)
2M parameters
1D convolutions
input 128x1x3
3 channels

17. Driver vs passenger: visualising the embeddings
• Users with ground truth (not used in the DL training)
• Assumption: most of the time driver
• Passenger are at the edge of the distribution and clustered together
• Build user-specific models using those feature vectors
Blue: driver
Red: passenger
User #1 User #2
Sensors Feature vectorsNetwork

18. Driver vs passenger: anomaly detection
• Consider passenger trips as outliers: they differ a lot from the user’s average profile
• User model = isolation forest characterising the 50 dimensional feature space
• Use past trips to learn model and distribution (save user features)
precision recall support
passenger 0.66 0.84 25
driver 0.91 0.91 129
avg/total 0.92 0.90 154

19. Driver vs passenger: universal background model
• Learn a gaussian mixture model for a big population = universal background model
• Learn user model: for each new trip, update the universal model
• Compare log likelihood of both models
precision recall support
passenger 0.72 0.75 24
driver 0.96 0.95 136
avg/total 0.92 0.92 180

20. Population results and limitations
• Both approaches works best for different users
• For some users, models take longer (need more data) to stabilise
• Combine both approaches: recall=0.73 and precision=0.76 on passenger prediction
• How good is the assumption
• Number of passenger trips received at the beginning of the user model learning
• Harder if passenger trips in the same car
• Harder if driver trips in different cars
Unsupervised driver/passenger classification

21. Meet the Sentiance team

23. www.sentiance.com
© 2016