Sensors are everywhere, and context detection can improve almost any service, from media recommendations and advertising, to mobility management and mHealth. These two use cases show what mobile sensor data and machine learning can do to help make technologies more 'smart' and context-aware. Presented at Strata Hadoop Barcelona, Internet of Things track, November 2014

1. Mobile Sensor Data, Machine Learning
and Context
Twitter: @arguslabs
© 2014 – info@arguslabs.com

2. “ Sense, understand and predict the context, behavior
and mood of your mobile audience ”

3. Argus has created a platform that enables
you to build context-aware solutions

4. Turning sensor data into behavioral, context,
and emotional awareness
ARGUS MOBILE SENSING
SD
K
ARGUS
PLATFORM
…
LVL 3
PROFILE
S
LVL 2
MOMENTS
LVL 1
EVENTS

5. Mobile Profiling
EVENTS
Sense and interpret the
contextual cues of a mobile user
MOMENTS
Uncover habits and predict
human behavior
PROFILES
Learn about the ever-changing
personalities of a
mobile user
LAYER 3
ACTIVENESS
DRIVING STYLE BEHAVIOUR
LAYER 2
ARRIVING AT HOME, WORK, ..
LAYER 1
…
WAKING UP …
…
SOCIALNESS
SLEEPING
IN A MEETING
DRIVING (CAR) TRAIN SUBWAY
WALKING RUNNING
BUS
SITTING STANDING
TRAM MOTORCYCLE
AIRPLANE BIKING
BUSY
BORED TIRED
LOUD ENVIRONMENT
ALONE
HOME
WORK
COMPANY
COMMUTING TYPE
CHATTY
CALM
DRIVING
BEHAVIOUR
FOR EXAMPLE PROFILES

6. ARGUS
About Argus Labs
Research Development Architecture Sales

7. ARGUS
YOU @ ARGUS
We are looking to expand our machine learning and data
research department. Email: vincent.spruyt@arguslabs.com, if
you..
1. Want to work with state-of-the-art machine learning
2. Find the use cases I’ll present on music and mobility
fascinating and want to work on this with us.
3. Most definitely, if you feel you can improve upon what we did
in these two use cases, or can suggest a better approach

8. FLEET &
MOBILITY
Two case studies
Detecting
transport types
based on low
level sensor data
MUSIC &
MOOD
Estimating a
user’s mood
based on
acoustic features

9. Transport type detection
1. Time series data: accelerometer, gyroscope
2. Categorical enrichment: road type, train stations, etc.
3. Missing/partially observable data: GPS locations
4. Small data!
HOW DO WE SOLVE THIS?

10. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction

11. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction

12. Prediction pipeline: pre-processing
1. Remove noise: low-pass filter
2. Isolate signal components: band-pass filter
3. Resample and interpolate
4. Sanity checks: sampling rate, sequence length, etc.

13. Prediction pipeline: pre-processing
1. Rotation invariance: PCA
2. Scale invariance: Whitening
3. Decorrelation and ICA

14. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction

15. Prediction pipeline: Feature calculation
1. Periodicity and rhythm
- Autocorrelation, beats, zero-crossings, etc.
2. Timbre
- Spectral envelope
1. Pitch
- fundamental frequencies and harmonics
2. Spectral Flux
- Temporal spectral behavior
3. Loudness
- Power/RMS
…

16. Prediction pipeline: Feature calculation
Deep Learning
- Convolutional neural network
- 1D convolutions across frequency axis!
- Max-pooling and dropout
=> avoid the curse of dimensionality
- Automatically discovers important non-linearities
- Disadvantage: needs lots of training data!

17. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction

18. Prediction pipeline: Data abstraction
Huge input dimensionality
- E.g. 6D input data (accelerometer and gyroscope) @50Hz
- 5-second fragments: 1500D!
Huge feature space dimensionality
- ± 500D for each 5-second fragment
Dimensionality reduction needed!
- Traditional methods are unsupervised:
Kernel PCA, SOM, IsoMAP, Spectral clustering, etc.

19. Prediction pipeline: Data abstraction

20. Prediction pipeline: Data abstraction
Idea:
- Learn non-linear abstraction in a supervised manner
- E.g. Random Forest, or deep CNN
- RF output: class probabilities
- Use these as input features for a temporal classifier

21. Transport type detection
Our general prediction pipeline
Pre-processing Feature calculation Data abstraction
Post-processing Temporal smoothing Temporal prediction

22. Prediction pipeline: Temporal prediction
Goal:
- Learn temporal correlations between input data (co-adaptations)
- Cope with missing or partially observable data
Constraints:
- Small training dataset!
- For some features more than 70% missing data
- Simple imputation techniques won’t work!

23. Prediction pipeline: Temporal prediction

24. Prediction pipeline: Temporal prediction

25. Prediction pipeline: Temporal prediction

26. Prediction pipeline: Temporal prediction
BUT…

27. Prediction pipeline: Results
Random Forest predictions
% Biking Bus Car Idle Train Tram Walking
Biking 66 0 16 0 1 0 17
Bus 0 54 0 4 13 19 10
Car 1 5 75 8 2 7 2
Idle 0 0 0 99 1 0 0
Train 1 7 0 16 61 13 3
Tram 2 6 8 14 11 59 1
Walking 1 4 0 2 4 0 89

28. Prediction pipeline: Results
DBN predictions (without location)
% Biking Bus Car Idle Train Tram Walking
Biking 89 0 1 0 0 0 10
Bus 0 60 0 0 26 13 0
Car 0 7 81 2 0 10 0
Idle 0 0 0 100 0 0 0
Train 0 2 0 5 91 2 0
Tram 0 3 2 0 11 84 0
Walking 0 4 0 2 0 0 94

29. Prediction pipeline: Results
DBN predictions (with location)
% Biking Bus Car Idle Train Tram Walking
Biking 87 0 12 0 0 0 1
Bus 0 75 0 0 10 14 0
Car 0 4 89 1 1 6 0
Idle 0 0 0 98 2 0 0
Train 0 2 0 2 95 1 0
Tram 0 3 0 0 7 90 0
Walking 0 1 0 5 0 0 95

30. Automatic tracking of
automotive journeys
• Start and stop time
• Traveled distance
• Time and duration
• Way points
Contextual driver
profiles through
clustering techniques
• Long term driver
profile classifications
• Real time anomaly
detections
Reliable
differentiation
between multiple
cars used
(Bluetooth, frequency,
charger, USB, …)
Back-end SAAS
platform providing
extensive API, reports
and dashboard
Track changes for
individuals, groups and
vehicles

31. EXTERNAL CONTEXTUAL INFLUENCERS
WEATHER
TIME OF
DAY
BASE EVENTS
ROAD
TYPES
TRAFFIC
BRAKE STOP LANE CHANGE TURN ACCELERATE
BEHAVIOURAL INFLUENCERS (OPTIONAL)
SPEED
LIMITS
STOP
LIGHTS
VEHICLE
TYPES
VEHICLE
CONDITION
Examples of human behaviour and mindset that we can take into account are phone interaction and usage,
alertness and stress, schedule, amount of time slept, ..

32. LEFT LANE
DRIVER
These drivers
consistently
opt for the fast
lane.
ZEN
DRIVER
Courteous and
calm. Nobody
more pleasant
to encounter
on the road.
THE
TAILGATER
Let us hope
the person in
front does not
decide to hit
the breaks.
ASOCIAL
DRIVER
Familiar to all
of us, these
drivers that do
not realize
there are
others on the
road as well.
LANE
SWITCHER
Left, right,
left.. . Then
right seems
faster. Maybe
middle lane
now?
10%
DRIVER
At least
there’s
consistency in
their
speeding. An
average 10%
above the
limit.

33. FLEET &
MOBILITY
Two case studies
Detecting
transport types
based on low
level sensor data
MUSIC &
MOOD
Estimating a
user’s mood
based on
acoustic features

34. Music is Emotion
= :-)
or :-( or ^^ or -.- or (°_°) or …

35. Music & Mood
AMPLITUDE PITCH MELODY TEMPO RHYTHM

36. Music & Mood
Ascending higher-pitch sequences vs Descending lower-pitch sequences
:-)
:-(

37. Music & Mood
RESEARCH QUESTIONS
Are emotions encapsulated in a raw music signal?
How can we automatically label millions of songs?

38. Music & Mood
VALENCE
AROUSAL
ACTIVATION
Stressed
ANGRY HAPPY
UNPLEASANT PLEASANT
RELAXED
DEACTIVATION
Upset
Tense Excited
Clated
Serene
Calm
Fatigued
Depressed
SAD

39. Music & Mood
DEMO

40. Music & Mood
Research questions:
1. Are emotions encapsulated in a raw music signal?
2. How can we automatically label millions of songs?

41. Music & Mood
Transfer learning:
1. For 200 songs, we have per-second valence/arousal data
-> Learned a prediction model based on this
2. For 1 million songs, we only have meta-data
-> Tags (e.g. ‘happy’, ‘sleepy’, ‘metal’, ‘super’, ‘cool’)
3. For 100 songs, we have both!
-> Transfer knowledge from 1 to 2 using LSA

42. Music & Mood
Latent Semantic Analysis:

43. Music & Mood
Transfer knowledge:
1. For each of the 1 million song
1. Find KNN of the 200 songs in latent space (cosine distance)
2. Interpolate
DEMO

44. Conclusion
1. Sensors are everywhere!
2. Context can improve almost any service, e.g.
1. Media recommendation
2. Insurance: driving behavior
3. Fleet and mobility
4. Advertising
3. We are hiring the best!
2. Data scientists and machine learning specialists
3. Big data analysts and architects