1. John K.S. Lau & Assoc Prof C.K. Tham , ECE, NUS
E-mail: {elelksj, eletck}@nus.edu.sg
Hidden Markov Models for
Abnormal Event Processing in
Transportation Data Streams
IEEE ICPADS’12 International Workshop on Scalable
Computing for Big Data Analytics (SC-BDA)

2. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Motivation
How to leverage computationally expensive model in
detecting abnormal events in real-time?
2

3. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Contributions
•Application of Hidden Markov Models (HMM) in
identifying not only hidden meaningful events but also
abnormal events. The focus in this work is on
abnormal events.
•Event-Driven Architecture (EDA) for decoupling of
HMM computations from real-time event processing.
3

4. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Event-Driven Architecture (EDA)
4

5. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Abnormal Event Processing (AEP)
5

6. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Abnormal Event Processing (AEP)
6

7. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Abnormal Event Processing (AEP)
•SELECT tmin, tmax FROM [table name] WHERE [unique
identifier]
– tmin and tmax pre-computed via HMM based on most
recent data. Normal event is within tmin <= t <= tmax.
•SELECT * FROM [stream name] HAVING duration <
tmin OR duration > tmax
– Any incoming event is flagged abnormal in real-time.
7

8. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Experimental Results
8
HMM
Find highest
probability

9. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Experimental Results
9

10. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Experimental Results
10

11. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Experimental Results
11

12. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Experimental Results
12

13. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
Future Works
•Ability to detect more meaningful hidden states such
as accident, heavy rain, flood, etc. besides just general
anomaly.
•Application of Map-Reduce in HMM computations.
13

14. Hidden Markov Models for Abnormal Event Processing in Transportation Data Streams
JKS Lau & CK Tham
THANK YOU
Questions?
14