Your SlideShare is downloading. ×
0
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System

254

Published on

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
254
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System Kan Ouivirach and Matthew N. Dailey Computer Science and Information Management Asian Institute of Technology ECTI-CON May 19-21, 2010 Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 1 / 29
  • 2. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Outline 1 Introduction 2 Human Behavior Pattern Clustering 3 Experimental Results 4 Conclusion Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 2 / 29
  • 3. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction Human behavior understanding is important for intelligent systems. Difficult due to the wide range of activities possible in any given context Figure: Reprinted from http://www.sourcesecurity.com/ A classic work by Yamato et al. who model tennis actions using hidden Markov models (HMMs) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 3 / 29
  • 4. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction Human behavior understanding is important for intelligent systems. Difficult due to the wide range of activities possible in any given context Figure: Reprinted from http://www.sourcesecurity.com/ A classic work by Yamato et al. who model tennis actions using hidden Markov models (HMMs) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 3 / 29
  • 5. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction Human behavior understanding is important for intelligent systems. Difficult due to the wide range of activities possible in any given context Figure: Reprinted from http://www.sourcesecurity.com/ A classic work by Yamato et al. who model tennis actions using hidden Markov models (HMMs) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 3 / 29
  • 6. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) To help security personnel work reliably and efficiently, filter out typical events; automatically present anomalous events to human operator. Figure: Reprinted from http://sikafutu.com/ Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 4 / 29
  • 7. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) To help security personnel work reliably and efficiently, filter out typical events; automatically present anomalous events to human operator. Figure: Reprinted from http://sikafutu.com/ Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 4 / 29
  • 8. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) To help security personnel work reliably and efficiently, filter out typical events; automatically present anomalous events to human operator. Figure: Reprinted from http://sikafutu.com/ Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 4 / 29
  • 9. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) One limitation of most of the work The number of “normal” behavior patterns need to be known beforehand. Nair and Clark (2002) use HMMs to model a common, predefined activity in a scene. Unsupervised analysis and clustering of behaviors for a variety of purposes has started to draw attentions. Li et al. (2006) cluster human gestures by constructing an affinity matrix using dynamic time warping (DTW), and apply the normalized-cut approach to cluster. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 5 / 29
  • 10. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) One limitation of most of the work The number of “normal” behavior patterns need to be known beforehand. Nair and Clark (2002) use HMMs to model a common, predefined activity in a scene. Unsupervised analysis and clustering of behaviors for a variety of purposes has started to draw attentions. Li et al. (2006) cluster human gestures by constructing an affinity matrix using dynamic time warping (DTW), and apply the normalized-cut approach to cluster. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 5 / 29
  • 11. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) One limitation of most of the work The number of “normal” behavior patterns need to be known beforehand. Nair and Clark (2002) use HMMs to model a common, predefined activity in a scene. Unsupervised analysis and clustering of behaviors for a variety of purposes has started to draw attentions. Li et al. (2006) cluster human gestures by constructing an affinity matrix using dynamic time warping (DTW), and apply the normalized-cut approach to cluster. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 5 / 29
  • 12. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) One limitation of most of the work The number of “normal” behavior patterns need to be known beforehand. Nair and Clark (2002) use HMMs to model a common, predefined activity in a scene. Unsupervised analysis and clustering of behaviors for a variety of purposes has started to draw attentions. Li et al. (2006) cluster human gestures by constructing an affinity matrix using dynamic time warping (DTW), and apply the normalized-cut approach to cluster. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 5 / 29
  • 13. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) One limitation of most of the work The number of “normal” behavior patterns need to be known beforehand. Nair and Clark (2002) use HMMs to model a common, predefined activity in a scene. Unsupervised analysis and clustering of behaviors for a variety of purposes has started to draw attentions. Li et al. (2006) cluster human gestures by constructing an affinity matrix using dynamic time warping (DTW), and apply the normalized-cut approach to cluster. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 5 / 29
  • 14. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) Some recent related works using HMMs to cluster behavior patterns Swears et al. (2008) propose hierarchical HMM-based clustering to find motion trajectories and velocities in a highway interchange scene. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 6 / 29
  • 15. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) Some recent related works using HMMs to cluster behavior patterns Swears et al. (2008) propose hierarchical HMM-based clustering to find motion trajectories and velocities in a highway interchange scene. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 6 / 29
  • 16. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 17. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 18. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 19. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 20. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 21. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 22. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Introduction (cont.) New method for clustering human behaviors in the context of video surveillance Combination of clustering and HMMs to group human behaviors Summary flow 1 After extracting sequences, we use DTW to measure the pairwise similarity between sequences. 2 Construct an agglomerative hierarchical clustering dendrogram based on the DTW similarity measure. 3 Recursively, find the optimal set of behavior clusters using HMMs. Oates et al. (2001) first proposed the idea of using the DTW with HMMs to cluster time series. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 7 / 29
  • 23. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Compared to the related works Potential to improve upon the state of the art in intelligent video surveillance applications by Bootstrapping human behavior classification and anomaly detection modules Supporting incremental HMM learning (performing statistical tests to select which cluster should be incrementally updated) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 8 / 29
  • 24. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Compared to the related works Potential to improve upon the state of the art in intelligent video surveillance applications by Bootstrapping human behavior classification and anomaly detection modules Supporting incremental HMM learning (performing statistical tests to select which cluster should be incrementally updated) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 8 / 29
  • 25. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Compared to the related works Potential to improve upon the state of the art in intelligent video surveillance applications by Bootstrapping human behavior classification and anomaly detection modules Supporting incremental HMM learning (performing statistical tests to select which cluster should be incrementally updated) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 8 / 29
  • 26. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Outline 1 Introduction 2 Human Behavior Pattern Clustering 3 Experimental Results 4 Conclusion Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 9 / 29
  • 27. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview We divide the proposed method into 2 phases. 1 Blob extraction 2 Behavior clustering Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 10 / 29
  • 28. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Extraction CCTV�camera Video Background Foreground model Background Extraction Modeling List�of�blobs Single�Blob Tracking Blob�features Vector Quantization Observation symbols Sequence Aggregation Discrete�symbol sequences Figure: Block Diagram of Blob Extraction Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 11 / 29
  • 29. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Extraction (cont.) We represent a blob at time t by the feature vector ft = xt yt st rt dxt dyt vt , where (xt , yt ) is the centroid of the blob. st is the size of the blob in pixels. rt is the aspect ratio of the blob’s bounding box. (dxt , dyt ) is the unit-normalized motion vector for the blob compared to the previous frame. vt is the blob’s speed compared to the previous frame. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 12 / 29
  • 30. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering Discrete�symbol sequences Similarity Measurement Distance matrix Agglomerative Hierarchical�Clustering Dendrogram HMM-based Hierarchical�Clustering Set�of�HMMs Figure: Block Diagram of Behavior Clustering Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 13 / 29
  • 31. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering (cont.) c����cluster�at�root 0 of�dendrogram c C���{��} 0 c����any�cluster in�C Train�a�HMM on�the�sequences in�c Is�the�HMM Replace�c�in�C No a�sufficient�model with�the�children of�the�sequences� of�c�from� in�c? DTW�dendrogram Yes Add�the�trained�HMM to�model�list�M Remove�c�from�C No Is�C�empty? Yes Figure: Processing flow of the use of HMM clustering method Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 14 / 29
  • 32. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering (cont.) How the processing flow works Root Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 15 / 29
  • 33. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering (cont.) How the processing flow works The HMM is sufficient? Root Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 15 / 29
  • 34. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering (cont.) How the processing flow works Not sufficient Root Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 15 / 29
  • 35. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering (cont.) How the processing flow works Root Child Child Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 15 / 29
  • 36. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Behavior Clustering (cont.) How the processing flow works Root The HMM is The HMM is sufficient? sufficient? Child Child Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 15 / 29
  • 37. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) The HMM is not sufficient to model the sequences When there are more than N sequences in a cluster whose per-observation log-likelihood is less than a threshold. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 16 / 29
  • 38. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) The HMM is not sufficient to model the sequences When there are more than N sequences in a cluster whose per-observation log-likelihood is less than a threshold. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 16 / 29
  • 39. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) To determine the optimal rejection threshold Use an approach similar to that of Oates et al. (2001). Generate random sequences from the HMM. Calculate µc and σc of the per-observation log-likelihood over the set of generated sequences. Let a threshold be pc = µc − zσc , where z is experimentally tuned. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 17 / 29
  • 40. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) To determine the optimal rejection threshold Use an approach similar to that of Oates et al. (2001). Generate random sequences from the HMM. Calculate µc and σc of the per-observation log-likelihood over the set of generated sequences. Let a threshold be pc = µc − zσc , where z is experimentally tuned. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 17 / 29
  • 41. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) To determine the optimal rejection threshold Use an approach similar to that of Oates et al. (2001). Generate random sequences from the HMM. Calculate µc and σc of the per-observation log-likelihood over the set of generated sequences. Let a threshold be pc = µc − zσc , where z is experimentally tuned. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 17 / 29
  • 42. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) To determine the optimal rejection threshold Use an approach similar to that of Oates et al. (2001). Generate random sequences from the HMM. Calculate µc and σc of the per-observation log-likelihood over the set of generated sequences. Let a threshold be pc = µc − zσc , where z is experimentally tuned. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 17 / 29
  • 43. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Blob Clustering (cont.) To determine the optimal rejection threshold Use an approach similar to that of Oates et al. (2001). Generate random sequences from the HMM. Calculate µc and σc of the per-observation log-likelihood over the set of generated sequences. Let a threshold be pc = µc − zσc , where z is experimentally tuned. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 17 / 29
  • 44. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Outline 1 Introduction 2 Human Behavior Pattern Clustering 3 Experimental Results 4 Conclusion Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 18 / 29
  • 45. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview Recorded videos at a resolution of 320 × 240 and 25 fps over 1 week. Used a motion detection to save disk space. Obtained videos corresponding to over 500 motion events, but selected the 298 videos containing only a single motion. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 19 / 29
  • 46. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Found that at least 4 common behaviors: Walking into the building (Walk-in) Walking out of the building (Walk-out) Parking a bicycle (Cycle-in) Riding a bicycle out (Cycle-out) Other less common activities: Walking while telephoning, etc. (Other) Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 20 / 29
  • 47. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Figure: Example of common human activities in our testbed scene. (a) Walking in. (b) Walking out. (c) Cycling in. (d) Cycling out. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 21 / 29
  • 48. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Our main hypothesis Using DTW as a pre-process prior to HMM-based clustering should improve the quality of the clusters in term of separating anomalous from typical behaviors. Compared to Using only HMMs Supervised classification with HMMs Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 22 / 29
  • 49. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Our main hypothesis Using DTW as a pre-process prior to HMM-based clustering should improve the quality of the clusters in term of separating anomalous from typical behaviors. Compared to Using only HMMs Supervised classification with HMMs Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 22 / 29
  • 50. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Our main hypothesis Using DTW as a pre-process prior to HMM-based clustering should improve the quality of the clusters in term of separating anomalous from typical behaviors. Compared to Using only HMMs Supervised classification with HMMs Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 22 / 29
  • 51. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Our main hypothesis Using DTW as a pre-process prior to HMM-based clustering should improve the quality of the clusters in term of separating anomalous from typical behaviors. Compared to Using only HMMs Supervised classification with HMMs Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 22 / 29
  • 52. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Performed 3 experiments. Evaluated the results according to how well the induced categories separate the anomalous sequences (hand-labeled with the category “Other”) from typical sequences (Walk-in, Walk-out, Cycle-in, Cycle-out). 1 Using our proposed method 2 Using only HMMs 3 Using HMMs with supervised learning In all 3 experiments, we chose linear HMMs with 4 states based on our previous empirical experience. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 23 / 29
  • 53. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Performed 3 experiments. Evaluated the results according to how well the induced categories separate the anomalous sequences (hand-labeled with the category “Other”) from typical sequences (Walk-in, Walk-out, Cycle-in, Cycle-out). 1 Using our proposed method 2 Using only HMMs 3 Using HMMs with supervised learning In all 3 experiments, we chose linear HMMs with 4 states based on our previous empirical experience. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 23 / 29
  • 54. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Performed 3 experiments. Evaluated the results according to how well the induced categories separate the anomalous sequences (hand-labeled with the category “Other”) from typical sequences (Walk-in, Walk-out, Cycle-in, Cycle-out). 1 Using our proposed method 2 Using only HMMs 3 Using HMMs with supervised learning In all 3 experiments, we chose linear HMMs with 4 states based on our previous empirical experience. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 23 / 29
  • 55. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Performed 3 experiments. Evaluated the results according to how well the induced categories separate the anomalous sequences (hand-labeled with the category “Other”) from typical sequences (Walk-in, Walk-out, Cycle-in, Cycle-out). 1 Using our proposed method 2 Using only HMMs 3 Using HMMs with supervised learning In all 3 experiments, we chose linear HMMs with 4 states based on our previous empirical experience. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 23 / 29
  • 56. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Performed 3 experiments. Evaluated the results according to how well the induced categories separate the anomalous sequences (hand-labeled with the category “Other”) from typical sequences (Walk-in, Walk-out, Cycle-in, Cycle-out). 1 Using our proposed method 2 Using only HMMs 3 Using HMMs with supervised learning In all 3 experiments, we chose linear HMMs with 4 states based on our previous empirical experience. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 23 / 29
  • 57. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Performed 3 experiments. Evaluated the results according to how well the induced categories separate the anomalous sequences (hand-labeled with the category “Other”) from typical sequences (Walk-in, Walk-out, Cycle-in, Cycle-out). 1 Using our proposed method 2 Using only HMMs 3 Using HMMs with supervised learning In all 3 experiments, we chose linear HMMs with 4 states based on our previous empirical experience. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 23 / 29
  • 58. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Overview (cont.) Our configuration the number of deviant patterns allowed in a cluster N = 10 z = 2.0 for a threshold pc = µc − zσc Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 24 / 29
  • 59. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Results for Experiment I Clustering results for Experiment I (DTW+HMMs). Cluster # Walk-in Walk-out Cycle-in Cycle-out Other 1 96 0 18 0 0 2 0 54 0 5 0 3 0 3 0 8 0 4 0 2 0 0 0 5 0 1 0 2 0 ... 14 0 0 0 0 4 15 0 0 0 0 4 16 0 0 0 0 2 17 0 0 0 0 2 One-seq clusters 4 17 34 21 4 Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 25 / 29
  • 60. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Results for Experiment II Begin by training a single HMM on all sequences. Assign every sequence with a per-observation log-likelihood above a threshold pc to a cluster. Repeat the process by training a new HMM on the remaining sequences. Clustering results for Experiment II (HMMs only). Cluster # Walk-in Walk-out Cycle-in Cycle-out Other 1 15 77 49 43 16 2 80 0 11 2 0 3 5 0 0 0 0 Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 26 / 29
  • 61. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Results for Experiment III Trained 4 HMMs on each of the four typical beahviors. Maximize the F1 value to determine the best per-observation log-likelihood threshold for each HMM. For the best separation between the positive and negative test patterns Results for Experiment III (Supervised classification with HMMs). Anomaly detection rate (%) False alarm rate (%) 50 24.6 Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 27 / 29
  • 62. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Outline 1 Introduction 2 Human Behavior Pattern Clustering 3 Experimental Results 4 Conclusion Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 28 / 29
  • 63. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Conclusion We have proposed and evaluated a new method for clustering human behaviors. Our method provides an initial partitioning of a set of behavior sequences, then automatically identifies where to cut off the hierarchical clustering dendrogram. could be used to bootstrap an anomaly detection module for intelligent video surveillance applications. shows a perfect separation between typical and anomalous behaviors on real-world surveillance data without any information about the labels. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 29 / 29
  • 64. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Conclusion We have proposed and evaluated a new method for clustering human behaviors. Our method provides an initial partitioning of a set of behavior sequences, then automatically identifies where to cut off the hierarchical clustering dendrogram. could be used to bootstrap an anomaly detection module for intelligent video surveillance applications. shows a perfect separation between typical and anomalous behaviors on real-world surveillance data without any information about the labels. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 29 / 29
  • 65. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Conclusion We have proposed and evaluated a new method for clustering human behaviors. Our method provides an initial partitioning of a set of behavior sequences, then automatically identifies where to cut off the hierarchical clustering dendrogram. could be used to bootstrap an anomaly detection module for intelligent video surveillance applications. shows a perfect separation between typical and anomalous behaviors on real-world surveillance data without any information about the labels. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 29 / 29
  • 66. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Conclusion We have proposed and evaluated a new method for clustering human behaviors. Our method provides an initial partitioning of a set of behavior sequences, then automatically identifies where to cut off the hierarchical clustering dendrogram. could be used to bootstrap an anomaly detection module for intelligent video surveillance applications. shows a perfect separation between typical and anomalous behaviors on real-world surveillance data without any information about the labels. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 29 / 29
  • 67. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion Conclusion We have proposed and evaluated a new method for clustering human behaviors. Our method provides an initial partitioning of a set of behavior sequences, then automatically identifies where to cut off the hierarchical clustering dendrogram. could be used to bootstrap an anomaly detection module for intelligent video surveillance applications. shows a perfect separation between typical and anomalous behaviors on real-world surveillance data without any information about the labels. Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System 29 / 29

×