"Toeplitz Inverse Covariance-Based Clustering of Multivariate Time Series Data" the paper of KDD 2017

1. 1 KYOTO UNIVERSITY
KYOTO UNIVERSITY
Toeplitz Inverse Covariance-Based Clustering of
Multivariate Time Series Data
Daiki Tanaka
Kashima lab., Kyoto University
Paper Reading Seminar, 2018/4/20(Fri)

2. 2 KYOTO UNIVERSITY
Today’s paper:
Paper in KDD ‘17
n Title : Toeplitz Inverse Covariance-Based Clustering
of Multivariate Time Series Data (KDD ‘17)
n テプリッツ逆共分散行列ベースの多変量時系列データ
のクラスタリング
n Authors:
l David Hallac
l Sagar Vare
l Stephan Boyd
l Jure Leskovec Stanford University

3. 3 KYOTO UNIVERSITY
Back ground[1/2]:
Motivating Example
p Many applications generate large amount of multivariate time-series
data.
p These time-series data can be broken down into a sequence of states.
Multiple entities

4. 4 KYOTO UNIVERSITY
Back ground [2/2]:
p To achieve this, it is necessary to simultaneously segment and cluster
the time-series.
p Traditional clustering methods are not particularly well-suited to
discover interpretable structure in the data. This is because they
typically rely on distance-based metrics, such as dynamic time
warping.
p Their method is the first to perform time series clustering based on
the graphical dependency structure of each subsequence.

5. 5 KYOTO UNIVERSITY
Problem setting

6. 6 KYOTO UNIVERSITY
Problem setting [1/3]:
Clustering time-series data
p A time series of T sequential observations !"#$% ∈ ℝ(×*
:
p Goal is to cluster these T observations into K clusters.
p Instead of clustering each +$ in isolation, they cluster a short
subsequence of size w ≪ . going from time / − 1 + 1 to /.
p Subsequence : 45 = {+589:;, … , +5} ∈ ℝ*9
n sensor data

7. 7 KYOTO UNIVERSITY
Problem setting[2/3]:
Input & Output
p Input
l !", !$, … , !& : subsequence
l Subsequence : !' = )'*+,", … , )'
p Output
l Clusters’ structure : -
l Clustering assignment result : .
….
!/
!0
!&
w
w
w

8. 8 KYOTO UNIVERSITY
Proposed method

9. 9 KYOTO UNIVERSITY
Proposed method[1/6]:
Toeplitz Inverse Covariance-Based Clustering (TICC)
p Define each cluster as a multilyer Markov random field
by a Gaussian inverse covariance Θ" ∈ ℝ%&×%&
.
p Inverse covariances show the conditional independency structure between the
variables. So, Θ" defines a Markov random field encoding the structural
representation of cluster (.
p Our objective is to solve for :
l Inverse covariances : ) = {Θ,, … , Θ/}
l Cluster assignment sets : 1 = {2,, … , 2/}
Ø 2" ⊂ {1,2, … , 6}
where K is the number of clusters.
t t+1 t+2 t t+1 t+2
Sensor data
MRF

10. 10 KYOTO UNIVERSITY
Proposed method [2/6]:
constraint on Θ to be block Toeplitz matrices
p Θ" ‘ s constraint :
Ø #(")
∈ ℝ(×(
p #"*
(+)
shows how sensor , at some time t is correlated to
sensor j at time - + /
0×0

11. 11 KYOTO UNIVERSITY
Proposed method [3/6]:
written as an optimization problem
p Optimization problem is :
p It is difficult to solve this directly, so we use EM-like algorithm.
L1-norm penalty
! "# $%&' ∉ )*
0("# $%&' ∈ )*)
/ and ! are regularization parameters
3* is empirical mean of cluster ".

12. 12 KYOTO UNIVERSITY
Proposed method [4/6]:
Overview of TICC
E-step → M-step → E-step → M-step → … until Stationary

13. 13 KYOTO UNIVERSITY
Proposed method [5/6]:
E-step (Assigning Points to Clusters)
p Fixing ! and solving the following combinatorial
optimization problem for " = {%&, … , %)},
p This problem can be solved by using dynamic programming.
Original optimization problem

14. 14 KYOTO UNIVERSITY
Proposed method [6/6]:
M-step (Toeplitz Graphical Lasso)
p Fixing the point assignments !, the next task is to update
the cluster parameters " = {Θ&, . . . , Θ)}.
p Negative log likelihood can be rewritten as :
p So, M-step of their EM algorithm is : (This is well-known graphical lasso)

15. 15 KYOTO UNIVERSITY
Experiment

16. 16 KYOTO UNIVERSITY
Experiment:
Performance of the proposed method is evaluated by using synthetic data
p Synthetic multivariate data
l Randomly generated data in ℝ"
l They generate a random ground truth Toeplitz inverse
covariance.

17. 17 KYOTO UNIVERSITY
Experiment :
They use following methods as baselines.
p TICC, ! = 0 − Each subsequence is assigned to a cluster
independently of its location in time series.
p GMM − Clustering using a Gaussian Mixture Model
p EEV − Regularized GMM with shape and volume constraints on the
Gaussian covariance matrix
p DTW, GAK − Dynamic time warping(DTW) based clustering using GAK
p DTW, Euclidean − DTW using a Euclidean distance metric
p Neural Gas − Artificial neural network clustering
p K-means − K-means clustering using Euclidean distance

18. 18 KYOTO UNIVERSITY
Experiment :
experiment setup
p They evaluate performance by clustering each point in the
time series and comparing to the ground truth clusters.
p The number of clusters K : given to be the true number
p Evaluation metrics : macro – F1
F1 =
2Recall×Precision
Recall + Precision

19. 19 KYOTO UNIVERSITY
Experiment : Result[1/2]
Proposed Method(TICC) outperforms baselines.

20. 20 KYOTO UNIVERSITY
Experiment : Result[2/2]
TICC needs fewer samples than the other methods to achieve similar performance.
Proposed method
better

21. 21 KYOTO UNIVERSITY
Case Study

22. 22 KYOTO UNIVERSITY
p Dataset
l 36,000 observations of a 7-dimensional time series
Ø Brake Pedal Position
Ø Forward (X-)Acceleration, Lateral (-Y)Acceleration
Ø Vehicle Velocity
Ø Engine RPM
Ø Steering Wheel Angle
Ø Gas Pedal Position
p Window size = 10 samples.
p Number of clusters = 5 (defined by using BIC.)
Case Study:
Applying TICC to real driving dataset.

23. 23 KYOTO UNIVERSITY
Case Study: Result
TICC seems to cluster real world time-series data well.

24. 24 KYOTO UNIVERSITY
p They have defined a method of clustering multivariate
time-series subsequences(TICC).
p TICC simultaneously segments and clusters the time-series data more
accurately than baseline methods.
Conclusion :
They made TICC, and it seems nice.