Anomaly Detection in High-Dimensional Time Series with Coarse-to-Fine Model Transfer

1. CTF: Anomaly Detection in High-Dimensional
Time Series with Coarse-to-Fine Model Transfer
Ming Sun, Ya Su, Shenglin Zhang, Yuanpu Cao, Yuqing Liu, Dan Pei,
Wenfei Wu, Yongsu Zhang, Xiaozhou Liu, Junliang Tang
INFOCOM 2021

2. Outline
Background Design Evaluation Conclusion
2

3. Outline
Background Design Evaluation Conclusion
3

4. DL Algorithms in the Infra Operation
4
• Advantages
– automation
– robustness
– Saving operator’s labor
• Example:
– RNN-VAE for anomaly detection

5. RNN-VAE Based Algorithms
5
Network architecture of RNN-VAE
models at time t
𝒙𝒕 (49) -> 𝒛𝒕 (3) -> 𝒙𝒕
%
(49)
RNN
Dense
layers
"# $# "#
%
RNN
Dense
layers
Variational Auto-Encoder (VAE)
KPI dimension reduced
Network Layers
• RNN: Shallow & general
• Dense layers: Deep & specific

6. Scalability is the problem for large scale
6
• High-Dimensional Data
– Machines: in millions
– KPI: in tens
– Time: Frequent data query (2880 samples/day)
Ø One model per machine: time
10X minutes * 1X million machines
Ø One model for all: accuracy

7. Scalability is the problem for large scale
7
• High-Dimensional Data
– Machines: in millions
– KPI: in tens
– Time: Frequent data query (2880 samples/day)
Goal: devise scalable deep learning (DL) algorithms for
large-scale anomaly detection

8. 8
• Intuition: Cluster Machines first, then run DL for each cluster
• Challenge 1: clustering model training
• Clustering cannot run on high-dimensional data
• DL cannot run on whole dataset without clustering
• Solution: Synthetic framework
Intuition and Challenges
dependency
Coarse-grained model -> clustering -> fine-grained models

9. 9
• Intuition: Cluster Machines first, then run DL for each cluster
• Challenge 1: clustering model training
• Clustering cannot run on high-dimensional data
• DL cannot run on whole dataset without clustering
• Solution: Synthetic framework
• Challenge 2: High dimension of time domain
• Hard to cluster even KPI is compressed
• Solution: compress sequence to z-distribution
Intuition and Challenges
dependency

10. 10
• Intuition: Cluster Machines first, then run DL for each cluster
• Challenge 1: clustering model training
• Clustering cannot run on high-dimensional data
• DL cannot run on whole dataset without clustering
• Solution: Synthetic framework
• Challenge 2: High dimension of time domain
• Hard to cluster even KPI is compressed
• Solution: compress sequence to z-distribution
• Challenge 3: Neural network training method
• Solution: fine-tuning strategy
• Freeze RNN and tune dense layers
Intuition and Challenges
dependency
RNN
Dense
layers
"# $# "#
%
RNN
Dense
layers

11. Outline
Background Design Evaluation Conclusion
11

12. Framework of model training
12
Framework of model training
:0 : 4 4 2 30 . : 0
2: 4 0 0
0 :0 0 : . 4
.34 0 . 0:4 2
4 :4- 4 1 .34 0
0
0 : 10: 14 0 4 2
14 0 2: 4 0 0 0: . 0:
. 0:
(M)
0
0 0 (K<<M)
0 1: : 4 .34 0

13. Framework of model training
13
Framework of model training
:0 : 4 4 2 30 . : 0
2: 4 0 0
0 :0 0 : . 4
.34 0 . 0:4 2
4 :4- 4 1 .34 0
0
0 : 10: 14 0 4 2
14 0 2: 4 0 0 0: . 0:
. 0:
(M)
0
0 0 (K<<M)
0 1: : 4 .34 0 • Sampling strategy:
• Machine sampling
• Time sampling

14. Framework of model training
14
Framework of model training
:0 : 4 4 2 30 . : 0
2: 4 0 0
0 :0 0 : . 4
.34 0 . 0:4 2
4 :4- 4 1 .34 0
0
0 : 10: 14 0 4 2
14 0 2: 4 0 0 0: . 0:
. 0:
(M)
0
0 0 (K<<M)
0 1: : 4 .34 0
𝒙𝒕 sequence
𝒛𝒕 sequence
𝒛𝒕 distribution

15. Framework of model training
15
Framework of model training
:0 : 4 4 2 30 . : 0
2: 4 0 0
0 :0 0 : . 4
.34 0 . 0:4 2
4 :4- 4 1 .34 0
0
0 : 10: 14 0 4 2
14 0 2: 4 0 0 0: . 0:
. 0:
(M)
0
0 0 (K<<M)
0 1: : 4 .34 0 𝒛𝒕 distribution
distance matrix
clustering results
Wasserstein distance
HAC algorithm

16. Framework of model training
16
Framework of model training
:0 : 4 4 2 30 . : 0
2: 4 0 0
0 :0 0 : . 4
.34 0 . 0:4 2
4 :4- 4 1 .34 0
0
0 : 10: 14 0 4 2
14 0 2: 4 0 0 0: . 0:
. 0:
(M)
0
0 0 (K<<M)
0 1: : 4 .34 0
RNN
Dense
layers
"# $# "#
%
RNN
Dense
layers
• Fine-tuning strategy:
• RNN: fixed
• Dense layers: tuned

17. System architecture
17
System architecture
Data API
Online Anomaly
Detection ( IV-C)
Offline Data
Online Data
Offline Model
Training ( IV-B)
Model Score
Outlier Alerting
( V-D)
Results &
Visualization
Data Preprocessing
( IV-A)
Monitored
machine entities
1. Data preprocessing
2. Offline model training
3. Online anomaly detection

18. Labeling tools
18
The interface of the labeling tool

19. Outline
Background Design Evaluation Conclusion
19

20. Dataset & performance metrics
20
• Dataset:
– # Machine entities: 533
– Dimension of each machine entity: 49 KPIs x 37440 time
points (frequency: 30s, 13 days)
– Training = first 5 days, Testing = last 8 days
• Metrics:
– F1, Precision, Recall: average of all machine entities.
– Model training time

21. Overall performance
• Scalability
– Pre-training: fixed (5493s)
21
The execution time of each step under different
numbers of machine entities
F1, Precision, and Recall scores of CTF without
and with alerting

22. Overall performance
• Scalability
– Pre-training: fixed (5493s)
– feature extraction: 0.3s /
machine
22
The execution time of each step under different
numbers of machine entities
F1, Precision, and Recall scores of CTF without
and with alerting

23. Overall performance
• Scalability
– Pre-training: fixed (5493s)
– feature extraction: 0.3s /
machine
– Clustering: much smaller
– Fine-tuning: 448s / model
23
The execution time of each step under different
numbers of machine entities
F1, Precision, and Recall scores of CTF without
and with alerting

24. Overall performance
• Scalability
– Pre-training: fixed (5493s)
– feature extraction: 0.3s /
machine
– Clustering: much smaller
– Fine-tuning: 448s / model
• Effectiveness
– F1: 0.830->0.892
24
The execution time of each step under different
numbers of machine entities
F1, Precision, and Recall scores of CTF without
and with alerting

25. Overall performance
• Validating the Synthetic
Framework
– One model/machine
– One model for all
– CTF w/o transfer
25
Comparison with model variations
F1 and training time under different numbers of
epochs for CTF w/o transfer

26. Overall performance
• Validating the Synthetic
Framework
– One model/machine
– One model for all
– CTF w/o transfer
26
Comparison with model variations
F1 and training time under different numbers of
epochs for CTF w/o transfer

27. Overall performance
• Validating the Synthetic
Framework
– One model/machine
– One model for all
– CTF w/o transfer
27
Comparison with model variations
F1 and training time under different numbers of
epochs for CTF w/o transfer

28. Overall performance
• Validating the Synthetic
Framework
– One model/machine
– One model for all
– CTF w/o transfer
28
Comparison with model variations
F1 and training time under different numbers of
epochs for CTF w/o transfer

29. Validating Design Choices
• Choice of Clustering Objects
– SPF, ROCKA, DCN
• Choice of Distance Measures
– KL divergence, JS divergence,
mean squared error
• Choice of Clustering Algorithms
– DBSCAN, K-medoids
29

30. Outline
Background Design Evaluation Conclusion
30

31. Conclusion
• CTF: synthetic framework, high-dimensional time series
(machine, KPI, time)
• Techniques: 𝒛𝒕 distribution clustering, model reuse, fine-tuning
• Evaluation: CTF scalability and effectiveness
• Labeling tool + labeled dataset
31

32. Thank you!
Q & A
sunm19@mails.tsinghua.edu.cn
INFOCOM 2021
32