Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Deep learning and feature extraction for time
series forecasting
Pavel Filonov
pavel.filonov@kaspersky.com
27 May 2016
Outlines
Motivation
Cyber Physical Security
Problem formulation
Anomaly detection
Time series forecasting
Artificial Neural...
Cyber Physical Security
Image from http://www.wallpaperup.com
”Pipeline” stand
Signal timeseries
Anomaly detection
Time series forecasting
Forecasting models
Auto-regression models and EMA (ARMA, ARIMA, GARCH)
Neural networks
Adaptive short term forecasting
Ada...
Neural networks for timeseries forecasting
Feed forward NN on window1
Recurrent NN
Hopfield networks
Elman networks
Long sh...
Neuron model
xi — inputs
b — bias
f — activation function
σ(t) = 1
1+e−t
tanh(t) = e2t
−1
e2t+1
f(t) = t
f(t) = H(t)
y — o...
LSTM
ft = σ(Wf · [ht−1, xt] + bf )
it = σ(Wi · [ht−1, xt] + bi)
˜Ct = tanh(WC · [ht−1, xt] + bC)
Ct = ftCt−1 + it
˜Ct
ot =...
RNN on raw data
NN topology: 722 input → 64 LSTM + Dropout(0.2) → 722 Linear
Forecast horizon: 5 minutes
Timeseries segmentation
Segmentation
Features
extractionClustering
...
signal segments
Features matrix
Clusters Sequence o...
RNN on extracted features
Let n be the number of clusters.
NN structure: n inputs → 10n LSTM → n SoftMax
Forecast horizon:...
Quasi-periodic timeseries
RNN on Quasi-periodic timeseries
NN structure:
61 → 32 LSTM+Dropout(0.2) → 64 LSTM+Dropout(0.2) → 1 Linear
Forecast horizo...
Quasi-periodic timeseries
NN structure:
61 → 32 LSTM+Dropout(0.2) → 64 LSTM+Dropout(0.2) → 1 Linear
Forecast horizon: 1 mi...
Conclusions
Picture from: http://www.simpsonscreative.co.uk/kiss-the-first-law-of-successful-copywriting/
References
http://keras.io/
https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-
56.pdf
Keras recurrent tutorial ...
Upcoming SlideShare
Loading in …5
×

Deep learning and feature extraction for time series forecasting

5,761 views

Published on

Slides for Moscow Data Science Meetup
http://www.meetup.com/Moscow-Data-Science/events/230869938/

Published in: Data & Analytics
  • Earn Up To $316/day! Easy Writing Jobs from the comfort of home! ◆◆◆ http://t.cn/AieXSfKU
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Deep learning and feature extraction for time series forecasting

  1. 1. Deep learning and feature extraction for time series forecasting Pavel Filonov pavel.filonov@kaspersky.com 27 May 2016
  2. 2. Outlines Motivation Cyber Physical Security Problem formulation Anomaly detection Time series forecasting Artificial Neural Networks Basic model RNN on raw data Feature engineering RNN on extracted features Quasi-periodic timeseries Conclusions
  3. 3. Cyber Physical Security Image from http://www.wallpaperup.com
  4. 4. ”Pipeline” stand
  5. 5. Signal timeseries
  6. 6. Anomaly detection
  7. 7. Time series forecasting
  8. 8. Forecasting models Auto-regression models and EMA (ARMA, ARIMA, GARCH) Neural networks Adaptive short term forecasting Adaptive auto-regression Adaptive model selection Adaption model composition Density forecast Quantile regression ...
  9. 9. Neural networks for timeseries forecasting Feed forward NN on window1 Recurrent NN Hopfield networks Elman networks Long short term memory2 Gated Recurrent Unit3 1 https://www.cs.cmu.edu/afs/cs/academic/class/15782- f06/slides/timeseries.pdf 2 http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 3 http://arxiv.org/pdf/1406.1078v3.pdf
  10. 10. Neuron model xi — inputs b — bias f — activation function σ(t) = 1 1+e−t tanh(t) = e2t −1 e2t+1 f(t) = t f(t) = H(t) y — output Figure: Single neuron
  11. 11. LSTM ft = σ(Wf · [ht−1, xt] + bf ) it = σ(Wi · [ht−1, xt] + bi) ˜Ct = tanh(WC · [ht−1, xt] + bC) Ct = ftCt−1 + it ˜Ct ot = σ(Wo · [ht−1, xt] + bo) ht = ot tanh(Ct) Picture from: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
  12. 12. RNN on raw data NN topology: 722 input → 64 LSTM + Dropout(0.2) → 722 Linear Forecast horizon: 5 minutes
  13. 13. Timeseries segmentation Segmentation Features extractionClustering ... signal segments Features matrix Clusters Sequence of labels
  14. 14. RNN on extracted features Let n be the number of clusters. NN structure: n inputs → 10n LSTM → n SoftMax Forecast horizon: 20 segments
  15. 15. Quasi-periodic timeseries
  16. 16. RNN on Quasi-periodic timeseries NN structure: 61 → 32 LSTM+Dropout(0.2) → 64 LSTM+Dropout(0.2) → 1 Linear Forecast horizon: 1 minute
  17. 17. Quasi-periodic timeseries NN structure: 61 → 32 LSTM+Dropout(0.2) → 64 LSTM+Dropout(0.2) → 1 Linear Forecast horizon: 1 minute
  18. 18. Conclusions Picture from: http://www.simpsonscreative.co.uk/kiss-the-first-law-of-successful-copywriting/
  19. 19. References http://keras.io/ https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015- 56.pdf Keras recurrent tutorial - https://github.com/Vict0rSch/deep learning/tree/master/keras/recu https://github.com/aurotripathy/lstm-anomaly-detect https://github.com/aurotripathy/lstm-ecg-wave-anomaly- detect http://simaaron.github.io/Estimating-rainfall-from-weather- radar-readings-using-recurrent-neural-networks/ http://danielhnyk.cz/predicting-sequences-vectors-keras- using-rnn-lstm/

×