Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Hybrid Neural Networks for
Time Series Learning
Tian Guo
Ph.D.
24 Nov. 2016
Outline
• Introduction
• Hybrid Neural Network (HNN)
• HNN for Real
• Preliminaries
• TreNet: a HNN for learning the local...
Introduction
3Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Introduction
• Time series data: a sequence of data points consisting of
successive measurements made over time.
• Interne...
Introduction
• Time series analytics in a variety of applications
• Classification
• Prediction
• Anomaly detection
• Patt...
Introduction
• Time series analytics tools
• Statistics
• Hidden Markov Model (HMM)
• State Space Model
• ARIMA
• Machine ...
Introduction
• Neural Network and Deep Learning
• Language translation
• E.g., Google’s Multilingual Neural Machine Transl...
Introduction
• Fundamental network architectures
• Convolutional neural network: input two-dimensional data, e.g.,
image
•...
Introduction
• Convolutional Neural Network (CNN)
• Feature learning for images
• To extract high-level features from raw ...
Introduction
• Convolutional Neural Network (CNN)
• CNN in the Human Activity Recognition Problem
• Multichannel time seri...
Introduction
• Recurrent Neural Network (RNN)
• A powerful tool to model sequence data
• To capture dependency in sequence...
Hybrid Neural Networks
12Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Hybrid Neural Networks
• CNN or RNN
• Work well for respective data, i.e. images and sequence data
13Introduction Hybrid N...
Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
• Electroencephalogram (EEG) : multiple tim...
Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
• A key challenge in correctly recognizing ...
Hybrid Neural Networks
• A cascade of CNN and RNN
• Classification of EEG data
16
P. Bashivan, et. al. “Learning Represent...
Hybrid Neural Networks
• A cascade of CNN and RNN
• EEG data classification
17
P. Bashivan, et. al. “Learning Representati...
Hybrid Neural Networks
• Time Series
• Noisy
• Non-stationary
• Hidden information: states, dynamics
• Auto-correlated on ...
Hybrid Neural Networks
• Hybrid architectures: end-to-end learning framework
• Loss function driven training
• Learning re...
Hybrid Neural Networks
• A cascade of CNN and RNN
20
P. Bashivan, et. al. “Learning Representations from EEG with Deep Rec...
HNN for Real:
TreNet for Learning the Local Trend
21
T. Guo, et al. TreNet: Hybrid Neural Networks for Learning the Local ...
Preliminaries
• Conventional trend analysis in time series
is the seasonal component at time t
is the trend component at t...
Preliminaries
• Local trend
• Measure the intermediate local behaviour, i.e. upward or downward
pattern of time series
• F...
Preliminaries
• Learning and forecasting the local trend
• Predict ,
• Useful in many applications
• Smart energy
• Stock ...
Preliminaries
• Learning and forecasting the local trend
• Local raw data
• Global contextual information
in the historica...
TreNet
• Overview of TreNet
• is derived by training the LSTM over sequence to
capture the dependency in the trend evolvin...
is derived by training the LSTM over sequence to capture
the dependency in the trend evolving.
corresponds to local featur...
TreNet
• Learning
28Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• LSTM
• Feed the sequence of
• Output
29Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• CNN
• Output
30Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• Feature Fusion and output layers
31Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
TreNet
• Learning
• Gradient descent
32Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Experiments
• Datasets
• Daily Household Power Consumption (HousePC)
• Gas Sensor (GasSensor)
• Stock Transaction (Stock)
...
Experiments
• Baselines
• CNN
• LSTM
• Support Vector Regression (SVR)
• Radial Basis kernel (SVRBF)
• Polynomial kernel (...
Experiments
• Results: overall accuracy
35Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Experiments
• Results: prediction visualization
36Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
Conclusion
37
Conclusion
38
• Hybrid neural networks
• TreNet for the Local Trend Learning
• Future work: a generic idea
• Social media ...
Thanks! Q & A
This work is supported by EU OpenIoT Project
(Open Source Solution for the Internet of Things)
http://www.op...
Upcoming SlideShare
Loading in …5
×

Hybrid neural networks for time series learning by Tian Guo, EPFL, Switzerland

2,069 views

Published on

Time series is prevalent in the IoT environment and used for monitoring the evolving behavior of involved entities or objects over time. Analyzing and mining such time series data serve for revealing insightful long-term and instantaneous information behind the data, e.g., trend, event, correlation and causality and so on.

Inspired by the recent successes of neural networks, in this talk we present a novel end-to-end hybrid neural network for learning the local and global contextual features of time series. In particular, we explore the idea of hybrid neural networks in a specific time series learning problem, namely learning the local trend of time series. Local trends of time series characterize the intermediate upward and downward patterns of time series. Learning and forecasting the local trend in time series data play an important role in many real applications, ranging from investing in the stock market, resource allocation in data centers and load schedule in the smart grid. We propose TreNet, a hybrid neural network which leverages convolutional neural networks (CNNs) to extract salient features from local raw data of time series and a long-short term memory recurrent neural network (LSTM) to capture such dependency of local trends. Preliminary experimental results on real datasets demonstrate the superiority of TreNet over conventional CNN, LSTM, HMM method and various kernel based baselines.

Published in: Data & Analytics

Hybrid neural networks for time series learning by Tian Guo, EPFL, Switzerland

  1. 1. Hybrid Neural Networks for Time Series Learning Tian Guo Ph.D. 24 Nov. 2016
  2. 2. Outline • Introduction • Hybrid Neural Network (HNN) • HNN for Real • Preliminaries • TreNet: a HNN for learning the local trend of time series • Experiment results • Conclusion 2
  3. 3. Introduction 3Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  4. 4. Introduction • Time series data: a sequence of data points consisting of successive measurements made over time. • Internet of Things • Sensor networks • Mobile phones • And more… 4Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  5. 5. Introduction • Time series analytics in a variety of applications • Classification • Prediction • Anomaly detection • Pattern discovery • And more… 5 Pattern 1 Pattern 2 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  6. 6. Introduction • Time series analytics tools • Statistics • Hidden Markov Model (HMM) • State Space Model • ARIMA • Machine learning • Random Forest • SVM • Gaussian Process • 6 Random Forest Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  7. 7. Introduction • Neural Network and Deep Learning • Language translation • E.g., Google’s Multilingual Neural Machine Translation System • Computer vision • E.g., Microsoft Research’s PReLU network outperforms Human-Level performance on ImageNet Classification • Speech recognition • E.g., Amazon Echo, Apple Siri • Time series classification • E.g. recognize patterns in multivariate time series of clinical measurements 7 Z. Lipton, et al. “Learning to Diagnose with LSTM Recurrent Neural Networks”. ICLR, 2016. Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  8. 8. Introduction • Fundamental network architectures • Convolutional neural network: input two-dimensional data, e.g., image • Recurrent neural network: input 8 Unfolded recurrent connections in a RNN Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  9. 9. Introduction • Convolutional Neural Network (CNN) • Feature learning for images • To extract high-level features from raw data • Such high-level features are further used for classification or regression. 9 K. He, et al. “Delving deep into rectifiers: Surpassing Human-Level Performance on ImageNet Classification”. arxiv.org, 2015 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  10. 10. Introduction • Convolutional Neural Network (CNN) • CNN in the Human Activity Recognition Problem • Multichannel time series acquired from a set of body-worn sensors • To predict human activities by training a CNN over time series 10 J. Yang, et al. “Deep Convolutional Neural Networks On Multichannel Time Series For Human Activity Recognition”. IJCAI 2015 Value Time Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  11. 11. Introduction • Recurrent Neural Network (RNN) • A powerful tool to model sequence data • To capture dependency in sequence data • Long-short term memory network (LSTM) • A widely used variant of RNNs • Equipped with memory and gate mechanism • To overcome gradient vanishing and explosion 11 S. Hochreiter, et al. “Long short-term memory’’. Neural computation, 1997. Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  12. 12. Hybrid Neural Networks 12Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  13. 13. Hybrid Neural Networks • CNN or RNN • Work well for respective data, i.e. images and sequence data 13Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  14. 14. Hybrid Neural Networks • A cascade of CNN and RNN • Classification of EEG data • Electroencephalogram (EEG) : multiple time series corresponding to measurements across different spatial locations over the cortex. • Mental load classification task: measures the working memory responsible for transient retention of information in the brain. 14 P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  15. 15. Hybrid Neural Networks • A cascade of CNN and RNN • Classification of EEG data • A key challenge in correctly recognizing mental states • EEG data often contains translation and deformation of signal in space, frequency, and time, due to inter- and intra-subject differences 15 P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  16. 16. Hybrid Neural Networks • A cascade of CNN and RNN • Classification of EEG data 16 P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  17. 17. Hybrid Neural Networks • A cascade of CNN and RNN • EEG data classification 17 P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  18. 18. Hybrid Neural Networks • Time Series • Noisy • Non-stationary • Hidden information: states, dynamics • Auto-correlated on the temporal dimension • Manual feature engineering • Preprocessing: de-trending, outlier removal, etc. • Dimension reduction: Fourier Transformation • Piecewise approximation: PAA, PCA, PLA, etc. • Application-specific, domain knowledge 18 Why do we need hybrid architectures? Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  19. 19. Hybrid Neural Networks • Hybrid architectures: end-to-end learning framework • Loss function driven training • Learning representative features • Capturing sequential dependency in data 19 Why do we need hybrid architectures? Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  20. 20. Hybrid Neural Networks • A cascade of CNN and RNN 20 P. Bashivan, et. al. “Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks”. ICLR, 2016 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  21. 21. HNN for Real: TreNet for Learning the Local Trend 21 T. Guo, et al. TreNet: Hybrid Neural Networks for Learning the Local Trend in Temporal Data. In submission to ICLR, 2017 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  22. 22. Preliminaries • Conventional trend analysis in time series is the seasonal component at time t is the trend component at t is the remainder 22Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  23. 23. Preliminaries • Local trend • Measure the intermediate local behaviour, i.e. upward or downward pattern of time series • For instance, the time series of household power consumption and the local trends are shown as follows: • Time series • Extracted local trend , is the duration and is the slope 23Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  24. 24. Preliminaries • Learning and forecasting the local trend • Predict , • Useful in many applications • Smart energy • Stock market • And more … 24Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  25. 25. Preliminaries • Learning and forecasting the local trend • Local raw data • Global contextual information in the historical sequence of trend • To learn a function is either or 25Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  26. 26. TreNet • Overview of TreNet • is derived by training the LSTM over sequence to capture the dependency in the trend evolving. • corresponds to local features extracted by CNN from 26Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  27. 27. is derived by training the LSTM over sequence to capture the dependency in the trend evolving. corresponds to local features extracted by CNN from TreNet • Overview • 27Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  28. 28. TreNet • Learning 28Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  29. 29. TreNet • LSTM • Feed the sequence of • Output 29Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  30. 30. TreNet • CNN • Output 30Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  31. 31. TreNet • Feature Fusion and output layers 31Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  32. 32. TreNet • Learning • Gradient descent 32Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  33. 33. Experiments • Datasets • Daily Household Power Consumption (HousePC) • Gas Sensor (GasSensor) • Stock Transaction (Stock) 33 E Keogh, et al. “An online algorithm for segmenting time series”. ICDM, 2001 Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  34. 34. Experiments • Baselines • CNN • LSTM • Support Vector Regression (SVR) • Radial Basis kernel (SVRBF) • Polynomial kernel (SVPOLY) • Sigmoid kernel (SVSIG) • Pattern-based Hidden Markov Model (pHMM) 34Introduction Hybrid Neural Network(HNN) TreNet for Local Trend P. Wang, et al. “Finding Semantics in Time Series”, SIGMOD 2011
  35. 35. Experiments • Results: overall accuracy 35Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  36. 36. Experiments • Results: prediction visualization 36Introduction Hybrid Neural Network(HNN) TreNet for Local Trend
  37. 37. Conclusion 37
  38. 38. Conclusion 38 • Hybrid neural networks • TreNet for the Local Trend Learning • Future work: a generic idea • Social media streams • Heterogeneous data • Influence analysis • And more…
  39. 39. Thanks! Q & A This work is supported by EU OpenIoT Project (Open Source Solution for the Internet of Things) http://www.openiot.eu/ Feel free to contact: tian.guo0980@gmail.com, tian.guo@epfl.ch

×