Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Time-series forecasting of indoor tem...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Index
1 Introduction and motivation
2...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Introduction and motivation
Index
1 I...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Introduction and motivation
Introduct...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Introduction and motivation
Deep arch...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Introduction and motivation
Forecasti...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Stacked Denoising Auto-Encoders
Index...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Stacked Denoising Auto-Encoders
Stack...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Stacked Denoising Auto-Encoders
Stack...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Time series forecasting
Index
1 Intro...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Time series forecasting
Time series f...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Index
1 Introduction ...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Dataset
Dataset
Captu...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Evaluation measures
E...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Experiments
Experimen...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Experiments
Experimen...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Results
Results
Best ...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Results
Results
20 ra...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Results
Results
MSE o...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Experimentation
Results
Results
MAE o...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Conclusions and future work
Index
1 I...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Conclusions and future work
Conclusio...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Conclusions and future work
Questions...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Appendix
Appendix: Hyper-parameter op...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Appendix
Appendix: hyper-parameters a...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Appendix
Appendix: hyper-parameters a...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Appendix
Appendix: hyper-parameters a...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Appendix
Appendix: hyper-parameters a...
Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks
Appendix
Appendix: results table
MAE
...
Upcoming SlideShare
Loading in …5
×

Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks

3,390 views

Published on

Artificial neural networks have proved to be good at time-series forecasting
problems, being widely studied at literature. Traditionally, shallow
architectures were used due to convergence problems when dealing with deep
models. Recent research findings enable deep architectures training, opening a
new interesting research area called deep learning. This paper presents a study
of deep learning techniques applied to time-series forecasting in a real indoor
temperature forecasting task, studying performance due to different
hyper-parameter configurations. When using deep models, better generalization
performance at test set and an over-fitting reduction has been observed.

Published in: Education, Technology, Business
  • Be the first to comment

Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks

  1. 1. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks P. Romeu, F. Zamora-Mart´ınez, P. Botella-Rocamora, J. Pardo Embedded Systems and Artificial Intelligence group Departamento de ciencias f´ısicas, matem´aticas y de la computaci´on Escuela Superior de Ense˜nanzas T´ecnicas (ESET) Universidad CEU Cardenal Herrera, 46115 Alfara del Patriarca, Valencia (Spain) ICANN – September 11, 2013
  2. 2. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
  3. 3. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
  4. 4. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Introduction and motivation Time series forecasting: prediction future values given past data. ¯s = s0,...,si−1,si,si+1,... Non-linear relationships could be found between the elements. ANNs were widely used for this task, normally shallow models. Deep architectures has been successful in computer vision, speech signal processing, classification, . . . Time series forecasting with deep architectures is starting to receive interest (as far as we know, using Restricted Boltzmann Machines).
  5. 5. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Deep architectures on time series Expectations Time series are characterized by more or less complex dependencies. For indoor temperature forecasting: Known dependencies: time of the day, day of the year. Hidden dependencies: number of people in a room. Short-term dependencies and long-term dependencies. Normally, expert knowledge is introduced to take into account known dependencies; data preprocessing: detrend, deseasoned. A deep model could learn some of these dependencies using several layers.
  6. 6. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Forecasting of indoor temperature with deep ANNs What have we done in this work? Evaluation of pre-training and denoising techniques in a time series forecasting task. Results: slightly better generalization, less over-fitting. Problems: lack of data, not complex enough task. 15 16 17 18 19 20 21 22 23 24 25 26 0 2000 4000 6000 8000 10000 ºC Time (minutes)
  7. 7. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Stacked Denoising Auto-Encoders Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
  8. 8. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Stacked Denoising Auto-Encoders Stacked Denoising Auto-Encoders A Denoising Auto-Encoder is a neural network which receives a noisy input and produces its cleaned version. Gaussian additive noise (σ): ˙x = x+N (0,σ2 I) Masking noise (p): ˜x = MN(˙x) with p probability. Encoding: h(˜x) = so ftsign(b+W ˜x) Decoding (denoising): ˆx = g(h(˜x)) = so ftsign(c+WT h(˜x)) ˙x h(˜x) ˜x ˆx W WT x GN(x) MN(˙x) x is an input vector, h(·) is the hidden layer vector, b and c are bias vectors, W is a weights matrix, softsign(·) = x 1+|x|
  9. 9. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Stacked Denoising Auto-Encoders Stacked Denoising Auto-Encoders Greedy training building layer-by-layer auto-encoders. Stack all the trained weights to produce the final result. Stack a forecasting layer (linear activation). Train the whole neural network.
  10. 10. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Time series forecasting Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
  11. 11. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Time series forecasting Time series forecasting Univariate vs multivariate. Single-step-ahead vs multi-step-ahead. Iterative forecasting vs direct forecasting. Multiple Input One Output vs Multiple Input Multiple Output. ˆst+H t+1 = F(st t−I+1) MIMO modelling is natural in ANNs, because they take profit of the input/output mapping. F is a forecasting model, H the number of predicted samples, I the number of past samples taken as input.
  12. 12. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
  13. 13. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Dataset Dataset Captured during 2011, March and June. 1 minute sampling period. Reduced and smoothed by computing mean every 15 samples. Differences between adjacent samples were computed to remove the trend. Partition # of samples # of days Training 2016 21 Validation 672 7 Test 672 7
  14. 14. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Evaluation measures Evaluation measures Mean Absolute Error (MAE) Root Mean Square Error (RMSE) MAE (t) = 1 |D| |D| ∑ t=I 1 H H ∑ h=1 |ˆst+h −st+h| RMSE (t) = 1 |D| |D| ∑ t=I 1 H H ∑ h=1 (ˆst+h −st+h)2 |D| is the size of the dataset, H the future horizon, ˆst+h the forecasted value, st+h the ground truth.
  15. 15. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Experiments Experiments Different training modes comparison TM-0 consists in a standard training of an ANN. TM-1 pre-train the ANN using SDAE and fine-tuning of the whole network TM-2 pre-train the ANN using SDAE and fine-tuning of only last layer (forecasting layer).
  16. 16. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Experiments Experiments Training description Back-propagation with mini-batch size 32. Mean Square Error (MSE) loss function. Future horizon of 12 samples (three hours). Minimum of 50 epochs, maximum of 4000. Random search hyper-parameter optimization: learning rate, momentum, weight decay, number of hidden layers, hidden layer sizes, number of inputs, mask noise percentage. 3600 experiments for tuning.
  17. 17. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results Best topologies - TM-0: 60 — 756 — 60 — 12 - TM-1: 48 — 648 — 920 — 16 — 12 - TM-2: 96 — 712 — 12 TM-0 has convergence problems with deep networks: 33% of two layered network experiments do not converge. 58% of three layered network experiments do not converge. Note that the topologies are not the same in the three cases, we took the best topology for each training mode.
  18. 18. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results 20 random initializations of best hyper-parameters 0.115 0.120 0.125 0.130 0.135 0.140 TM-0 TM-1 TM-2 MAE* Validation Test 0.135 0.140 0.145 0.150 0.155 0.160 0.165 0.170 TM-0 TM-1 TM-2 RMSE*
  19. 19. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results MSE of training partition during training 0.010 0.014 0.019 0.025 0.034 0.046 0.063 0.086 0.117 0.159 0 200 400 600 800 1000 1200 1400 TrainingMSE(log-scaled) Epochs TM-0 TM-1 TM-2
  20. 20. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results MAE of test partition during training 0.117 0.159 0.216 0.293 0.398 0 200 400 600 800 1000 1200 1400 TestMAE*(log-scaled) Epochs best val TM-0 best val TM-1 best val TM-2 TM-0 TM-1 TM-2
  21. 21. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Conclusions and future work Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
  22. 22. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Conclusions and future work Conclusions and future work Pre-training, denoising techniques, and random hyper-parameter optimization were used to carry out deep ANNs training in a forecasting task. Slightly better generalization performance at test set and a reduction in over-fitting was observed (TM-1). Fine-tuning phase of the whole deep model was needed to ensure good results (TM-1 vs TM-2). The short benefit of SDAE could be due to the low dimensionality of the task. In the future, this work will be extended by using larger forecasting input window combined with multivariate forecasting.
  23. 23. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Conclusions and future work Questions? Thanks for your attention!
  24. 24. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: Hyper-parameter optimization Grid search part Train Mode: TM-0, TM-1, TM-2 Number of hidden layers: 1, 2, 3 Mask Noise: 0.02, 0.04, 0.10, 0.20 Random search part 100 random trials for every grid sweep Input size: 12, 24, 36, 48, 60, 72, 84, 96 Learning rate: [10−3 ,10−2 ] Momentum: ∼ N (10−3 ,5×10−3 ), ignoring negative values Weight decay: [0,10−5 ] Hidden layer sizes: [4,1024]
  25. 25. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Input size 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 12 36 60 84 TM-0 12 36 60 84 TM-1 1 layer 2 layers 3 layers 12 36 60 84 TM-2
  26. 26. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Encoding layer size 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0 300 600 900 TM-0 0 300 600 900 TM-1 0 300 600 900 TM-2
  27. 27. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Masking noise 0.12 0.13 0.14 0.15 0.16 0.17 0.02 0.10 0.18 TM-0 0.02 0.10 0.18 TM-1 0.02 0.10 0.18 TM-2
  28. 28. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Learning rate of forecasting phase 0.12 0.13 0.14 0.15 0.16 0.17 0 0.003 0.006 0.009 TM-0 0 0.003 0.006 0.009 TM-1 0 0.003 0.006 0.009 TM-2
  29. 29. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: results table MAE Validation (µ±σ) Test (µ±σ) ETS 0.3004 0.3254 TM-0 0.1289±0.0011 0.12482±0.0010 TM-1 0.1287±0.0033 0.1223±0.0033 TM-2 0.1374±0.0007 0.1279±0.0011 RMSE Validation (µ±σ) Test (µ±σ) ETS 0.3648 0.3930 TM-0 0.1563±0.0011 0.1511±0.0012 TM-1 0.1565±0.0040 0.1473±0.0039 TM-2 0.1663±0.0009 0.1538±0.0013

×