Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

3,390 views

Published on

problems, being widely studied at literature. Traditionally, shallow

architectures were used due to convergence problems when dealing with deep

models. Recent research findings enable deep architectures training, opening a

new interesting research area called deep learning. This paper presents a study

of deep learning techniques applied to time-series forecasting in a real indoor

temperature forecasting task, studying performance due to different

hyper-parameter configurations. When using deep models, better generalization

performance at test set and an over-fitting reduction has been observed.

License: CC Attribution-ShareAlike License

No Downloads

Total views

3,390

On SlideShare

0

From Embeds

0

Number of Embeds

12

Shares

0

Downloads

104

Comments

0

Likes

3

No embeds

No notes for slide

- 1. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks P. Romeu, F. Zamora-Mart´ınez, P. Botella-Rocamora, J. Pardo Embedded Systems and Artiﬁcial Intelligence group Departamento de ciencias f´ısicas, matem´aticas y de la computaci´on Escuela Superior de Ense˜nanzas T´ecnicas (ESET) Universidad CEU Cardenal Herrera, 46115 Alfara del Patriarca, Valencia (Spain) ICANN – September 11, 2013
- 2. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
- 3. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
- 4. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Introduction and motivation Time series forecasting: prediction future values given past data. ¯s = s0,...,si−1,si,si+1,... Non-linear relationships could be found between the elements. ANNs were widely used for this task, normally shallow models. Deep architectures has been successful in computer vision, speech signal processing, classiﬁcation, . . . Time series forecasting with deep architectures is starting to receive interest (as far as we know, using Restricted Boltzmann Machines).
- 5. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Deep architectures on time series Expectations Time series are characterized by more or less complex dependencies. For indoor temperature forecasting: Known dependencies: time of the day, day of the year. Hidden dependencies: number of people in a room. Short-term dependencies and long-term dependencies. Normally, expert knowledge is introduced to take into account known dependencies; data preprocessing: detrend, deseasoned. A deep model could learn some of these dependencies using several layers.
- 6. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Introduction and motivation Forecasting of indoor temperature with deep ANNs What have we done in this work? Evaluation of pre-training and denoising techniques in a time series forecasting task. Results: slightly better generalization, less over-ﬁtting. Problems: lack of data, not complex enough task. 15 16 17 18 19 20 21 22 23 24 25 26 0 2000 4000 6000 8000 10000 ºC Time (minutes)
- 7. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Stacked Denoising Auto-Encoders Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
- 8. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Stacked Denoising Auto-Encoders Stacked Denoising Auto-Encoders A Denoising Auto-Encoder is a neural network which receives a noisy input and produces its cleaned version. Gaussian additive noise (σ): ˙x = x+N (0,σ2 I) Masking noise (p): ˜x = MN(˙x) with p probability. Encoding: h(˜x) = so ftsign(b+W ˜x) Decoding (denoising): ˆx = g(h(˜x)) = so ftsign(c+WT h(˜x)) ˙x h(˜x) ˜x ˆx W WT x GN(x) MN(˙x) x is an input vector, h(·) is the hidden layer vector, b and c are bias vectors, W is a weights matrix, softsign(·) = x 1+|x|
- 9. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Stacked Denoising Auto-Encoders Stacked Denoising Auto-Encoders Greedy training building layer-by-layer auto-encoders. Stack all the trained weights to produce the ﬁnal result. Stack a forecasting layer (linear activation). Train the whole neural network.
- 10. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Time series forecasting Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
- 11. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Time series forecasting Time series forecasting Univariate vs multivariate. Single-step-ahead vs multi-step-ahead. Iterative forecasting vs direct forecasting. Multiple Input One Output vs Multiple Input Multiple Output. ˆst+H t+1 = F(st t−I+1) MIMO modelling is natural in ANNs, because they take proﬁt of the input/output mapping. F is a forecasting model, H the number of predicted samples, I the number of past samples taken as input.
- 12. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
- 13. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Dataset Dataset Captured during 2011, March and June. 1 minute sampling period. Reduced and smoothed by computing mean every 15 samples. Differences between adjacent samples were computed to remove the trend. Partition # of samples # of days Training 2016 21 Validation 672 7 Test 672 7
- 14. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Evaluation measures Evaluation measures Mean Absolute Error (MAE) Root Mean Square Error (RMSE) MAE (t) = 1 |D| |D| ∑ t=I 1 H H ∑ h=1 |ˆst+h −st+h| RMSE (t) = 1 |D| |D| ∑ t=I 1 H H ∑ h=1 (ˆst+h −st+h)2 |D| is the size of the dataset, H the future horizon, ˆst+h the forecasted value, st+h the ground truth.
- 15. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Experiments Experiments Different training modes comparison TM-0 consists in a standard training of an ANN. TM-1 pre-train the ANN using SDAE and ﬁne-tuning of the whole network TM-2 pre-train the ANN using SDAE and ﬁne-tuning of only last layer (forecasting layer).
- 16. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Experiments Experiments Training description Back-propagation with mini-batch size 32. Mean Square Error (MSE) loss function. Future horizon of 12 samples (three hours). Minimum of 50 epochs, maximum of 4000. Random search hyper-parameter optimization: learning rate, momentum, weight decay, number of hidden layers, hidden layer sizes, number of inputs, mask noise percentage. 3600 experiments for tuning.
- 17. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results Best topologies - TM-0: 60 — 756 — 60 — 12 - TM-1: 48 — 648 — 920 — 16 — 12 - TM-2: 96 — 712 — 12 TM-0 has convergence problems with deep networks: 33% of two layered network experiments do not converge. 58% of three layered network experiments do not converge. Note that the topologies are not the same in the three cases, we took the best topology for each training mode.
- 18. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results 20 random initializations of best hyper-parameters 0.115 0.120 0.125 0.130 0.135 0.140 TM-0 TM-1 TM-2 MAE* Validation Test 0.135 0.140 0.145 0.150 0.155 0.160 0.165 0.170 TM-0 TM-1 TM-2 RMSE*
- 19. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results MSE of training partition during training 0.010 0.014 0.019 0.025 0.034 0.046 0.063 0.086 0.117 0.159 0 200 400 600 800 1000 1200 1400 TrainingMSE(log-scaled) Epochs TM-0 TM-1 TM-2
- 20. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Experimentation Results Results MAE of test partition during training 0.117 0.159 0.216 0.293 0.398 0 200 400 600 800 1000 1200 1400 TestMAE*(log-scaled) Epochs best val TM-0 best val TM-1 best val TM-2 TM-0 TM-1 TM-2
- 21. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Conclusions and future work Index 1 Introduction and motivation 2 Stacked Denoising Auto-Encoders 3 Time series forecasting 4 Experimentation 5 Conclusions and future work
- 22. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Conclusions and future work Conclusions and future work Pre-training, denoising techniques, and random hyper-parameter optimization were used to carry out deep ANNs training in a forecasting task. Slightly better generalization performance at test set and a reduction in over-ﬁtting was observed (TM-1). Fine-tuning phase of the whole deep model was needed to ensure good results (TM-1 vs TM-2). The short beneﬁt of SDAE could be due to the low dimensionality of the task. In the future, this work will be extended by using larger forecasting input window combined with multivariate forecasting.
- 23. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Conclusions and future work Questions? Thanks for your attention!
- 24. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: Hyper-parameter optimization Grid search part Train Mode: TM-0, TM-1, TM-2 Number of hidden layers: 1, 2, 3 Mask Noise: 0.02, 0.04, 0.10, 0.20 Random search part 100 random trials for every grid sweep Input size: 12, 24, 36, 48, 60, 72, 84, 96 Learning rate: [10−3 ,10−2 ] Momentum: ∼ N (10−3 ,5×10−3 ), ignoring negative values Weight decay: [0,10−5 ] Hidden layer sizes: [4,1024]
- 25. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Input size 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 12 36 60 84 TM-0 12 36 60 84 TM-1 1 layer 2 layers 3 layers 12 36 60 84 TM-2
- 26. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Encoding layer size 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.20 0.21 0 300 600 900 TM-0 0 300 600 900 TM-1 0 300 600 900 TM-2
- 27. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Masking noise 0.12 0.13 0.14 0.15 0.16 0.17 0.02 0.10 0.18 TM-0 0.02 0.10 0.18 TM-1 0.02 0.10 0.18 TM-2
- 28. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: hyper-parameters analysis Learning rate of forecasting phase 0.12 0.13 0.14 0.15 0.16 0.17 0 0.003 0.006 0.009 TM-0 0 0.003 0.006 0.009 TM-1 0 0.003 0.006 0.009 TM-2
- 29. Time-series forecasting of indoor temperature using pre-trained Deep Neural Networks Appendix Appendix: results table MAE Validation (µ±σ) Test (µ±σ) ETS 0.3004 0.3254 TM-0 0.1289±0.0011 0.12482±0.0010 TM-1 0.1287±0.0033 0.1223±0.0033 TM-2 0.1374±0.0007 0.1279±0.0011 RMSE Validation (µ±σ) Test (µ±σ) ETS 0.3648 0.3930 TM-0 0.1563±0.0011 0.1511±0.0012 TM-1 0.1565±0.0040 0.1473±0.0039 TM-2 0.1663±0.0009 0.1538±0.0013

No public clipboards found for this slide

Be the first to comment