This document describes Kostas Hatalis's research on using recurrent neural networks for multi-step time series prediction of renewable power generation. It discusses using nonlinear autoregressive networks and particle swarm optimization learning in time delayed recurrent neural networks to forecast ocean wave characteristics. Evaluation of the models found the particle swarm optimization approach improved accuracy over nonlinear autoregressive networks, particularly for noisier, longer term predictions.

1. Time Delayed Recurrent Neural Network
for Multi-Step Prediction
By Kostas Hatalis
hatalis@gmail.com
Dept. of Electrical & Computer Engineering
Lehigh University, Bethlehem, PA
2014
Kostas Hatalis Recurrent Neural Network 2014 1 / 18

2. Renewable Power Forecasting
Forecasting is essential to the integration of renewable power
generation to the smart grid.
Kostas Hatalis Recurrent Neural Network 2014 2 / 18

3. General Short Term Time Frames and Methods
Resolution of forecasting:
Seconds: e.g. turbine/converter control.
Minutes: e.g. balancing and transmission.
Hours: e.g. storage and scheduling.
Days: e.g. farm operations.
Two categories of forecasting methods:
Physical: meteorological models.
Statistical: time series / machine learning.
My research focusing on statistical models.
Kostas Hatalis Recurrent Neural Network 2014 3 / 18

4. Time Series Forecasting
Time series forecasts are made as follows:
1 Study features of the series.
2 Remove trend and seasonality to get stationary residuals.
3 Test if IID noise then fit an ARMA model to residuals.
4 Forecast residual series by minimizing the mean squared error of the
expected future value.
Kostas Hatalis Recurrent Neural Network 2014 4 / 18

5. Trend and Seasonal Fitting
The classical decomposition model:
Kostas Hatalis Recurrent Neural Network 2014 5 / 18

6. Fitting ARMA Models
For our residual sequence, Yt is an ARMA(p,q) process if:
Yt − φ1Yt−1 − ... − φpYt−p = Zt + θ1Zt−1 + ... + θqZt−q
where Zt is white noise with mean zero and variance σ2.
Once we estimate all the parameters we can forecast future values at time
t + k. Eg. of an ARMA(2,1) model forecasting at t + 1 ahead:
ˆYt+1 = φ1Yt + φ2Yt−1 + Zt+1 − θ1Zt
ARMA is a model for the conditional mean of a process.
Kostas Hatalis Recurrent Neural Network 2014 6 / 18

7. Time Series Forecasting
Other Time Series forecasting models include:
Weighted moving average
Kalman filtering
Exponential smoothing
Autoregressive integrated moving average (ARIMA)
Seasonal ARIMA (SARIMA)
Extrapolation methods
Downside? Most assume data is stationary (distribution and
parameters do not change overtime), renewables are non-stationary!
Kostas Hatalis Recurrent Neural Network 2014 7 / 18

8. Evaluation Methods
Methods evaluating the predicted values vs the observed value
include:
Mean Squared Error (MSE)
Root Mean Squared Error (RMSE)
Median Absolute Deviation (MAD)
Mean Absolute Percentage Error (MAPE)
Sum of Squared Error (SSE)
Mean Absolute Error (MAE)
Signed Mean Squared Error (SMSE)
Kostas Hatalis Recurrent Neural Network 2014 8 / 18

9. Machine Learning Approach
Machine learning based forecasting updates it’s model with every
new observation
Prediction inference based on supervised learning which consists in
learning the link between two datasets: the observed data X and an
variable y that we are trying to predict, usually called “targets”.
Kostas Hatalis Recurrent Neural Network 2014 9 / 18

10. Statistical Learning Forecasting Methods
A number of machine learning methods have been applied for point
forecasting such as support vector machines, Bayesian networks, k-nearest
neighbors, etc. For point forecasting Artificial Neural Networks (ANN) are
amongst the most popular.
Universal Approximation Theorem
An artificial neural network with a single hidden layer containing a finite
number of neurons can approximate any function provided the activation
function f of the hidden neurons are non-linear.
Kostas Hatalis Recurrent Neural Network 2014 10 / 18

11. NARNET - Forecasting Ocean Wave
In my initial forecasting work I used an ANN called nonlinear
autoregressive network (NARNET) to forecast wave heights.
https://en.wikipedia.org/wiki/Significantwaveheight
Kostas Hatalis Recurrent Neural Network 2014 11 / 18

12. NARNET - Building a Neural Network
When building an ANN its important to define:
Architecture: number of layers and nodes.
Activation function: sigmoid, tanh, etc.
Cost function: mean squared error, etc.
Learning: gradient descent backpropagation, etc.
Kostas Hatalis Recurrent Neural Network 2014 12 / 18

13. NARNET - Model
For time series prediction NARNET is a recurrent ANN:
Kostas Hatalis Recurrent Neural Network 2014 13 / 18

14. NARNET - Results
Kostas Hatalis Recurrent Neural Network 2014 14 / 18

15. PSONET - Simulated Ocean Waves
Short term forecasting of ocean waves must be simulated.
Also NARNET has few problems, such as using too much memory
and occasionally getting stuck in local minamas. Needed to find new
training solution.
Kostas Hatalis Recurrent Neural Network 2014 15 / 18

16. PSONET - Particle Swarm Optimization
The adaptive particle swarm optimization (APSO) algorithm inspired by
flock of birds/fish searching for food:
−→v i (t + 1) =
ω−→v i (t) + c1φ1(−→p i (t) − −→x i (t)) + c2φ2(−→p g (t) − −→x i (t))
−→x i (t + 1) = −→x i (t) + ∆t−→v i (t + 1)
Kostas Hatalis Recurrent Neural Network 2014 16 / 18

17. PSONET - Results
Table: Table of error statistics with higher noisy data.
PSONAR 5s 10s 30s 60s
MSE 0.1603 0.9153 8.9066 13.4073
RMSE 0.4579 0.9567 2.9843 3.6615
MAPE 0.1734 0.2964 2.0758 3.2429
MAD 0.3470 0.6783 2.4161 2.8644
CC 0.9974 0.9693 0.3743 0.0902
NARNET 5s 10s 30s 60s
MSE 0.2097 1.4119 10.8491 15.0918
RMSE 0.4003 1.1882 3.2937 3.7525
MAPE 0.2531 0.7421 1.3229 4.7647
MAD 0.3698 0.8938 2.0769 3.0842
CC 0.9971 0.9165 0.31433 0.12301
Kostas Hatalis Recurrent Neural Network 2014 17 / 18

18. Work Done
[1] Hatalis, Kostas, et al. ”Multi-step forecasting of wave power using a
nonlinear recurrent neural network.” 2014 IEEE PES General Meeting—
Conference & Exposition. IEEE, 2014.
[2] Hatalis, Kostas, et al. ”Adaptive particle swarm optimization learning
in a time delayed recurrent neural network for multi-step prediction.”
Foundations of Computational Intelligence (FOCI), 2014 IEEE Symposium
on. IEEE, 2014.
[3] Hatalis, Kostas, et al. ”Swarm Based Parameter Estimation of Wave
Characteristics for Control in Ocean Energy Farms.” In Proceedings of the
IEEE Power Energy Society General Meeting, 2015.
[4] Hatalis, Kostas, et al. ”Particle Swarm Based Model Exploitation for
Parameter Estimation of Wave Realizations.” IEEE Symposium Series on
Computational Intelligence , 2016
Kostas Hatalis Recurrent Neural Network 2014 18 / 18