Neural probabilistic forecasting using a novel smooth loss function for uncertainty prediction with a primal penalty to ensure monotonic composite quantile estimation.

1. Smooth Pinball based Quantile Network:
Neural Probabilistic Forecasting
By Kostas Hatalis
hatalis@gmail.com
Dept. of Electrical & Computer Engineering
Lehigh University, Bethlehem, PA
2018
Kostas Hatalis Quantile Neural Network 2018 1 / 21

2. Introduction
Inspired by SVQR we want to expand the investigation of nonlinear
nonparametric probabilistic forecasting using deep learning.
1 We propose and investigate a new objective function for neural
networks.
2 We introduce a way to prevent the quantile crossover problem.
3 We showcase how a multiple quantile based neural network can be
used for probabilistic forecasting of wind.
4 We design experiments using publicly available data from the Global
Energy Forecasting Competition 2014.
5 We show our method improves the skill, reliability, and sharpness over
various benchmarks.
Kostas Hatalis Quantile Neural Network 2018 2 / 21

3. Smooth Pinball Function
Pinball Loss Function
ρτ (u) =
τu if u ≥ 0
(τ − 1)u if u < 0
Quantile regression optimization problem:
min
W ,b
1
N
N
t=1
ρτ (yt − ˆq
(τ)
t ) (1)
The smooth approximation of the pinball function:
Sτ,α(u) = τu + α log 1 + exp −
u
α
Smooth quantile regression:
min
W ,b
1
N
N
t=1
Sτ,α(yt − ˆq
(τ)
t )
Kostas Hatalis Quantile Neural Network 2018 3 / 21

4. Smooth Pinball Function
Kostas Hatalis Quantile Neural Network 2018 4 / 21

5. Smooth Pinball Neural Network (SPNN)
The objective function for SPNN is then given by
E =
λ1
2NM
W [1] 2
F +
λ2
2NM
W [2] 2
F +
1
NM
N
t=1
M
m=1
...
τm(yt − ˆq
(τm)
t ) + α log 1 + exp −
yt − ˆq
(τm)
t
α
.
Kostas Hatalis Quantile Neural Network 2018 5 / 21

6. Smooth Pinball Neural Network (SPNN)
Gradient descent with backpropagation can be used to train SPNN.
Example gradients for a 2 layer SPNN:
∂Et
∂W [2]
=
λ2
M
W [2]
+
∂Et
∂ ˆQt
·
∂ ˆQt
∂Z
[2]
t
·
∂Z
[2]
t
∂W [2]
=
λ2
M
W [2]
+
1
M

 1
1 + exp yt − ˆQt
α
− T

 Ht
With respect to the weights of the first layer W[1] as follows
∂Et
∂W [1]
=
λ1
M
W
[1]
+


∂Et
∂ ˆQt
·
∂ ˆQt
∂Z
[2]
t
·
∂Z
[2]
t
∂Ht

 ·
∂Ht
∂Z
[1]
t
·
∂Z
[1]
t
∂W [1]
=
λ1
M
W
[1]
+
1
M




1
1 + exp
yt − ˆQt
α
− T



 W
[2]
1 − H
2
t Xt
.
Kostas Hatalis Quantile Neural Network 2018 6 / 21

7. Smooth Pinball Neural Network (SPNN)
Kostas Hatalis Quantile Neural Network 2018 7 / 21

8. Kostas Hatalis Quantile Neural Network 2018 8 / 21

9. Noncrossing Quantiles - Smooth Penalty Constraint
The condition 0 < τ1 < ... < τM are defined as the orders of M
conditional quantiles to be estimated.
To ensure these quantiles do not cross each other the following
constraint is needed q
(τ1)
t ≤ ... ≤ q
(τM )
t , ∀t.
We define the non-crossing quantile penalty term p as follows
penalty = c
N
t=1
M
m=1
max 0, − ˆq
(τm−1)
t − ˆq
(τm)
t
2
where ˆq
(τ0)
t = 0, is the least amount that the two quantile should differ
by, and c is the penalty parameter with a high value.
Kostas Hatalis Quantile Neural Network 2018 9 / 21

10. Empirical Evaluation
Datasets
Publicly available Global Energy Forecasting Competition 2014
(GEFCom2014)
Evaluation Metrics
QS, IS, ACE, and Sharpness
Benchmark Methods
Uniform, Climatology, Persistence, SVQR, QR
Case Studies Both studies we test SPNN having one and two hidden
layers denoted as SPNN1 and SPNN2. Only use raw data, no feature
engineering!
1. Estimate prediction intervals with nominal coverage from 10% to
90% in increments of 10%. Two wind farms. Also looked at QVSS
score.
2. We estimate 99 quantiles across all 10 wind farms.
Kostas Hatalis Quantile Neural Network 2018 10 / 21

11. Quantile Verification Skill Score (QVSS)
Quantile verification skill score (QVSS):
QVSS = 1 −
QSfor
QSref
QSfor is QS for the forecast method of interest (SPNN in our case)
QSref is the QS value for the reference forecast of a benchmark
method, which we use linear quantile regression.
Measures relative performance between methods.
Kostas Hatalis Quantile Neural Network 2018 11 / 21

12. Example Forecast
Kostas Hatalis Quantile Neural Network 2018 12 / 21

13. Sensitivity Analysis
Partial Derivatives (PaD) Method:
dt,i,M =
∂ ˆQτm
t
∂xt,i
=
∂ ˆQτm
t
∂Z
[2]
t
·
∂Z
[2]
t
∂Ht
·
∂Ht
∂Z[1],t
·
∂Z
[1]
t
∂xt,i
Sum of Square Derivatives (SSD):
SSDi,M =
N
t=1
(dt,i,M)
Kostas Hatalis Quantile Neural Network 2018 13 / 21

14. Sensitivity Analysis
Kostas Hatalis Quantile Neural Network 2018 14 / 21

15. Sensitivity Analysis, partial derivative of U100 for 90%
Kostas Hatalis Quantile Neural Network 2018 15 / 21

16. Case Study 1: Zone 1
Kostas Hatalis Quantile Neural Network 2018 16 / 21

17. Case Study 1: Zone 2
Kostas Hatalis Quantile Neural Network 2018 17 / 21

18. Case Study 2: ACE and Sharpness
Kostas Hatalis Quantile Neural Network 2018 18 / 21

19. Case Study 2: QS and IS
Kostas Hatalis Quantile Neural Network 2018 19 / 21

20. Case Study 2: GEFCom2014
Kostas Hatalis Quantile Neural Network 2018 20 / 21

21. Work Done
Probabilistic Forecasting by Deep Learning
[1] Kostas Hatalis, Alberto J Lamadrid, Katya Scheinberg, and Shalinee
Kishore. Multi quantile estimation based neural network for probabilistic
forecasting of wind power. IEEE Transactions on Neural Networks and
Learning Systems (In-Submission).
Kostas Hatalis Quantile Neural Network 2018 21 / 21