Universal approximators for Direct Policy Search in multi-purpose water reservoir management: A comparative analysis

1. IFAC
2014
CAPE
TOWN
-­‐ZA
Universal approximators for Direct Policy Search
in multi-purpose water reservoir management: A
comparative analysis
Matteo Giuliani, Emanuele Mason, Andrea Castelletti, Francesca Pianosi,
Rodolfo Soncini-Sessa
Dipartimento di Elettronica, Informazione, e Bioingegneria, Politecnico di Milano, Milano, Italy
Hydroinformatics Lab, Como Campus, Politecnico di Milano, Italy
Department of Civil and Environmental Engineering, University of Bristol, Bristol, UK
Modelling
and
Control
of
Water
Systems

2. Controlling hydro-environmental systems
The long-term optimal operation of hydro-environmental systems can be
formulated as a q-objective stochastic optimal control problem
min
μt(·)
J = |J1 J2...J q|
xt+1 = ft(xt, ut, t+1)
Ji = lim
h⇤⌅
E
⇥h1
⇥
h⇤−1
t=0
i-th immediate cost
i = 1, t(xt, ut, ⇥t+1)
!tgi
⇥
i-th objective discount factor
state control disturbance
subject to
!
ut = μt(xt)
t+1 ⇠ (·)
xt 2 Rnx
ut 2 Rnu
t 2 Rn

3. SDP and the 3 curses
Stochastic Dynamic Programming is - in principle - the best approach
to solve the problem - in practice - it suffers from 3 major shortcomings
1) Curse of dimensionality: computational cost grows exponentially with
state, control and disturbance dimension [Bellman, 1967];
ut
Qt
ut
xt
Look-up table
Q-function
unknown
Q-function
computations are numerically
performed on a discretized variable
domain

4. SDP and the 3 curses
Stochastic Dynamic Programming is - in principle - the best approach
to solve the problem - in practice - it suffers from 3 major shortcomings
1) Curse of dimensionality: computational cost grows exponentially with
state, control and disturbance dimension [Bellman, 1967];
ut
Qt
ut
xt
Look-up table
Q-function
unknown
Q-function
computations are numerically
performed on a discretized variable
domain
2) Curse of modelling: any variable considered among the operating rule’s
arguments has to be modelled [Bertsekas and Tsitsiklis, 1996];
t t+1 time
xt
ut, t+1
models are use in a multiple one-step-
ahead-simulation mode

5. SDP and the 3 curses
Stochastic Dynamic Programming is - in principle - the best approach
to solve the problem - in practice - it suffers from 3 major shortcomings
3) Curse of multiple objectives: computational cost grows exponentially
with the number of objectives considered [Powell, 2011].
PARETO frontier
multi-objective problems are solved
by reiteratively solving single
objective problems
J1
J2
J3

6. Beyond SDP: ADP and RL
Approximate Dynamic Programming and Reinforcement Learning
provide a framework to overcome some or all the SDP’s curses.
[Powell, 2007; Busoniu et al. 2011
VALUE FUNCTION-BASED APPROCHES:
• Approximate value iteration
• Approximate policy iteration
• Approximate policy evaluation
Model-free or model-based // parametric or non-parametric
POLICY SEARCH-BASED APPROACHES:
• Direct policy search
Simulation-based optimization // parametric

7. Beyond SDP: ADP and RL
Approximate Dynamic Programming and Reinforcement Learning
provide a framework to overcome some or all the SDP’s curses.
[Powell, 2007; Busoniu et al. 2011
VALUE FUNCTION-BASED APPROCHES:
• Approximate value iteration
• Approximate policy iteration
• Approximate policy evaluation
Model-free or model-based // parametric or non-parametric
POLICY SEARCH-BASED APPROACHES:
• Direct policy search
Simulation-based optimization // parametricv

8. Multi-objective Direct Policy Search (MODPS)
Assuming the operating rule belong to a given family of functions and
search the optimal solution in the policy’s parameter space
ORIGINAL PROBLEM POLICY SEARCH PROBLEM
min
μt(·)
J = |J1 J2...J q|
subject to
!
xt+1 = ft(xt, ut, t+1)
ut = μt(xt)
t+1 ⇠ (·)
xt 2 Rnx
ut 2 Rnu
t 2 Rn
ut = μt(xt, ✓t)
min
μt(·)
J = |J1 J2...J q|
✓t
subject to
!
xt+1 = ft(xt, ut, t+1)
ut = μt(xt,✓t)
t+1 ⇠ (·)
xt 2 Rnx
ut 2 Rnu
t 2 Rn
✓t 2 ⇥t2 Rn✓

9. Selecting the policy approximation: Ad hoc/Empirism
WHEN
1. The system is already operated
500!
450!
400!
350!
300!
250!
200!
1
150!
100!
3
2 4
0 25 50 75 100 125 150 175 200 225
250!
50!
0!
release [m3/s]
storage [Mm3]
5
Identify existing regularities in a
sample of the operator behaviour
2. the system is simple (i.e. one reservoir) AND/OR the systems has
one single objective (e.g. water supply)
• NEW York City rule [Clark, 1950]
• Space rule [Clark, 1956]
• Standard Operating Policy [Draper, 2004]
• …..
Empirical rules identified in
the past

10. Selecting the policy approximation: Universal Approx.
Provided that some conditions are met, an Universal Approximator is
approximate arbitrarily closely every continuous function.
ARTIFICIAL NEURAL NETWORKS [Cybenko 1989, Funahashi 1989, Hornik et al. 1989]
Parameter dimension
n✓ = nu(N(nx + 2) + 1)
GAUSSIAN RADIAL BASIS FUNCTIONS [Busoniu et al. 2011]
Number of NEURONS
Parameter dimension
n✓ = N(2nx + nu)
Number of BASES
x1
x2
x3
u1
x1
x2
x3
u1

11. Selecting the optimization algorithm
Key problem features
• High dimensional search spaces (rich parameterizations)
• Complex search spaces (many local minima)
• Sensitivity to parameter initialization (no-preconditioning)
• Multiple objectives
• Non differentiable objective functions
• Sensitivity to noise

12. Selecting the optimization algorithm
Key problem features
• High dimensional search spaces (rich parameterizations)
• Complex search spaces (many local minima)
• Sensitivity to parameter initialization (no-preconditioning)
• Multiple objectives
• Non differentiable objective functions
• Sensitivity to noise
BORG [Hadka and Reed 2012; Reed et al. 2013]
a MULTI-OBJECTIVE EVOLUTIONARY ALGORITHM
BORG is self-adaptive and employs
• multiple search operators adaptively selected during the optimization
• e-dominance archiving with internal operators to detect search stagnation
• randomized restarts to escape local optima

13. CASE STUDY

14. Red-Thai Binh River System - Vietnam
HaGiang BaoLac
Hanoi
BacMe
Thao Lo
HoaBinh
TamDuong
NamGiang
TaBu
MuongTe
LaiChau
YenBai VuQuang
VIETNAM
CHINA
LAOS
THAILAND
CAMBODIA
Da
Integrated Management of Red-Thai Binh Rivers System (IMRR) funded by the Italian
Ministry of Foreign Affairs http://www.imrr.info/

15. Hoa Binh reservoir - Vietnam
Main characteristics
• Catchment area 52,000 km2
• Active capacity 6 x 109 m3
• 8 penstocks 2,360 m3/s (240 MW)
• 12 bottom gates 22,000 m3/s
• 6 spillways 14,000 m3/s
• 15% national energy (7,800 GWh)
source: IWRP2008
Operating objectives
• Hydropower production
• Flood control (Hanoi)
RESERVOIR
CATCHMENT
POWER PLANT
DIVERSION DAM
COMSUMPTIVE USE THAO
LO
DA
HOA
BINH

16. Experimental Setting: ANN vs RBF
STATE VECTOR (n_x=5)
• 2 time indexes (sin, cosin)
• Storage
• Previous day inflow to reservoir
• Previous day lateral inflow
CONTROL VECTOR (n_u=1)
• release from the reservoir
RESERVOIR
CATCHMENT
POWER PLANT
DIVERSION DAM
COMSUMPTIVE USE THAO
LO
DA
HOA
BINH
ALGORITHM SETTING and RUNNING
• Default Borg MOEA parameterization [Hadka and Reed 2013]
• NFE = 500,000 per replication
• 20 replications to avoid dependence on randomness
• Historical horizon 1962-1969, which comprises normal, wet and dry years

17. Policy perfomance – operating objectives
2.6 x 107
2.5
2.4
2.3
2.2
2.1
2
Legend: ANN - RBF
number of neurons/basis
1.80 100 200 300 400 500 600 700
(b) Generational distance (c) Additive ε-indicator (d) Hypervolume
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
4 6 8 101214 16 4 6 8 10 121416 4 6 8 101214 16 4 6 8 10 121416 4 6 8 101214 16 4 6 8 10 121416
# of neurons/basis # of neurons/basis # of neurons/basis
1.9
Jflo
Jhyd
468
10
12
14
16
(a) Policy Performance with different ANN and RBF architectures
ANN
RBF
Hydropower
-­‐
kWh/d
Floods
–
cm2/d

18. Policy perfomance – front approximation quality
2.6 x 107
2.5
2.4
2.3
2.2
2.1
2
Legend: ANN - RBF
number of neurons/basis
1.80 100 200 300 400 500 600 700
(b) Generational distance (c) Additive ε-indicator (d) Hypervolume
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
4 6 8 101214 16 4 6 8 10 121416 4 6 8 101214 16 4 6 8 10 121416 4 6 8 101214 16 4 6 8 10 121416
# of neurons/basis # of neurons/basis # of neurons/basis
1.9
Jflo
Jhyd
468
10
12
14
16
(a) Policy Performance with different ANN and RBF architectures
ANN
RBF
Hydropower
-­‐
kWh/d
Floods
–
cm2/d
CONVERGENCE CONSISTENCY DIVERSITY

19. Policy reliability
(a) 75% best metric value (b) 95% best metric value
4 6 8 101214 16 4 6 8 10 121416
1
0.8
0.6
0.4
0.2
0
1
0.8
0.6
0.4
0.2
1
0.8
0.6
0.4
0.2
0
4 6 8 101214 16 4 6 8 10 121416
0
4 6 8 101214 16 4 6 8 10 121416
1
0.8
0.6
0.4
0.2
0
4 6 8 101214 16 4 6 8 10 121416
hypervolume additive ε-indicator generational distance
4 6 8 101214 16 4 6 8 10 121416
4 6 8 101214 16 4 6 8 10 121416
1
0.8
0.6
0.4
0.2
0
1
0.8
0.6
0.4
0.2
0
# of neurons/basis # of neurons/basis
ANN
RBF
CONVERGENCE
CONSISTENCY
DIVERSITY
FIG. 3. Probability of attainment with a threshold equal to 75% (a) and to 95% (b) of

20. Run time search dynamics (NFA = 2M)
(a) Generational distance (b) Additive ε−indicator (c) Hypervolume
ANN ( 6 neurons )
RBF (6 bases)
0 0.5 1 1.5 2
1.4
1.2
1
0.8
0.6
0.4
0.2
0
0 0.5 1 1.5 2
1.4
1.2
1
0.8
0.6
0.4
0.2
0
0 0.5 1 1.5 2
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
NFA (x106) NFA (x106) NFA (x106)
FIG. 4. Analysis CONVERGENCE of runtime search dynamics CONSISTENCY for ANN (red lines) and DIVERSITY
RBF (blue lines)
operating policy optimization in terms of generational distance (a), additive -indicator
(b), and hypervolume (c).

21. Policy validation
2.6 x 107
2.4
2.2
2
1.80 100 200 300 400 500 600 700 800 900
Jflo
Jhyd
2.6 x 107
2.4
2.2
2
1.80 100 200 300 400 500 600 700 800 900
Jflo
Jhyd
ANN
RBF
ANN
RBF
(a) Results over the optimization horizon (1962-1969)
(b) Results over the validation horizon (1995-2004)
ANN
RBF
Hydropower
-­‐
kWh/d
Floods
–
cm2/d
Hydropower
-­‐
kWh/d
Floods
–
cm2/d

22. Conclusions
§ MODPS is an interesting alternative to SDP familiy methods for a number of
good reasons
1. No discretization required: NO curse of dimensionality;
2. Does not require separability in time of constraints and objective
functions (e.g. duration curves): NO curse of dimensionality;
3. Can easily include any model-free information as long as this is control-indipendent:
NO curse of modelling;
4. Can be combined with any simulation model (also high fidelity ones): NO
curse of modelling;
5. Can be easily combined with truly multi-objective optimization
algorithms: NO curse of the multiple objectives.

23. Conclusions
§ RBFs and ANNs seem to perform comparatively well when evaluated
in terms of policy performance
§ RBFs outperform ANNs in terms of quality of the Pareto front
approximation, reliability and run time search dynamics
§ Future works will focus on exploring multiple output policies (e.g.
network of reservoirs)

24. THANKS