The document discusses a distributed solution for a stochastic optimal control problem using GPUs, highlighting the advantages of accelerated proximal gradient algorithms for solving complex control challenges across various applications like microgrids and drinking water networks. It includes a formulation of the optimization problem, details about the algorithm's implementation and performance, and presents simulation results demonstrating significant speed improvements compared to traditional methods like interior-point solvers. The findings indicate a substantial reduction in runtime, emphasizing the efficacy of GPU-based solutions for large-scale stochastic control systems.

1. Distributed solution of stochastic optimal control
problem on GPUs
Ajay K. Sampathiraoa, P. Sopasakisa, A. Bemporada and P. Patrinosb
a IMT Institute for Advanced Studies Lucca, Italy
b Dept. Electr. Eng. (ESAT), KU Leuven, Belgium
December 18, 2015

2. Applications
Microgrids [Hans et al. ’15]
Drinking water networks [Sampathirao et al. ’15]
HVAC [Long et al. ’13, Zhang et al. ’13, Parisio et al. ’13]
Financial systems [Patrinos et al. ’11, Bemporad et al., ’14]
Chemical process [Lucia et al. ’13]
Distillation column [Garrido and Steinbach, ’11]
1 / 28

3. Motivation
Stochastic optimisation is not fit for control applications.
2 / 28

4. Spoiler alert!
Example:
920, 000 decision variables
Interior point runtime 35s
GPU APG solver < 3s
3 / 28

5. Outline
1. Stochastic optimal control problem formulation
2. Accelerated proximal gradient algorithm
3. Parallelisable implementation
4. Simulations
4 / 28

6. I. Stochastic Optimal Control

7. System description
Discrete-time uncertain linear system:
xk+1 = Aξk
xk + Bξk
uk + wξk
,
ξk is a random variable on a prob. space (Ωk, Fk, Pk). At time k we
observe xk but not ξk.
5 / 28

8. Stochastic optimal control problem
Optimisation problem:
V (p) = min
π={uk}k=N−1
k=0
E Vf (xN , ξN ) +
N−1
k=0
k(xk, uk, ξk) ,
s.t x0 = p,
xk+1 = Aξk
xk + Bξk
uk + wξk
,
6 / 28

9. Stochastic optimal control problem
Optimisation problem:
V (p) = min
π={uk}k=N−1
k=0
E Vf (xN , ξN ) +
N−1
k=0
k(xk, uk, ξk) ,
s.t x0 = p,
xk+1 = Aξk
xk + Bξk
uk + wξk
,
where:
E[·]: conditional expectation wrt the product probability measure
6 / 28

10. Stochastic optimal control problem
Optimisation problem:
V (p) = min
π={uk}k=N−1
k=0
E Vf (xN , ξN ) +
N−1
k=0
k(xk, uk, ξk) ,
s.t x0 = p,
xk+1 = Aξk
xk + Bξk
uk + wξk
,
where:
E[·]: conditional expectation wrt the product probability measure
Casual policy uk = ψk(p,ξξξk−1), with ξξξk = (ξ0, ξ1, . . . , ξk)
6 / 28

11. Stochastic optimal control problem
Optimisation problem:
V (p) = min
π={uk}k=N−1
k=0
E Vf (xN , ξN ) +
N−1
k=0
k(xk, uk, ξk) ,
s.t x0 = p,
xk+1 = Aξk
xk + Bξk
uk + wξk
,
where:
E[·]: conditional expectation wrt the product probability measure
Casual policy uk = ψk(p,ξξξk−1), with ξξξk = (ξ0, ξ1, . . . , ξk)
and Vf can encode constraints
6 / 28

12. Stage cost
The stage cost is a function k : Rn × Rm × Ωk → ¯R
k(xk, uk, ξk) = φk(xk, uk, ξk) + ¯φk(Fkxk + Gkuk, ξk),
where φ is real-valued, convex, smooth, e.g.,
φk(xk, uk, ξk) = xkQξk
xk + ukRξk
uk,
and ¯φ is proper, convex, lsc, and possibly non-smooth, e.g.,
¯φk(xk, uk, ξk) = δ(Fkxk + Gkuk | Yξk
).
7 / 28

13. Terminal cost
The terminal cost is a function Vf : Rn × ΩN → ¯R which can be written
as
Vf (x) = φN (xN , ξN ) + ¯φN (xN , ξN ),
where φN is real-valued, convex, smooth and ¯φN is proper, convex, lsc
and possibly non-smooth.
8 / 28

14. Total cost
The total cost function can be written as E(f(x) + g(Hx)), where
x = ((xk)k, (uk)k)
f(x) =
N−1
k=0
φk(xk, uk, ξk) + φN (xN , ξN ) + δ(x | X(p))
g(Hx) =
N−1
k=0
¯φk(Fkxk + Gkuk, ξk) + ¯φN (FN xN , ξN ),
and φk and φN are such that f is σ-strongly convex on its domain, that
is, the affine space which defines the system dynamics, i.e.,
X(p) = {x : xj
k+1 = Aj
kxi
k + Bj
kui
k + wj
k, j ∈ child(k, i)}
9 / 28

15. II. Proximal gradient algorithm

16. Proximal operator
We define a mapping proxγf : Rn → Rn of a closed, convex, proper
extended-real valued function f : Rn → ¯R as
proxγf (v) = arg min
x∈Rn
f(x) +
1
2γ
x − v 2
2 ,
for γ>0.
10 / 28

17. Proximal of the conjugate function
For a function f : Rn → ¯R we define its conjugate function to be1
f∗
(y) = sup
x∈Rn
{ y, x − f(x)}.
If we can compute proxγf , then we can also compute proxγf∗ using the
Moreau decomposition formula
v = proxγf (v) + γ proxγ−1f∗ (γ−1
v)
1
R. Rockafellar, Convex analysis. Princeton university press, 1972.
11 / 28

18. Optimisation problem
Consider the optimisation problem :
P = min
z=Hx
f(x) + g(z),
where f : Rn → ¯R is σ-strongly convex and g : Rm → ¯R is closed, proper
and convex. The Fenchel dual of this problem is:
D = min
y
f∗
(−H y) + g∗
(y),
where f∗ has Lipschitz-continuous gradient with constant 1/σ.
12 / 28

19. The basic algorithm
The proximal point algorithm applied to the dual optimisation problem
is defined by the recursion on dual variables2:
y0
= 0,
yν+1
= proxλg∗ (yν
+ λH f∗
(−H yν
)).
Using the conjugate subgradient theorem we can define
xν
:= f∗
(−H yν
) = arg min
z
{ z, H yν
+ f(z)}.
2
P. Combettes and J. Pesquet, “Proximal splitting methods in signal processing”, Fixed-Point Algorithms for Inverse
Problems in Science and Engineering, 2011.
13 / 28

20. Dual APG algorithm
Nesterov’s accelerated proximal gradient algorithm (APG) converges
at a rate of O(1/ν2) and is defined by the recursion:
vν
= yν
+ θν(θ−1
ν−1 − 1)(yν
− yν−1
),
xν
= arg min
z
{ z, H vν
+ f(z)},
zν
= proxλ−1g(λ−1
vν
+ Hxν
),
yν+1
= vν
+ λ(Hzv
− tv
),
θν+1 =
1
2
( θ4
ν + 4θ2
ν − θ2
ν).
14 / 28

21. Dual APG algorithm
Nesterov’s accelerated proximal gradient algorithm (APG) converges
at a rate of O(1/ν2) and is defined by the recursion:
vν
= yν
+ θν(θ−1
ν−1 − 1)(yν
− yν−1
),
xν
= arg min
z
{ z, H vν
+ f(z)},
zν
= proxλ−1g(λ−1
vν
+ Hxν
),
yν+1
= vν
+ λ(Hzv
− tv
),
θν+1 =
1
2
( θ4
ν + 4θ2
ν − θ2
ν).
14 / 28

22. Characteristics of the algoritm
Dual iterates converge at a rate of O(1/ν2)
An ergodic (averaged) primal iterate converges at a rate of O(1/ν2)3
Preconditioning is of crucial importance
Terminate the algorithm when the iterate (xν, zν) satisfies
f(x) + g(z) − P ≤ V
x − Hz ∞ ≤ g.
3
P. Patrinos and A. Bemporad, “An accelerated dual gradient-projection algorithm for embedded linear model predictive
control,” IEEE Trans. Aut. Contr., vol. 59, no. 1, pp. 18–33, 2014.
15 / 28

23. III. APG for Stochastic Optimal
Control Problems

24. Scenario tree formulation
16 / 28

25. Splitting for proximal formulation
We have
Ef(x)=
N−1
k=0
µ(k)
i=1
pi
kφ(xi
k,ui
k,i) +
µ(N)
i=1
pi
N φN (xi
N , i)+δ(x|X(p)),
Eg(Hx)=
N−1
k=0
µ(k)
i=1
pi
k
¯φ(Fi
kxi
k+Gi
kui
k,i)+
µ(N)
i=1
pi
N
¯φN (Fi
N xi
N , i),
17 / 28

26. Splitting for proximal formulation
We have
Ef(x)=
N−1
k=0
µ(k)
i=1
pi
kφ(xi
k,ui
k,i) +
µ(N)
i=1
pi
N φN (xi
N , i)+δ(x|X(p)),
Eg(Hx)=
N−1
k=0
µ(k)
i=1
pi
k
¯φ(Fi
kxi
k+Gi
kui
k,i)+
µ(N)
i=1
pi
N
¯φN (Fi
N xi
N , i),
where
X(p) = {x : xj
k+1 = Aj
kxi
k + Bj
kui
k + wj
k, j ∈ child(k, i)}
17 / 28

27. Computation of the dual gradient
Using dynamic programming, we solve the problem
xν
= arg min
z
{ z, H yν
+ Ef(z)}.
where
Ef(x)=
N−1
k=0
µ(k)
i=1
pi
kφ(xi
k,ui
k,i) +
µ(N)
i=1
pi
N φN (xi
N , i)+δ(x|X(p)),
18 / 28

28. Computation of the dual gradient
Factor step:
Performed once
Parallelisable
For time-invariant problems,
can be performed once offline
Algorithm 1 Solve step
qi
N ← yi
N , ∀i ∈ N[1,µ(N)], %Backward substitution
for k = N − 1, . . . , 0 do
for i ∈ µ(k) do {in parallel}
ui
k ← Φi
kyi
k + j∈child(k,i) Θ
j
k
q
j
k+1
+ σi
k
qi
k ← Di
k yi
k + j∈child(k,i) Λ
j
k
q
j
k+1
+ ci
k
end for
end for
x1
0 = p, %Forward substitution
for k = 0, . . . , N − 1 do
for i ∈ µ(k) do {in parallel}
ui
k ← Ki
kxi
k + ui
k
for j ∈ child(k, i) do {in parallel}
x
j
k+1
← A
j
k
xi
k + B
j
k
ui
k + w
j
k
end for
end for
end for
19 / 28

29. Computation of the dual gradient
Dynamic programming approach
Parallelisable across all nodes of a stage
The solve step involves only matrix-vector products
20 / 28

30. IV. Simulations

31. Simulation Results
Linear spring-mass system
GPU CUDA-C implementation (NVIDIA Tesla 2075)
Average and maximum runtime for a random sample of 100 initial
points
Compared against interior-point solver of Gurobi
21 / 28

32. Number of scenarios
22 / 28

33. Number of scenarios
log 2
(scenarios)
7 8 9 10 11 12 13
max.time(sec)
10 -2
10 -1
10 0
10 1
10 2
APG 0.005
APG 0.01
APG 0.05
Gurobi IP
23 / 28

34. Number of scenarios
In numbers:
8192 scenarios
6.39 · 105 primal variables
2.0 · 106 dual variables
Using g = V = 0.01 we are 40× faster (average)
24 / 28

35. Prediction horizon
prediction horizon
10 20 30 40 50 60
averagetime(sec)
10 -1
10 0
10 1
APG 0.005
APG 0.01
APG 0.05
Gurobi IP
25 / 28

36. Prediction horizon
prediction horizon
10 20 30 40 50 60
max.time(sec)
10 -1
10 0
10 1
APG 0.005
APG 0.01
APG 0.05
Gurobi IP
26 / 28

37. Prediction horizon
In numbers:
N = 60 and 500 scenarios
0.92 · 106 primal variables
2.0 · 106 dual variables
Using g = V = 0.01 we are 23× faster (average)
27 / 28

38. Stochastic MPC of drinking water networks
Recent results (to be submitted):
About 2 million primal variables
593 scenarios, N = 24
Gurobi requires 1329s on average
GPU APG runtime is about 58s
28 / 28

39. Thank you for your attention.
This work was financially supported by the EU FP7 research project EFFINET “Efficient Integrated Real-time
monitoring and Control of Drinking Water Networks,” grant agreement no. 318556.