This presentation proposes an approach to algorithmically synthesize control strategies for set-to-set transitions of discrete-time uncertain systems based on reachable set computations in a stochastic setting. For given Gaussian distributions of the initial states and disturbances, state sets wich are reachable to a chosen confidence level under the effect of time-variant control laws are computed by using principles of the ellipsoidal calculus. The proposed algorithm iterates over LMI-constrained semi-definite programming problems to compute probabilistically stabilizing controllers, while ellipsoidal input constraints are considered. An example for illustration is included.

1. Probabilistic Control of Uncertain Linear Systems Using
Stochastic Reachability
Leonhard Asselborn Olaf Stursberg
Control and System Theory
University of Kassel (Germany)
l.asselborn@uni-kassel.de
stursberg@uni-kassel.de
www.control.eecs.uni-kassel.de

2. Introduction
Problem:
• discrete-time uncertain linear system: xk+1 = Axk + Buk + Gvk
• Gaussian distribution of the initial states and the disturbances
• input constraints
• steer the uncertain linear system into a terminal region T with a defined
confidence δ
Solution Approach:
• forward propagation of ellipsoidal reachable sets Xδ
k with predefined
confidence
• offline algorithmic synthesis by semidefinite programming (SDP)
• time-variant control laws for set-to-set transitions
Xδ
0
Xδ
N
x2
x1
T
Introduction Problem Definition Method Example Conclusion Appendix 2

3. Relevant literature (excerpt):
• Girard [2005]: uncertain linear systems, reachability computations, no
controller synthesis
• Rakovic, Kerrigan, Mayne, and Lygeros [2006]: linear system dynamics,
polytopic sets, input-dependend disturbance, no controller synthesis
• Cannon, Kouvaritakis, Ng [2009]: polytopic confidence sets, finite support
of random disturbance, model predictive control
• Althoff, Stursberg, Buss [2009]: linear system dynamics, verification, no
controller synthesis, confidence sets
• Asselborn, Gross, Stursberg [2013]: nonlinear system dynamics with
disturbances, controller synthesis
• Bashkirtseva, Ryashko [2013]: nonlinear system dynamics with
disturbances, no controller synthesis, confidence sets
Introduction Problem Definition Method Example Conclusion Appendix 3

4. Basics: Sets, Distribution and Dynamic System
1. Set representation:
Ellipsoid: E = ε(q, Q) = x ∈ Rn
|(x − q)T
Q−1
(x − q) ≤ 1
with q ∈ Rn
, Q ∈ Rn×n
2. Multivariate Normal Distribution
ξ ∼ N(µ, Ω)
The sum of two Gaussian variables ξ1 ∼ N(µ1, Ω1) and ξ2 ∼ N(µ2, Ω2) is
again a Gaussian variable:
ξ1 + ξ2 ∼ N(µ1 + µ2, Ω1 + Ω2)
3. Dynamical system:
xk+1 = Axk + Buk + Gvk, k ∈ {0, 1, 2, . . .}
x0 ∼ N(q0, Q0), xk ∈ Rn
vk ∼ N(0, Σ), vk ∈ Rn
uk ∈ ε(p, P) = U ⊆ Rm
Introduction Problem Definition Method Example Conclusion Appendix 4

5. Basics: Probabilistic Reachable Sets with Confidence δ
• Surfaces of equal density for ξ ∼ N(µ, Ω)
(Krzanowski and Marriott [1994]):
(ξ − µ)T
Ω−1
(ξ − µ) = c
c is a χ2
-distributed random variable.
δ := Pr (ξ ∈ ε(µ, Ωc)) = Fχ2 (c, n)
cumulativ distribution function
• Initial state confidence ellipsoid:
Xδ
0 := ε(q0, Q0c) with Pr(x0 ∈ Xδ
0 ) = δ
contour of pdf
samples of ξ ∼ N (µ, Ω)
ξ2
ξ1
• Evolution of the initial state distribution:
qk+1 = Aqk + Buk, Qk+1 = AQkAT
+ GΣGT
, Xδ
k+1 := ε(qk+1, Qk+1c
=:Qδ
k+1
)
Xδ
k+1 is the confidence ellipsoid for xk+1 with confidence δ.
Introduction Problem Definition Method Example Conclusion Appendix 5

6. Stochastic Stability
Stability with confidence δ
The uncertain linear system is said to be stable with confidence δ, if finite
parameters ¯q ∈ Rn
and ¯Q ∈ Rn×n
exist for any initial condition x0 ∈ Xδ
0 and
any vk ∈ V δ
= ε(0, Σc) holds:
||qN || ≤ ||¯q||, ||QN || ≤ || ¯Q||,
and for any 0 ≤ k ≤ N − 1:
||qk+1|| ≤ ||qk||, ||Qk+1|| ≤ ||Qk||.
Interpretation:
• Mean value qk has to converge to a finite neighbourhood of the origin.
• The covariance matrix Qk converges, such that the confidence ellipsoid is
of decreasing size over k (while rotation is still possible).
Introduction Problem Definition Method Example Conclusion Appendix 6

7. Problem Definition
Problem
Determine a control law uk = κ(xk, k), for which it holds that:
• uk ∈ U and xk ∈ X δ
k ∀ k ∈ {0, 1, . . . , N − 1}, N ∈ N
• the closed-loop system is rendered stable with confidence δ,
• Xδ
N ⊆ T in a finite number of N steps.
Thus, any initial state x0 ∈ Xδ
0 has to be transferred into the terminal set T
with probability δ after N steps.
• Selected structure of the control law for xk ∈ Xδ
k = ε(qk, Qkc):
uk = −Kkxk
• Closed loop dynamic:
xk+1 = Axk + Buk + Gvk = (A − BKk)
:=Acl,k
xk + Gvk.
Introduction Problem Definition Method Example Conclusion Appendix 7

8. Solution based on SDP: Covariance Matrix and Expected Value
Convergence of the covariance matrix Qk:
• Covariance matrix of the pdf N(qk+1, Qk+1):
Qk+1 = Acl,kQkAT
cl,k + GΣGT
• constrained by a symmetric matrix to be optimized in SDP:
Sk+1 ≥ Qk+1.
• or with the Schur complement:


Sk+1 Acl,kQk GΣ
QkAT
cl,k Qk 0
ΣGT
0 Σ

 ≥ 0
Convergence of the expected value qk
• is achieved implicitly by the convergence of the covariance matrix.
|λi(Acl,k)| < 1, i ∈ {1, . . . , n}
Introduction Problem Definition Method Example Conclusion Appendix 8

9. Solution based on SDP: Input Constraint
• Set-valued mapping of the control law:
¯Uk = −KkXδ
k = ε(−Kkqk, KkQδ
kKT
k ) ⊆ U
• Over-approximation of the shape matrix:
¯Pk ≥ KkQδ
kKT
k → ε(−Kkqk, ¯Pk)
!
⊆ ε(p, P)
• Matrix inequality for an ellipse-in-ellipse
problem (Lemma 3.7.3 in Boyd et al.
[1994]):



−P p + Kkqk
¯P
1/2
k
(p + Kkqk)T
λ − 1 0
¯P
1/2
k 0 −λI


 ≤ 0, λ > 0
• Overapproximation of ¯P
1/2
k :
˜Pk ≥ ¯P
1/2
k
¯Uk = ε(−Kkqk, KkQδ
kKT
k )
ε(−Kkqk, ¯Pk)
ε(−Kkqk, ˜P 2
k )
U = ε(p, P )
Introduction Problem Definition Method Example Conclusion Appendix 9

10. Formulation of the SDP
Semidefinite program to be solved in any k:
min
Sk+1,λ, ˜Pk, ¯Pk,Kk
trace(Sk+1)
ellipsoidal shape convergence:






Sk+1 Acl,kQk GΣ
QkAT
cl,k Qk 0
ΣGT
0 Σ


 ≥ 0
trace(Sk+1) ≤ trace(Qk)
input constraint:



¯Pk KkQδ
k
Qδ
kKT
k Qδ
k
≥ 0
0 ≤ ˜Pk ≤ P1/2



−P p + Kkqk
˜Pk
(p + Kkqk)T
λ − 1 0
˜Pk 0 −λI


 ≤ 0
¯Pk ≤ α2
i P − j,h
∂g( ˜Pk)
∂ ˜pk,jh ˜Pk=PL
eT
j ( ˜Pk − αiP)eh
Introduction Problem Definition Method Example Conclusion Appendix 10

11. Algorithm for Controller Synthesis (1)
Algorithm
Probabilistic Ellipsoidal Control Algorithm (PECA)
Given: (A, B, G), x0 ∼ N(q0, Q0), vk ∼ N(0, Σ), and U = ε(p, P), as well as
T, δ, γmin, αi ∈ [0, 1], i ∈ {1, . . . , nα}.
Define: k := 0, γ0 = γmin
while Xδ
k T and γk ≥ γmin do
• compute the confidence ellipsoid Xδ
k for N(qk, Qk)
• solve the optimization problem for any αi and choose the best solution
according to the cost function
if no feasible Kk is found do
stop algorithm (synthesis failed)
end if
• compute the distribution of xk+1 ∼ N(qk+1, Qk+1)
• compute γk+1 = ||qk+1 − qk||
• k := k + 1
end while
Introduction Problem Definition Method Example Conclusion Appendix 11

12. Termination of PECA
Lemma
The control problem with a confidence δ, an initialization x0 ∼ N(q0, Q0),
vk ∼ N(0, Σ), and uk ∈ U ∀ k is successfully solved with selected parameters
γmin and αi i = {1, . . . , nα}, if PECA terminates in N steps with Xδ
N ⊆ T.
Proof (sketch):
• Successful termination of the control algorithm implies that Xδ
N ∈ T and
consequently xN ∈ Xδ
N ⊆ T holds for all initial states x0 ∈ Xδ
0 and all
disturbances vk ∼ N(0, Σ).
• By construction Pr(xk ∈ Xδ
k) = δ is ensured for each time step k.
• A successful termination shows stability with confidence δ; ¯q and ¯Q are
provided by the terminal region T.
Introduction Problem Definition Method Example Conclusion Appendix 12

13. Example in 3-D: Problem (1)
Initial distribution and disturbance:
x0 ∼ N(q0, Q0) with q0 =


2
9
2

 , Q0 =


0.2 0.5 0
0.5 0.1 0.5
0 0.5 0.2

 (1)
vk ∼ N(0, Σ) with Σ =


0.2 0.5 0
0.5 0.1 0.5
0 0.5 0.2

 (2)
System dynamics:
xk+1 =


0.87 0.06 −0.03
0 1.09 1.20
0 0.06 0.91

 xk
+


0.64 0.0003
−0.24 0.34
0.62 0.61

 uk +


0.1 0 0
0 0.2 0
0 0 0.3

 vk (3)
Introduction Problem Definition Method Example Conclusion Appendix 13

14. Example in 3-D: Problem (2)
Input constraints:
uk ∈ U = ε
0
0
,
30 15
15 30
(4)
Target set
T = ε




0
0
0

 ,


0.96 0.64 0.24
0.64 0.8 0.64
0.24 0.64 0.96



 (5)
PECA is parametrized by: δ = 0.95, γmin = 0.01, and αi = 1
6
i for i ∈ 0, . . . , 6.
Introduction Problem Definition Method Example Conclusion Appendix 14

15. 3-Dimensional Example: Results (1)
Sample trajectory of the linear dynamic
Target set T
Confidence reachable sets Xδ
k, δ = 0.95
x3
x2
x1
−2
0
2
4
6 −5
0
5
10
15
−4
−3
−2
−1
0
1
2
3
4
• Termination after 20 time
steps in 80s using 2.8 Ghz
Quad-Core CPU.
• Implementation with Matlab
7.12.0, YALMIP 3.0, SeDuMi
1.3, and ellipsoidal toolbox ET
(Kurzhanskiy and Varaiya
[2006]).
k P r(xk ∈ Xδ
k) k P r(xk ∈ Xδ
k)
0 95.0% 10 96.4%
1 96.2% 11 95.0%
2 95.3% 12 95.8%
3 95.4% 13 94.6%
4 94.7% 14 94.7%
5 95.0% 15 94.1%
6 95.2% 16 95.2%
7 95.6% 17 95.6%
8 95.7% 18 94.7%
9 95.8% 19 95.4%
Introduction Problem Definition Method Example Conclusion Appendix 15

16. 3-Dimensional Example: Results (2)
Sample trajectory of the linear dynamic
Target set T
Confidence reachable sets Xδ
k, δ = 0.8
x3
x2
x1
−2
0
2
4
6 −5
0
5
10 15
−4
−3
−2
−1
0
1
2
3
• Modified confidence:
δ = 0.8
• Termination after 15 time
steps.
• Smaller reachable sets lead
to more freedom in
choosing Kk within the
input constraints.
Introduction Problem Definition Method Example Conclusion Appendix 16

17. Conclusion and Outlook
Summary:
• Algorithm for the stabilization with confidence δ of an uncertain
discrete-time linear system
• Control law synthesis based on probabilistic reachability analysis
• Explicit consideration of input constraints
• Stabilization problem reformulated as an iterative problem
• Resorts, if PECA terminates without success: adjust δ, γmin
• Numerical example to show a successful application
Future work:
• Extension to switched dynamics (ADHS’15)
• Different confidence levels for the state and disturbance distribution
• Chance constraints for the state
Introduction Problem Definition Method Example Conclusion Appendix 17

18. References
[1] Althoff, M:
Reachability analysis and its application to the safety assessment of autonomous cars;
Ph.D. thesis, Technische Universit¨at M¨unchen, 2010.
[2] Althoff, M., Stursberg, O., and Buss, M.:
Safety assessment for stochastic linear systems using enclosing hulls of probability density functions;
In 10th European Control Conf., p 625-630, 2009.
[3] Asselborn, L., Jilg, M., and Stursberg, O.:
Control of uncertain hybrid nonlinear systems using particle filters;
In 4th IFAC Conf. on Analysis and Design of Hybrid Systems, p 436-441, 2012.
[4] Asselborn, L., Gross, D., and Stursberg, O.:
Control of uncertain nonlinear systems using ellipsoidal reachability calculus;
In 9th IFAC Symp. on Nonlinear Control Systems, p 50-5, 2012.
[5] Astr¨om,K.J.:
Introduction to stochastic control theory;
Courier Corporation, 2012.
[6] Bashkirtseva, I. and Ryashko, L.:
Attainability analysis in the problem of stochastic equilibria synthesis for nonlinear discrete systems;
Athena Scientific, 2007.
[7] Bertsekas, D.P. and Shreve, S.E.:
Stochastic Optimal Control: The Discrete-Time Case;
Int. Journal of Applied Mathematics and Computer Science, 23(1), p 5-16, 2013.
[8] Boyd, S.P., El Ghaoui, L., Feron, E., and Balakrishnan, V.:
Linear Matrix Inequalities in System and Control Theory;
SIAM Studies in Applied Mathematics, 1994.
[9] Cannon, M., Cheng, Q., Kouvaritakis, B., and Rakovic, S.V.:
Stochastic tube mpc with state estimation;
Automatica, 48(3), p 536-541, 2012.
[10] Cannon, M., Kouvaritakis, B., and Ng, D.:
Probabilistic tubes in linear stochastic model predictive control.;
Systems & Control Letters, 58(10), p 747-753, 2009.
Introduction Problem Definition Method Example Conclusion Appendix 18

19. References
[11] Do, C.B.:
More on multivariate gaussians;
URL http://cs229.stanford.edu/section/more on gaussians.pdf. [Online; accessed 09-December-2014], 2014.
[12] Dueri, D., Acikmese, B., Baldwin, M., and Erwin, R.S.:
Finite-horizon controllability and reachability for deterministic and stochastic linear control systems with convex constraints;
In American Control Conf., p 5016-5023, 2014.
[13] Girard, A.:
Reachability of uncertain linear systems using zonotopes;
In Hybrid Systems: Computation and Control, p 291-305. Springer, 2005.
[14] Girard, A., Le Guernic, C., and Maler, O.:
Efficient computation of reachable sets of linear time-invariant systems with inputs;
In Hybrid Systems: Computation and Control, 2006.
[15] Harris, M.W. and Ac?kmese, B.:
Lossless convexification of non-convex optimal control problems for state constrained linear systems;
Automatica, 50(9), p 2304-2311, 2014.
[16] Kappen, H.J.:
An introduction to stochastic control theory, path integrals and reinforcement learning;
In Cooperative Behavior in Neural Systems, volume 887, p 149-181, 2007.
[17] Kouvaritakis, B., Cannon, M., Rakovic, S.V., and Cheng, Q.:
Explicit use of probabilistic distributions in linear predictive control;
Automatica, 46(10), p 1719-1724, 2010.
[18] Krzanowski, W.J. and Marriott, F.H.C.:
Multivariate Analysis: Distributions, Ordination and Inference;
Wiley Interscience, volume 2, 1994.
[19] Kurzhanski, A. and Varaiya, P.:
Ellipsoidal Calculus for Estimation and Control;
Birkh¨auser, 1996.
[20] A. A. Kurzhanskiy, P. Varaiya:
Ellipsoidal Toolbox;
Technical Report UCB/EECS-2006-46, EECS Department, University of California, Berkeley. URL
http://code.google.com/p/ellipsoids, 2006.
Introduction Problem Definition Method Example Conclusion Appendix 19

20. References
[21] Le Guernic, C. and Girard, A.:
Reachability analysis of linear systems using support functions;
Nonlinear Analysis: Hybrid Systems, 4(2), p 250-262, 2010.
[22] Pham, H. et al. :
On some recent aspects of stochastic control and their applications;
Probability Surveys, 2(506-549), p 3-4, 2005.
[23] Rakovic, S.V., Kerrigan, E.C., Mayne, D.Q., and Lygeros, J.:
Reachability Analysis of Discrete-Time Systems with Disturbances;
IEEE Trans. on Automatic Control, Vol. 51, pp. 546-561,2006.
[24] Stursberg, O. and Krogh, B. :
Efficient Representation and Computation of Reachable Sets for Hybrid Systems;
Hybrid Systems: Control and Computation, Springer-LNCS, Vol. 2623, pp 482-497, 2003
Introduction Problem Definition Method Example Conclusion Appendix 20

21. Input constraint (1)
˜Pk ≥ ¯P
1/2
k
g( ˜Pk) = ¯Pk − ˜P2
k ≤ 0.
A first-order Taylor approximation:
g( ˜Pk) ≈ ˜g( ˜Pk) = g(PL) + dg( ˜Pk) ≤ 0.
˜g( ˜Pk) = g(PL) +
m
j,h
∂g( ˜Pk)
∂ ˜pk,jh ˜Pk=PL
d˜pk,jh
= g(PL) +
∂g( ˜Pk)
∂ ˜pk,11 ˜Pk=PL
d˜pk,11 + . . . +
∂g( ˜Pk)
∂ ˜pk,mm ˜Pk=PL
d˜pk,mm
= ¯Pk − P2
L +
∂g( ˜Pk)
∂ ˜pk,11 ˜Pk=PL
d˜pk,11 + . . . +
∂g( ˜Pk)
∂ ˜pk,mm ˜Pk=PL
d˜pk,mm,
with:
d˜pk,jh = ˜pk,jh − pL,jh.
Introduction Problem Definition Method Example Conclusion Appendix 21

22. Input constraint (2)
The linearization point PL,k is limited by the shape matrix of the input ellipsoid
U:
PL = αP1/2
, α ∈ [0, 1]
With this expression, the resulting constraints for ¯Pk and ˜Pk are:
0 ≤ ˜Pk ≤ P1/2
,
¯Pk ≤ α
2
i P −
∂g( ˜Pk)
∂ ˜pk,11 ˜Pk=PL
(˜pk,11 − αip11) − . . . −
∂g( ˜Pk )
∂ ˜pk,mm ˜Pk=PL
(˜pk,mm − αipmm)
In order to obtain a matrix inequality which is linear in ¯Pk and ˜Pk,:
¯Pk ≤ α2
i P −
m
jh
∂g( ˜Pk)
∂ ˜pk,jh ˜Pk=PL
eT
j ( ˜Pk − αiP)eh
Introduction Problem Definition Method Example Conclusion Appendix 22