Upcoming SlideShare
×

# Markov Tutorial CDC Shanghai 2009

16,197 views
16,100 views

Published on

A crash coarse in stochastic Lyapunov theory for Markov processes (emphasis is on continuous time)

https://netfiles.uiuc.edu/meyn/www/spm_files/MarkovTutorial/MarkovTutorialUCSB2010.html

3 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
16,197
On SlideShare
0
From Embeds
0
Number of Embeds
13,530
Actions
Shares
0
111
0
Likes
3
Embeds 0
No embeds

No notes for slide

### Markov Tutorial CDC Shanghai 2009

1. 1. Lyapunov functions, value functions, and performance bounds Sean Meyn Department of Electrical and Computer Engineering University of Illinois and the Coordinated Science Laboratory Joint work with R. Tweedie, I. Kontoyiannis, and P. Mehta Supported in part by NSF (ECS 05 23620, and prior funding), and AFOSR
2. 2. Objectives Nonlinear state space model ≡ (controlled) Markov process, state process X Typical form: dX(t) = f (X(t), U (t)) dt + σ(X(t), U (t)) dW (t) noise control
3. 3. Objectives Nonlinear state space model ≡ (controlled) Markov process, state process X Typical form: dX(t) = f (X(t), U (t)) dt + σ(X(t), U (t)) dW (t) noise control Questions: For a given feedback law, • Is the state process stable? • Is the average cost finite? E[c(X(t), U (t))] • Can we solve the DP equations? min c(x, u) + Du h∗ (x) = η ∗ u • Can we approximate the average cost η∗? The value function h∗ ?
4. 4. Outline Markov Models P t (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C π(f ) < ∞ Representations Lyapunov Theory DV (x) ≤ −f (x) + bIC (x) Conclusions
5. 5. I Markov Models
6. 6. Notation Markov chain: X = X(t) : t ≥ 0 Countable state space, X Transition semigroup, P t (x, y) = P X(s + t) = y X(s) = x , x, y ∈ X
7. 7. Notation: Generators & Resolvents Markov chain: X = X(t) : t ≥ 0 Countable state space, X Transition semigroup, P t (x, y) = P X(s + t) = y X(s) = x , x, y ∈ X Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t
8. 8. Notation: Generators & Resolvents Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t Rate matrix: Dh (x) = Q(x, y)h(y) P t = eQt y
9. 9. α µ Example: MM1 Queue  x + 1 Prob εα  Sample paths: X(t + ε) ≈ x − 1 Prob εµ   x Prob 1 − ε(α + µ) Rate matrix:   −α α 0 0 0 0 ···  µ −α − µ  α 0 0 0 · · ·   0  µ −α − µ α 0 0 · · ·   Q= 0 0 µ −α − µ α 0 · · ·   0  0 0 µ −α − µ α · · ·   0  0 0 0 µ −α − µ · · ·  . . . . . . . . . . . . . . . . . .
10. 10. 2 2 σW = 0 σW = 1 Example: O-U Model Sample paths: dX(t) = AX(t) dt + B dW (t) A n × n, Bn × 1, W standard BM 2 Generator: Dh (x) = (Ax)T h (x) + B T h (x)B
11. 11. 2 2 σW = 0 σW = 1 Example: O-U Model Sample paths: dX(t) = AX(t) dt + B dW (t) A n × n, Bn × 1, W standard BM 2 Generator: Dh (x) = (Ax)T h (x) + B T h (x)B h quadratic, h(x) = 1 xT P x 2 h (x) = P x 2 h (x) = P Dh (x) = 1 xT (P A + AT P )x + B T P B 2
12. 12. Notation: Generators & Resolvents Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t Rate matrix: Dh (x) = Q(x, y)h(y) P t = eQt y Resolvent: ∞ Rα = e−αt P t 0
13. 13. Notation: Generators & Resolvents Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t Rate matrix: Dh (x) = Q(x, y)h(y) P t = eQt y Resolvent: Resolvent equations: ∞ Rα = e−αt P t Rα = [ Iα − Q]−1 0 QRα = Rα Q = αRα − I
14. 14. Notation: Generators & Resolvents Motivation: Dynamic programming. For a cost function c, hα (x) = Rα c (x) = Rα (x, y)c(y) y∈X ∞ = eαt E[c(X(t)) X(0) = x] dt 0 Discounted-cost value function
15. 15. Notation: Generators & Resolvents Motivation: Dynamic programming. For a cost function c, hα (x) = Rα c (x) = Rα (x, y)c(y) y∈X ∞ = eαt E[c(X(t)) X(0) = x] dt 0 Discounted-cost value function Resolvent equation = dynamic programming equation, c + Dhα = αhα
16. 16. Notation: Steady State Distribution Invariant (probability) measure π: X is stationary. In particular, X(t) ∼ π, t≥0
17. 17. Notation: Steady State Distribution Invariant (probability) measure π: X is stationary. In particular, X(t) ∼ π, t≥0 Characterizations: π(x)P t (x, y) = π(y) x∈X α π(x)Rα (x, y) = π(y), α > 0 x∈X π(x)Q(x, y) = 0 x∈X y ∈X
18. 18. Notation: Relative Value Function Invariant measure π, cost function c , steady-state mean η Relative value function: ∞ h(x) = E[c(X(t)) − η X(0) = x] dt 0
19. 19. Notation: Relative Value Function Invariant measure π, cost function c , steady-state mean η Relative value function: ∞ h(x) = E[c(X(t)) − η X(0) = x] dt 0 Solution to Poisson’s equation (average-cost DP equation): c + Dh = η
20. 20. II Representations π ∝ ν[I − (R − s ⊗ ν)]−1 h = [I − (R − s ⊗ ν)]−1 c ˜
21. 21. Irreducibility ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x
22. 22. Small Functions and Small Measures ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x Small functions and measures: For a function s and probability ν, R(x, y) ≥ s(x)ν(y), x, y ∈ X ∞ R= e−t P t dt 0
23. 23. Small Functions and Small Measures ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x Small functions and measures: For a function s and probability ν, R(x, y) ≥ s(x)ν(y), x, y ∈ X Resolvent dominates rank-one matrix, R= ∞ e−t P t dt 0 R ≥s⊗ν
24. 24. Small Functions and Small Measures ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x Small functions and measures: For a function s and probability ν, R(x, y) ≥ s(x)ν(y), x, y ∈ X Resolvent dominates rank-one matrix, R= ∞ e−t P t dt 0 R ≥s⊗ν ψ-Irreducibility justi es assumption: s(x) > 0 for all x and WLOG, ν = δx ∗ , where ψ(x∗ ) > 0
25. 25. α µ Example: MM1 Queue R(x, y) > 0 for all x and y (irreducible in usual sense) Conclusion: R(x, y ) ≥ s(x)ν(y ) where s(x) := R(x, 0) ν := δ0
26. 26. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) R(0, . ) Gaussian Full rank if and only if (A, B) is controllable. Conclusion: Under controllability, for any m, there is ε s.t., R(x, A) ≥ s(x)ν(A) all x and A where s(x) = ε I x ≤m ν(A) uniform on x ≤m
27. 27. Potential Matrix ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 G = [I − (R − s ⊗ ν)]−1
28. 28. Representation of π ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 π ∝ νG νG (y) = ν(x)G(x, y) x∈X
29. 29. Representation of h ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 h = RG c ˜ + constant c(x) = c(x) − η ˜ G˜ (y) = c G(x, y)˜(y) c η= π(x)c(x) y∈X y∈X
30. 30. Representation of h ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 h = RG c ˜ + constant c(x) = c(x) − η ˜ G˜ (y) = c G(x, y)˜(y) c η= π(x)c(x) y∈X y∈X If sum converges, then Poisson’s equation is solved: c(x) + Dh (x) = η
31. 31. III Lyapunov Theory P n (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C π(f ) < ∞ ∆V (x) ≤ −f (x) + bIC (x)
32. 32. Lyapunov Functions DV ≤ −g + bs
33. 33. Lyapunov Functions DV ≤ −g + bs General assumptions: V : X → (0,∞) g : X → [1, ∞) b < ∞, s small e.g., s (x) = IC (x), C nite
34. 34. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs
35. 35. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs Since s⊗ν is non-negative, −[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs G−1
36. 36. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs Since s⊗ν is non-negative, −[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs G−1 More positivity, V ≥ GRg − bGRs Some algebra, GR = G(R − s ⊗ ν) + (Gs) ⊗ ν ≥ G − I Gs ≤ 1
37. 37. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs Since s⊗ν is non-negative, −[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs G−1 More positivity, V ≥ GRg − bGRs Some algebra, GR = G(R − s ⊗ ν) + (Gs) ⊗ ν ≥ G − I Gs ≤ 1 General bound: GRg ≤ V + 2b Gg ≤ V + g + 2b
38. 38. Existence of π DV ≤ −g + bs Condition (V2) DV ≤ −1 + bs Representation: π ∝ νG Bound: GRg ≤ V + 2b =⇒ G(x, X) ≤ V (x) + 2b Conclusion: π exists as a probability measure on X
39. 39. Existence of moments DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: π ∝ νG Bound: Gg ≤ V + g + 2b
40. 40. Existence of moments DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: π ∝ νG Bound: Gg ≤ V + g + 2b Conclusion: π exists as a probability measure on X and the steady-state mean is nite, π(g) := π(x)g(x) ≤ b x∈X
41. 41. α α µ Example: MM1 Queue ρ = µ Linear Lyapunov function, V (x) = x ∞ DV (x) = Q(x, y)y y=0 = α(x + 1) + µ(x − 1) − (α + µ)x = −(µ − α) x>0 Conclusion: (V2) holds if and only if ρ < 1
42. 42. α α µ Example: MM1 Queue ρ = µ QuadraticLyapunov function, V (x) = x 2 ∞ DV (x) = Q(x, y)y 2 y=0 = α(x + 1)2 + µ(x − 1)2 − (α + µ)x2 = α(x2 + 2x + 1) + µ(x2 − 2x + 1)2 − (α + µ)x2 = −2(µ − α)x + α + µ Conclusion: (V3) holds, g(x) = 1 + x if and only if ρ < 1
43. 43. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) h quadratic, h(x) = 1 xT P x 2 h (x) = P x 2 h (x) = P Dh (x) = 1 xT (P A + AT P )x + B T P B 2 Suppose that P > 0 solves the Lyapunov equation, P A + AT P = -I
44. 44. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) h quadratic, h(x) = 1 xT P x 2 h (x) = P x 2 h (x) = P Dh (x) = 1 xT (P A + AT P )x + B T P B 2 Suppose that P > 0 solves the Lyapunov equation, P A + AT P = -I Then (V3) follows from the identity, 2 2 2 Dh (x) = − 1 x 2 + σX , σX = B T P B
45. 45. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) The function h(x) = 1 xT P x solves Poisson’s equation, 2 1 2 Dh = −g + η g(x) = 2 x 2 η = σX Suppose that P > 0 solves the Lyapunov equation, P A + AT P = -I Then (V3) follows from the identity, 2 2 2 Dh (x) = − 1 x 2 + σX , σX = B T P B
46. 46. Poisson’s Equation DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: h = RG c ˜ + constant c(x) = c(x) − η ˜ Bound: RGg ≤ V + 2b =⇒ RGg (x) ≤ V (x) + 2b
47. 47. Poisson’s Equation DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: h = RG c ˜ + constant c(x) = c(x) − η ˜ Bound: RGg ≤ V + 2b =⇒ RGg (x) ≤ V (x) + 2b Conclusion: If c is bounded by g, then h is bounded, h(x) ≤ V (x) + 2b
48. 48. α α µ Example: MM1 Queue ρ = µ Poisson’s equation with g (x) = x Dh = −g + η We have (V3) with V a quadratic function of x: Recall, with h (x) = x 2 Dh (x) = −2(µ − α)x + α + µ x>0
49. 49. α α µ Example: MM1 Queue ρ = µ Poisson’s equation with g (x) = x Dh = −g + η Solved with x2 + x ρ h(x) = 1 2 µ−α η= 1−ρ
50. 50. IV Conclusions
51. 51. P t (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C Final words π(f ) < ∞ DV (x) ≤ −f (x) + bIC (x) Just as in linear systems theory, Lyapunov functions provide a characterization of system properties, as well as a practical verification tool
52. 52. P t (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C Final words π(f ) < ∞ DV (x) ≤ −f (x) + bIC (x) Just as in linear systems theory, Lyapunov functions provide a characterization of system properties, as well as a practical verification tool Much is left out of this survey - in particular, • Converse theory • Limit theory • Approximation techniques to construct Lyapunov functions or approximations to value functions • Application to controlled Markov processes, and approximate dynamic programming
53. 53. References [1,4] ψ-Irreducible foundations [2,11,12,13] Mean- eld models, ODE models, and Lyapunov functions [1,4,5,9,10] Operator-theoretic methods. See also appendix of [2] [3,6,7,10] Generators and continuous time models [1] S. P. Meyn and R. L. Tweedie. Markov chains and stochastic [9] I. Kontoyiannis and S. P. Meyn. Spectral theory and limit stability. Cambridge University Press, Cambridge, second theorems for geometrically ergodic Markov processes. Ann. edition, 2009. Published in the Cambridge Mathematical Appl. Probab., 13:304–362, 2003. Presented at the INFORMS Library. Applied Probability Conference, NYC, July, 2001. [2] S. P. Meyn. Control Techniques for Complex Networks. Cam- [10] I. Kontoyiannis and S. P. Meyn. Large deviations asymptotics bridge University Press, Cambridge, 2007. Pre-publication and the spectral theory of multiplicatively regular Markov edition online: http://black.csl.uiuc.edu/˜meyn. processes. Electron. J. Probab., 10(3):61–123 (electronic), [3] S. N. Ethier and T. G. Kurtz. Markov Processes : Charac- 2005. terization and Convergence. John Wiley & Sons, New York, [11] W. Chen, D. Huang, A. Kulkarni, J. Unnikrishnan, Q. Zhu, 1986. P. Mehta, S. Meyn, and A. Wierman. Approximate dynamic [4] E. Nummelin. General Irreducible Markov Chains and Non- programming using ﬂuid and diffusion approximations with negative Operators. Cambridge University Press, Cambridge, applications to power management. Accepted for inclusion in 1984. the 48th IEEE Conference on Decision and Control, December [5] S. P. Meyn and R. L. Tweedie. Generalized resolvents 16-18 2009. and Harris recurrence of Markov processes. Contemporary [12] P. Mehta and S. Meyn. Q-learning and Pontryagin’s Minimum Mathematics, 149:227–250, 1993. Principle. Accepted for inclusion in the 48th IEEE Conference [6] S. P. Meyn and R. L. Tweedie. Stability of Markovian on Decision and Control, December 16-18 2009. processes III: Foster-Lyapunov criteria for continuous time [13] G. Fort, S. Meyn, E. Moulines, and P. Priouret. ODE processes. Adv. Appl. Probab., 25:518–548, 1993. methods for skip-free Markov chain stability with applications [7] D. Down, S. P. Meyn, and R. L. Tweedie. Exponential to MCMC. Ann. Appl. Probab., 18(2):664–707, 2008. and uniform ergodicity of Markov processes. Ann. Probab., 23(4):1671–1691, 1995. [8] P. W. Glynn and S. P. Meyn. A Liapounov bound for solutions of the Poisson equation. Ann. Probab., 24(2):916–931, 1996. See also earlier seminal work by Hordijk, Tweedie, ... full references in [1].