Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Like this presentation? Why not share!

16,197 views

16,100 views

16,100 views

Published on

See also the survey for models in discrete time,

https://netfiles.uiuc.edu/meyn/www/spm_files/MarkovTutorial/MarkovTutorialUCSB2010.html

No Downloads

Total views

16,197

On SlideShare

0

From Embeds

0

Number of Embeds

13,530

Shares

0

Downloads

111

Comments

0

Likes

3

No embeds

No notes for slide

- 1. Lyapunov functions, value functions, and performance bounds Sean Meyn Department of Electrical and Computer Engineering University of Illinois and the Coordinated Science Laboratory Joint work with R. Tweedie, I. Kontoyiannis, and P. Mehta Supported in part by NSF (ECS 05 23620, and prior funding), and AFOSR
- 2. Objectives Nonlinear state space model ≡ (controlled) Markov process, state process X Typical form: dX(t) = f (X(t), U (t)) dt + σ(X(t), U (t)) dW (t) noise control
- 3. Objectives Nonlinear state space model ≡ (controlled) Markov process, state process X Typical form: dX(t) = f (X(t), U (t)) dt + σ(X(t), U (t)) dW (t) noise control Questions: For a given feedback law, • Is the state process stable? • Is the average cost finite? E[c(X(t), U (t))] • Can we solve the DP equations? min c(x, u) + Du h∗ (x) = η ∗ u • Can we approximate the average cost η∗? The value function h∗ ?
- 4. Outline Markov Models P t (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C π(f ) < ∞ Representations Lyapunov Theory DV (x) ≤ −f (x) + bIC (x) Conclusions
- 5. I Markov Models
- 6. Notation Markov chain: X = X(t) : t ≥ 0 Countable state space, X Transition semigroup, P t (x, y) = P X(s + t) = y X(s) = x , x, y ∈ X
- 7. Notation: Generators & Resolvents Markov chain: X = X(t) : t ≥ 0 Countable state space, X Transition semigroup, P t (x, y) = P X(s + t) = y X(s) = x , x, y ∈ X Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t
- 8. Notation: Generators & Resolvents Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t Rate matrix: Dh (x) = Q(x, y)h(y) P t = eQt y
- 9. α µ Example: MM1 Queue x + 1 Prob εα Sample paths: X(t + ε) ≈ x − 1 Prob εµ x Prob 1 − ε(α + µ) Rate matrix: −α α 0 0 0 0 ··· µ −α − µ α 0 0 0 · · · 0 µ −α − µ α 0 0 · · · Q= 0 0 µ −α − µ α 0 · · · 0 0 0 µ −α − µ α · · · 0 0 0 0 µ −α − µ · · · . . . . . . . . . . . . . . . . . .
- 10. 2 2 σW = 0 σW = 1 Example: O-U Model Sample paths: dX(t) = AX(t) dt + B dW (t) A n × n, Bn × 1, W standard BM 2 Generator: Dh (x) = (Ax)T h (x) + B T h (x)B
- 11. 2 2 σW = 0 σW = 1 Example: O-U Model Sample paths: dX(t) = AX(t) dt + B dW (t) A n × n, Bn × 1, W standard BM 2 Generator: Dh (x) = (Ax)T h (x) + B T h (x)B h quadratic, h(x) = 1 xT P x 2 h (x) = P x 2 h (x) = P Dh (x) = 1 xT (P A + AT P )x + B T P B 2
- 12. Notation: Generators & Resolvents Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t Rate matrix: Dh (x) = Q(x, y)h(y) P t = eQt y Resolvent: ∞ Rα = e−αt P t 0
- 13. Notation: Generators & Resolvents Generator: For some domain of functions h, 1 Dh (x) = lim E[h(X(s + t)) − h(X(s)) X(s) = x] t→0 t 1 t = lim (P h (x) − h(x)) t→0 t Rate matrix: Dh (x) = Q(x, y)h(y) P t = eQt y Resolvent: Resolvent equations: ∞ Rα = e−αt P t Rα = [ Iα − Q]−1 0 QRα = Rα Q = αRα − I
- 14. Notation: Generators & Resolvents Motivation: Dynamic programming. For a cost function c, hα (x) = Rα c (x) = Rα (x, y)c(y) y∈X ∞ = eαt E[c(X(t)) X(0) = x] dt 0 Discounted-cost value function
- 15. Notation: Generators & Resolvents Motivation: Dynamic programming. For a cost function c, hα (x) = Rα c (x) = Rα (x, y)c(y) y∈X ∞ = eαt E[c(X(t)) X(0) = x] dt 0 Discounted-cost value function Resolvent equation = dynamic programming equation, c + Dhα = αhα
- 16. Notation: Steady State Distribution Invariant (probability) measure π: X is stationary. In particular, X(t) ∼ π, t≥0
- 17. Notation: Steady State Distribution Invariant (probability) measure π: X is stationary. In particular, X(t) ∼ π, t≥0 Characterizations: π(x)P t (x, y) = π(y) x∈X α π(x)Rα (x, y) = π(y), α > 0 x∈X π(x)Q(x, y) = 0 x∈X y ∈X
- 18. Notation: Relative Value Function Invariant measure π, cost function c , steady-state mean η Relative value function: ∞ h(x) = E[c(X(t)) − η X(0) = x] dt 0
- 19. Notation: Relative Value Function Invariant measure π, cost function c , steady-state mean η Relative value function: ∞ h(x) = E[c(X(t)) − η X(0) = x] dt 0 Solution to Poisson’s equation (average-cost DP equation): c + Dh = η
- 20. II Representations π ∝ ν[I − (R − s ⊗ ν)]−1 h = [I − (R − s ⊗ ν)]−1 c ˜
- 21. Irreducibility ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x
- 22. Small Functions and Small Measures ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x Small functions and measures: For a function s and probability ν, R(x, y) ≥ s(x)ν(y), x, y ∈ X ∞ R= e−t P t dt 0
- 23. Small Functions and Small Measures ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x Small functions and measures: For a function s and probability ν, R(x, y) ≥ s(x)ν(y), x, y ∈ X Resolvent dominates rank-one matrix, R= ∞ e−t P t dt 0 R ≥s⊗ν
- 24. Small Functions and Small Measures ψ-Irreducibility: ψ(y) > 0 =⇒ P X(t) reaches y X(0) = x > 0 all x ψ(y) > 0 =⇒ R(x, y) > 0 all x Small functions and measures: For a function s and probability ν, R(x, y) ≥ s(x)ν(y), x, y ∈ X Resolvent dominates rank-one matrix, R= ∞ e−t P t dt 0 R ≥s⊗ν ψ-Irreducibility justi es assumption: s(x) > 0 for all x and WLOG, ν = δx ∗ , where ψ(x∗ ) > 0
- 25. α µ Example: MM1 Queue R(x, y) > 0 for all x and y (irreducible in usual sense) Conclusion: R(x, y ) ≥ s(x)ν(y ) where s(x) := R(x, 0) ν := δ0
- 26. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) R(0, . ) Gaussian Full rank if and only if (A, B) is controllable. Conclusion: Under controllability, for any m, there is ε s.t., R(x, A) ≥ s(x)ν(A) all x and A where s(x) = ε I x ≤m ν(A) uniform on x ≤m
- 27. Potential Matrix ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 G = [I − (R − s ⊗ ν)]−1
- 28. Representation of π ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 π ∝ νG νG (y) = ν(x)G(x, y) x∈X
- 29. Representation of h ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 h = RG c ˜ + constant c(x) = c(x) − η ˜ G˜ (y) = c G(x, y)˜(y) c η= π(x)c(x) y∈X y∈X
- 30. Representation of h ∞ Potential matrix: G(x, y) = (R − s ⊗ ν)n (x, y) n=0 h = RG c ˜ + constant c(x) = c(x) − η ˜ G˜ (y) = c G(x, y)˜(y) c η= π(x)c(x) y∈X y∈X If sum converges, then Poisson’s equation is solved: c(x) + Dh (x) = η
- 31. III Lyapunov Theory P n (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C π(f ) < ∞ ∆V (x) ≤ −f (x) + bIC (x)
- 32. Lyapunov Functions DV ≤ −g + bs
- 33. Lyapunov Functions DV ≤ −g + bs General assumptions: V : X → (0,∞) g : X → [1, ∞) b < ∞, s small e.g., s (x) = IC (x), C nite
- 34. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs
- 35. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs Since s⊗ν is non-negative, −[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs G−1
- 36. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs Since s⊗ν is non-negative, −[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs G−1 More positivity, V ≥ GRg − bGRs Some algebra, GR = G(R − s ⊗ ν) + (Gs) ⊗ ν ≥ G − I Gs ≤ 1
- 37. Lyapunov Bounds on G DV ≤ −g + bs Resolvent equation gives RV − V ≤ −Rg + bRs Since s⊗ν is non-negative, −[I − (R − s ⊗ ν)]V ≤ RV − V ≤ −Rg + bRs G−1 More positivity, V ≥ GRg − bGRs Some algebra, GR = G(R − s ⊗ ν) + (Gs) ⊗ ν ≥ G − I Gs ≤ 1 General bound: GRg ≤ V + 2b Gg ≤ V + g + 2b
- 38. Existence of π DV ≤ −g + bs Condition (V2) DV ≤ −1 + bs Representation: π ∝ νG Bound: GRg ≤ V + 2b =⇒ G(x, X) ≤ V (x) + 2b Conclusion: π exists as a probability measure on X
- 39. Existence of moments DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: π ∝ νG Bound: Gg ≤ V + g + 2b
- 40. Existence of moments DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: π ∝ νG Bound: Gg ≤ V + g + 2b Conclusion: π exists as a probability measure on X and the steady-state mean is nite, π(g) := π(x)g(x) ≤ b x∈X
- 41. α α µ Example: MM1 Queue ρ = µ Linear Lyapunov function, V (x) = x ∞ DV (x) = Q(x, y)y y=0 = α(x + 1) + µ(x − 1) − (α + µ)x = −(µ − α) x>0 Conclusion: (V2) holds if and only if ρ < 1
- 42. α α µ Example: MM1 Queue ρ = µ QuadraticLyapunov function, V (x) = x 2 ∞ DV (x) = Q(x, y)y 2 y=0 = α(x + 1)2 + µ(x − 1)2 − (α + µ)x2 = α(x2 + 2x + 1) + µ(x2 − 2x + 1)2 − (α + µ)x2 = −2(µ − α)x + α + µ Conclusion: (V3) holds, g(x) = 1 + x if and only if ρ < 1
- 43. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) h quadratic, h(x) = 1 xT P x 2 h (x) = P x 2 h (x) = P Dh (x) = 1 xT (P A + AT P )x + B T P B 2 Suppose that P > 0 solves the Lyapunov equation, P A + AT P = -I
- 44. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) h quadratic, h(x) = 1 xT P x 2 h (x) = P x 2 h (x) = P Dh (x) = 1 xT (P A + AT P )x + B T P B 2 Suppose that P > 0 solves the Lyapunov equation, P A + AT P = -I Then (V3) follows from the identity, 2 2 2 Dh (x) = − 1 x 2 + σX , σX = B T P B
- 45. 2 2 σW = 0 σW = 1 Example: O-U Model dX(t) = AX(t) dt + B dW (t) The function h(x) = 1 xT P x solves Poisson’s equation, 2 1 2 Dh = −g + η g(x) = 2 x 2 η = σX Suppose that P > 0 solves the Lyapunov equation, P A + AT P = -I Then (V3) follows from the identity, 2 2 2 Dh (x) = − 1 x 2 + σX , σX = B T P B
- 46. Poisson’s Equation DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: h = RG c ˜ + constant c(x) = c(x) − η ˜ Bound: RGg ≤ V + 2b =⇒ RGg (x) ≤ V (x) + 2b
- 47. Poisson’s Equation DV ≤ −g + bs Condition (V3) DV ≤ −g + bs Representation: h = RG c ˜ + constant c(x) = c(x) − η ˜ Bound: RGg ≤ V + 2b =⇒ RGg (x) ≤ V (x) + 2b Conclusion: If c is bounded by g, then h is bounded, h(x) ≤ V (x) + 2b
- 48. α α µ Example: MM1 Queue ρ = µ Poisson’s equation with g (x) = x Dh = −g + η We have (V3) with V a quadratic function of x: Recall, with h (x) = x 2 Dh (x) = −2(µ − α)x + α + µ x>0
- 49. α α µ Example: MM1 Queue ρ = µ Poisson’s equation with g (x) = x Dh = −g + η Solved with x2 + x ρ h(x) = 1 2 µ−α η= 1−ρ
- 50. IV Conclusions
- 51. P t (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C Final words π(f ) < ∞ DV (x) ≤ −f (x) + bIC (x) Just as in linear systems theory, Lyapunov functions provide a characterization of system properties, as well as a practical verification tool
- 52. P t (x, · ) − π f →0 sup Ex [SτC (f )] < ∞ C Final words π(f ) < ∞ DV (x) ≤ −f (x) + bIC (x) Just as in linear systems theory, Lyapunov functions provide a characterization of system properties, as well as a practical verification tool Much is left out of this survey - in particular, • Converse theory • Limit theory • Approximation techniques to construct Lyapunov functions or approximations to value functions • Application to controlled Markov processes, and approximate dynamic programming
- 53. References [1,4] ψ-Irreducible foundations [2,11,12,13] Mean- eld models, ODE models, and Lyapunov functions [1,4,5,9,10] Operator-theoretic methods. See also appendix of [2] [3,6,7,10] Generators and continuous time models [1] S. P. Meyn and R. L. Tweedie. Markov chains and stochastic [9] I. Kontoyiannis and S. P. Meyn. Spectral theory and limit stability. Cambridge University Press, Cambridge, second theorems for geometrically ergodic Markov processes. Ann. edition, 2009. Published in the Cambridge Mathematical Appl. Probab., 13:304–362, 2003. Presented at the INFORMS Library. Applied Probability Conference, NYC, July, 2001. [2] S. P. Meyn. Control Techniques for Complex Networks. Cam- [10] I. Kontoyiannis and S. P. Meyn. Large deviations asymptotics bridge University Press, Cambridge, 2007. Pre-publication and the spectral theory of multiplicatively regular Markov edition online: http://black.csl.uiuc.edu/˜meyn. processes. Electron. J. Probab., 10(3):61–123 (electronic), [3] S. N. Ethier and T. G. Kurtz. Markov Processes : Charac- 2005. terization and Convergence. John Wiley & Sons, New York, [11] W. Chen, D. Huang, A. Kulkarni, J. Unnikrishnan, Q. Zhu, 1986. P. Mehta, S. Meyn, and A. Wierman. Approximate dynamic [4] E. Nummelin. General Irreducible Markov Chains and Non- programming using ﬂuid and diffusion approximations with negative Operators. Cambridge University Press, Cambridge, applications to power management. Accepted for inclusion in 1984. the 48th IEEE Conference on Decision and Control, December [5] S. P. Meyn and R. L. Tweedie. Generalized resolvents 16-18 2009. and Harris recurrence of Markov processes. Contemporary [12] P. Mehta and S. Meyn. Q-learning and Pontryagin’s Minimum Mathematics, 149:227–250, 1993. Principle. Accepted for inclusion in the 48th IEEE Conference [6] S. P. Meyn and R. L. Tweedie. Stability of Markovian on Decision and Control, December 16-18 2009. processes III: Foster-Lyapunov criteria for continuous time [13] G. Fort, S. Meyn, E. Moulines, and P. Priouret. ODE processes. Adv. Appl. Probab., 25:518–548, 1993. methods for skip-free Markov chain stability with applications [7] D. Down, S. P. Meyn, and R. L. Tweedie. Exponential to MCMC. Ann. Appl. Probab., 18(2):664–707, 2008. and uniform ergodicity of Markov processes. Ann. Probab., 23(4):1671–1691, 1995. [8] P. W. Glynn and S. P. Meyn. A Liapounov bound for solutions of the Poisson equation. Ann. Probab., 24(2):916–931, 1996. See also earlier seminal work by Hordijk, Tweedie, ... full references in [1].

No public clipboards found for this slide

×
### Save the most important slides with Clipping

Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.

Be the first to comment