Kristoffer Arnsfelt Hansen, Rasmus Ibsen-Jensen and Peter Bro Miltersen. The complexity of solving reachability games using value and strategy iteration

1. The complexity of solving reachability games using value and strategy iteration Kristoffer Arnsfelt Hansen Rasmus Ibsen-Jensen Peter Bro Miltersen Aarhus University Denmark CSR 2011, 14’th June

3. Matrix games von Neumann 1928 2/42 0 1 -1 -1 0 1 1 -1 0

4. Matrix games von Neumann 1928 2/42 0 1 -1 -1 0 1 1 -1 0

5. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 Each entry can be either 0, 1 or a pointer vs. Dante* Lucifer* 0 1 * Naming convention from Hansen, Koucky and Miltersen, 2009 3/42 0 1 -1 -1 0 1 1 -1 0

6. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 vs. Dante* Lucifer* Each entry can be either 0, 1 or a pointer * Naming convention from Hansen, Koucky and Miltersen, 2009 3/42

7. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 Each entry can be either 0, 1 or a pointer 3/42

8. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 Each entry can be either 0 , 1 or a pointer 3/42 0 0 0 0 0 0 0 0 0 0 0 0

9. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 Each entry can be either 0, 1 or a pointer 3/42 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0

10. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 Each entry can be either 0, 1 or a pointer S: 3/42 1 0 0 S 1 0 S S 1 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S

11. Concurrent reachability games Everett 1957/de Alfaro, Henzinger, Kupferman 1998 Each entry can be either 0, 1 or a pointer S: 3/42 1 0 0 S 1 0 S S 1 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S

12. Histories Each entry can be either 0, 1 or a pointer S: 4/42 1 0 0 S 1 0 S S 1 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S

15. Everett 1957 Value of i 7/42

20. Value iteration example – G 0 S: 12/42 1 0 0 S 1 0 S S 1 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S

21. Value iteration example – G 0 S: 0 0 0 0 12/42 1 0 0 S 1 0 S S 1 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S

22. Value iteration example – G 1 S: 0 0 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

23. Value iteration example – G 1 S: 0 0 0 0 0 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

24. Value iteration example – G 1 S: 0 0 0 0 1 1 1 1 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1 0 0 0

25. Value iteration example – G 1 S: 0 0 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1 1 0 0 1 0 1

26. Value iteration example – G 1 S: 0 0 0 0 0 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1 1 0 0 1 0 1

27. Value iteration example – G 1 0 S: 0.33333/ 0 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1 1 0 0 0 1 0 0 0 1

28. Value iteration example – G 1 S: 0 0 0.33333/ 0 0 0 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

29. Value iteration example – G 1 S: 0 0 0 0 0 0 0 0 0 0.33333/ 0 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1 0 0 0

30. Value iteration example – G 1 S: 0 0.33333/ 0 0 0/ 0 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1 0 0 0 0 0 0 0 0 0

31. Value iteration example – G 1 S: 0 0 0 0.33333/ 0 0/ 0/ 0/ 13/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

32. Value iteration example – G 2 S: 0 0 0 0.33333/ 0.33333 0.11111/ 0/ 0/ 14/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

33. Value iteration example – G 3 S: 0.11111 0 0 0.33333/ 0.33333 0.11111/ 0/ 0.03704/ 15/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

34. Value iteration example – G 4 S: 0.11111 0.03704 0 0.33333/ 0.33333 0.11111/ 0.01235/ 0.03704/ 16/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

35. Value iteration example – G 5 S: 0.11111 0.03704 0.01235 0.33748/ 0.33333 0.11533/ 0.01754/ 0.04147/ 17/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

36. Value iteration example – G 6 S: 0.11533 0.04147 0.01754 0.33925/ 0.33748 0.11855/ 0.02172/ 0.04493/ 18/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

37. Value iteration example – G 7 S: 0.11855 0.04493 0.02172 0.34068/ 0.33925 0.12064/ 0.02519/ 0.04772/ 19/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

38. Value iteration example – G 8 S: 0.12064 0.04772 0.02519 0.34187/ 0.34068 0.12388/ 0.02815/ 0.04991/ 20/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

39. Value iteration example – G 9 S: 0.12388 0.04991 0.02815 0.34378/ 0.34187 0.12517/ 0.03070/ 0.05129 / 21/42 0 0 S 0 S S 0 0 S 0 S S 0 0 S 0 S S 1 0 0 S 1 0 S S 1

40. Strategy iteration Chatterjee, de Alfaro, Henzinger ’06 22/42 Was conjectured to be fast

103. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices 37/42 1 0 1 0 0 1 0 1 1 0 1

104. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix t:=0 Strategy iteration on 3 matrices 37/42 1 0 1 0 0 1 0 1 1 0 1

105. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=0 0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 37/42 1 0 1 0 0 1 0 1 1 0 1

106. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=1 0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 38/42 1 0 1 0 0 1 0 1 1 0 1

107. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=1 0 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 38/42 1 0 1 0 0 1 0 1 1 0 1

108. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=1 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.25 0.125 38/42 1 0 1 0 0 1 0 1 1 0 1

109. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=1 0.5 0.66667 0.33333 0.66667 0.33333 0.57143 0.42857 0.53333 0.46667 0.5 0.5 0.25 0.125 38/42 1 0 1 1 0 1 0 0 1 0 1

110. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=2 0.5 0.66667 0.33333 0.66667 0.33333 0.57143 0.42857 0.53333 0.46667 0.5 0.5 0.25 0.125 39/42 1 0 1 1 0 1 0 0 1 0 1

111. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=2 0.5 0.66667 0.33333 0.66667 0.33333 0.57143 0.42857 0.53333 0.46667 0.5 0.5 0.25 0.125 39/42 1 0 1 1 0 1 0 0 1 0 1

112. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=2 0.66667 0.66667 0.33333 0.66667 0.33333 0.57143 0.42857 0.53333 0.46667 0.66667 0.53333 0.30476 0.20317 39/42 1 0 1 1 0 1 0 0 1 0 1

113. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=2 0.66667 0.75000 0.25000 0.75000 0.25000 0.61765 0.38235 0.55654 0.44346 0.66667 0.53333 0.30476 0.20317 39/42 1 0 1 1 0 1 0 0 1 0 1

114. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=3 0.66667 0.75000 0.25000 0.75000 0.25000 0.61765 0.38235 0.55654 0.44346 0.66667 0.53333 0.30476 0.20317 40/42 1 0 1 1 0 1 0 0 1 0 1

115. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=3 0.66667 0.75000 0.25000 0.75000 0.25000 0.61765 0.38235 0.55654 0.44346 0.66667 0.53333 0.30476 0.20317 40/42 1 0 1 1 0 1 0 0 1 0 1

116. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=3 0.75000 0.75000 0.25000 0.75000 0.25000 0.61765 0.38235 0.55654 0.44346 0.75000 0.55654 0.34374 0.25781 40/42 1 0 1 1 0 1 0 0 1 0 1

117. Exemplifying important facts Value iteration on 1 matrix Strategy iteration on 1 matrix Strategy iteration on 3 matrices t:=3 0.75000 0.80000 0.20000 0.80000 0.20000 0.65072 0.34928 0.57399 0.42601 0.75000 0.55654 0.34374 0.25781 41/42 1 0 1 1 0 1 0 0 1 0 1