From Monte-Carlo
to win rate first search
for “Dobutsu Shogi”

                        2010/05/22
                    IHAR...
Abstract
• On algorithm for computer Shogi (Japanese
  chess)
• Contents
   – Exhibition of Dobutsu Shogi
   – Min-max met...
Dobutsu shogi
• This slide mentions computer game
  algorithm by using Dobutsu Shogi
• Dobutsu Shogi: a miniature shogi
• ...
Rule of Dobutsu Shogi 1
Five kind of pieces
Initial position is as figure
Win if you catch lion
Win if your lion reaches
t...
Rule of Dobutsu Shogi 2
All pieces move by one step
                              vertical horizontal and
          forwar...
Copy right of Dobutsu shogi
• I do not know who has copy right
   – FUJITA Maiko (illustration)
   – KITAO Madoka (making ...
Illustration on this slide
• Because of that complex copy right, I use
  the illustrations on the website below in this
  ...
Exhibition initial position
                Black: win rate first
                search (presented)
                White...
Exhibition 1st move
              Black advanced
              giraffe
Exhibition 2nd move
              White advanced
              giraffe
Exhibition 3rd move
              Black took chick by
              chick
Exhibition 4th move
              White took chick by
              elephant
Exhibition 5th move
              Black advanced
              elephant
Exhibition 6th move
              White dropped chick
              for defense
Exhibition 7th move
              Black moved giraffe
              backward
Exhibition 8th move
              White advanced
              giraffe
Exhibition 9th move
              Black dropped chick
              for defense
Exhibition 10th move
              White took elephant
              by giraffe
Exhibition 11th move
              Black took giraffe by
              lion
Exhibition 12th move
              White dropped
              elephant
              This elephant
              combinat...
Exhibition 13th move
              Black lion escaped
Exhibition 14th move
              White advanced lion
Exhibition 15th move
              Black dropped giraffe
              and check
Exhibition 16th move
              White escaped lion
Exhibition 17th move
              Black advanced
              giraffe
              Black forced white to
              ...
Exhibition 18th move
              White took giraffe by
              elephant
Exhibition 19th move
              Black took elephant by
              lion
Exhibition 20th move
              White dropped giraffe
Exhibition 21st move
              Black dropped
              elephant behind lion
Exhibition 22nd move
             White moved elephant
             backward
Exhibition 23rd move
              Black advanced
              elephant
Exhibition 24th move
              White check by giraffe
Exhibition 25th move
              Black took giraffe by
              elephant
Exhibition 26th move
              White took elephant
              by chick
              If white had taken by
        ...
Exhibition 27th move
              Black lion escaped
Exhibition 28th move
              White dropped
              elephant
Exhibition 29th move
              Black check by giraffe
Exhibition 30th move
              White took giraffe by
              elephant
Exhibition 31st move
              Black took chick by
              lion, and white
              resigned
              ...
Min-max method
• A conventional method
• Today the most successful method for shogi
• Explanation using tree structure fro...
Min-max        Example: 3 depth

                    Present board position




                                   after 1...
Suppose scores after 3 moves
          were revealed
                                  -8
                                ...
Scores after 2 moves are
          maximum of each score
                                     -8
                         ...
Scores after 1 moves are
          minimum of each score
                                     -8
                         ...
Select the move having
          maximum score
                                   -8
                         23


       ...
Min-max method
• Theoretically you can select the move that
  has the maximum score after N moves
• Theoretically if we co...
Min-max method
• Although many methods for reducing
  computational cost is presented, they will
  be not mentioned this s...
Conclusion of min-max method
• It uses tree structure
• Scores after N moves are needed
• Pruning is needed
Monte-Carlo method
• While I do not know the history of Monte-
  Carlo method, it have been successful for
  computer “go”...
Outline of Monte-Carlo
 first move                • Repeat random
                             moves
                     ...
Outline of Monte-Carlo
            • Repeat playout
            • Obtain win rate of
              the first move
        ...
Outline of Monte-Carlo
• Outline is only it
• As to “Go”, this method has become
  stronger by combining tree structure an...
Example of knowledge of “Go”
     • Observe 3x3 squares
     • Set low probability to drop
       black stone the center o...
Monte-Carlo for shogi
• Simple Monte-Carlo method does not work
  for shogi (too many bad moves appear)
• A causal must be...
Why Monte-Carlo for shogi
• Ability to determine the move by result of
  the end of game, which seems beautiful
• No evalu...
Discussion Monte using tree




                                        green and red
                                    ...
Discussion Monte using tree




                                           after 3 moves
                                 ...
Discussion Monte using tree




                                                    ones of min-max method
               ...
Discussion Monte using tree
                  • Q: How do you calculate
                    parent node 0.6 by children
  ...
Discussion Monte using tree
                  • Q: How do you ignore 0.2?
                  • A1: Always search maximum
  ...
Discussion Monte using tree




                                           maximum win rate
                              ...
Win rate first search
• Remember win rate of searched node
• Almost always search node that has
  maximum win rate
• Somet...
Additional explanation
• Update win rate at every playout
• Keep numerator and denominator as win
  rate
• Add constant nu...
Problems of presented method
• Win rates of the nodes that have not been
  searched are mentioned from the next pages
• Ma...
Unreached node
                              • On the node that has
                                not been searched
    ...
Another win rate
• Before this page, knowledge of shogi does
  not appear and only graph is used
• This win rate uses know...
Another win rate
• Calculate win rate by these factors
   – Piece position before and after move
   – Kind of pieces movin...
Another smaller win rate
• Another smaller win rate table is prepared
   – Kind of pieces moving and taken
   – Is positio...
Conclusion of presented method
• Win rates of all searched nodes are
  remembered and learned by playout
• Select node tha...
Condition of simulation game
• Win rate first search vs. Simple min-max
  method (evaluation function is composed
  by onl...
Result of simulation 1
   Number
   of playout 10000     30000     100000
   Presented
   method:    22-76     44-52     4...
Result of simulation 2
Depth of
min-max    4      5     6     7     8     9
Present
method:    94-6   77-20 48-49 37-61 24...
Impression by human viewer
• Frequently presented method take bad
  moves
• Although it is a variation of Monte-Carlo
  me...
Conclusion and future issue
• Conclusion
   – Playout by win rate first
   – Select moves without preset knowledge
   – Se...
Upcoming SlideShare
Loading in …5
×

win rate first search

932
-1

Published on

discussion of Monte-Carlo search method for computer game, especially shogi (Japanese chess). presenting "win rate first search."

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
932
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

win rate first search

  1. 1. From Monte-Carlo to win rate first search for “Dobutsu Shogi” 2010/05/22 IHARA Takehiro
  2. 2. Abstract • On algorithm for computer Shogi (Japanese chess) • Contents – Exhibition of Dobutsu Shogi – Min-max method (conventional) – Monte-Carlo method (conventional) – Win rate first search (presented)
  3. 3. Dobutsu shogi • This slide mentions computer game algorithm by using Dobutsu Shogi • Dobutsu Shogi: a miniature shogi • Shogi: Japanese chess • Dobutsu: animal • Normal shogi is too large to examine new methods
  4. 4. Rule of Dobutsu Shogi 1 Five kind of pieces Initial position is as figure Win if you catch lion Win if your lion reaches to opposite end Chick promotes chicken
  5. 5. Rule of Dobutsu Shogi 2 All pieces move by one step vertical horizontal and forward forward-diagonal around 8 diago vertical squares nal horizontal You can reuse (drop) the pieces that you took
  6. 6. Copy right of Dobutsu shogi • I do not know who has copy right – FUJITA Maiko (illustration) – KITAO Madoka (making rule) – LPSA (the two designers had belonged to) – GENTOSHA Education (toy seller)
  7. 7. Illustration on this slide • Because of that complex copy right, I use the illustrations on the website below in this slide, instead of FUJITA's ones • “SOZAIYA JUN” • (http://park18.wakwak.com/~osyare/)
  8. 8. Exhibition initial position Black: win rate first search (presented) White: min-max method, search depth 9, evaluation function is composed by only piece value (conventional)
  9. 9. Exhibition 1st move Black advanced giraffe
  10. 10. Exhibition 2nd move White advanced giraffe
  11. 11. Exhibition 3rd move Black took chick by chick
  12. 12. Exhibition 4th move White took chick by elephant
  13. 13. Exhibition 5th move Black advanced elephant
  14. 14. Exhibition 6th move White dropped chick for defense
  15. 15. Exhibition 7th move Black moved giraffe backward
  16. 16. Exhibition 8th move White advanced giraffe
  17. 17. Exhibition 9th move Black dropped chick for defense
  18. 18. Exhibition 10th move White took elephant by giraffe
  19. 19. Exhibition 11th move Black took giraffe by lion
  20. 20. Exhibition 12th move White dropped elephant This elephant combination style is strong
  21. 21. Exhibition 13th move Black lion escaped
  22. 22. Exhibition 14th move White advanced lion
  23. 23. Exhibition 15th move Black dropped giraffe and check
  24. 24. Exhibition 16th move White escaped lion
  25. 25. Exhibition 17th move Black advanced giraffe Black forced white to select taking giraffe or escaping elephant
  26. 26. Exhibition 18th move White took giraffe by elephant
  27. 27. Exhibition 19th move Black took elephant by lion
  28. 28. Exhibition 20th move White dropped giraffe
  29. 29. Exhibition 21st move Black dropped elephant behind lion
  30. 30. Exhibition 22nd move White moved elephant backward
  31. 31. Exhibition 23rd move Black advanced elephant
  32. 32. Exhibition 24th move White check by giraffe
  33. 33. Exhibition 25th move Black took giraffe by elephant
  34. 34. Exhibition 26th move White took elephant by chick If white had taken by elephant, white would be mate
  35. 35. Exhibition 27th move Black lion escaped
  36. 36. Exhibition 28th move White dropped elephant
  37. 37. Exhibition 29th move Black check by giraffe
  38. 38. Exhibition 30th move White took giraffe by elephant
  39. 39. Exhibition 31st move Black took chick by lion, and white resigned After it, white drops giraffe on side of lion, black giraffe takes elephant and check, white lion takes it, black chick advances, white lion moves backward, black drops chick, check mate
  40. 40. Min-max method • A conventional method • Today the most successful method for shogi • Explanation using tree structure from next page
  41. 41. Min-max Example: 3 depth Present board position after 1 and 2 moves Board position Board position after 3 moves
  42. 42. Suppose scores after 3 moves were revealed -8 23 5 -9 Min-max 3 10 -3 -4
  43. 43. Scores after 2 moves are maximum of each score -8 23 23 5 5 -9 Min-max 3 10 10 -3 -3 -4
  44. 44. Scores after 1 moves are minimum of each score -8 23 23 5 5 5 -9 Min-max 3 10 10 -3 -3 -3 -4
  45. 45. Select the move having maximum score -8 23 23 5 5 5 -9 5 Min-max 3 10 10 -3 -3 -3 -4
  46. 46. Min-max method • Theoretically you can select the move that has the maximum score after N moves • Theoretically if we could obtain the score of the end of the game, we would always win the game • Practically because of too large computational cost, we cannot calculate all moves
  47. 47. Min-max method • Although many methods for reducing computational cost is presented, they will be not mentioned this slide (It is called pruning to reduce the number of searched nodes)
  48. 48. Conclusion of min-max method • It uses tree structure • Scores after N moves are needed • Pruning is needed
  49. 49. Monte-Carlo method • While I do not know the history of Monte- Carlo method, it have been successful for computer “go” (precisely successful by Monte-Carlo tree search) • They say that it is difficult to apply computer shogi (or chess-like game) yet
  50. 50. Outline of Monte-Carlo first move • Repeat random moves • Then game finishes random move and winner is playout revealed • making game end by random moves is called playout end of game
  51. 51. Outline of Monte-Carlo • Repeat playout • Obtain win rate of the first move • (number of win) / (number of playout) • Select move having highest win rate at the last
  52. 52. Outline of Monte-Carlo • Outline is only it • As to “Go”, this method has become stronger by combining tree structure and making Monte-Carlo tree search (this slide does not mention it) • Another improvement is that playout uses moves by knowledge of “Go” instead of simple random moves
  53. 53. Example of knowledge of “Go” • Observe 3x3 squares • Set low probability to drop black stone the center of above figure • Set high probability to drop black stone the center of below figure
  54. 54. Monte-Carlo for shogi • Simple Monte-Carlo method does not work for shogi (too many bad moves appear) • A causal must be that few moves in all legal moves are good on shogi • I do not want to use knowledge of shogi by neither machine learning nor manual setting
  55. 55. Why Monte-Carlo for shogi • Ability to determine the move by result of the end of game, which seems beautiful • No evaluation function is needed, no preset knowledge is needed
  56. 56. Discussion Monte using tree green and red equal win rate between Simple random moves lead Truth is that green win and red lose It tells importance of tree structure
  57. 57. Discussion Monte using tree after 3 moves Suppose you obtain win rate 0.1 0.3 0.7 0.8 0.2 0.6 0.9 0.4 Obtain win rate of green and red from These 3-move-after rates by playout
  58. 58. Discussion Monte using tree ones of min-max method Ideally the rates are equal to 0.3 0.6 0.3 0.8 0.6 0.9 0.1 0.3 0.7 0.8 0.2 0.6 0.9 0.4
  59. 59. Discussion Monte using tree • Q: How do you calculate parent node 0.6 by children nodes 0.2 and 0.6 0.6 • A: Ignore 0.2 0.2 0.6
  60. 60. Discussion Monte using tree • Q: How do you ignore 0.2? • A1: Always search maximum 0.6 win rate node • A2: sometimes search through node randomly 0.2 0.6
  61. 61. Discussion Monte using tree maximum win rate Search node that has 0.1 0.3 0.7 0.8 0.2 0.6 0.9 0.4 This tactics finds the best path
  62. 62. Win rate first search • Remember win rate of searched node • Almost always search node that has maximum win rate • Sometimes search randomly (ideally it is not needed) • Then this algorithm finds the best move
  63. 63. Additional explanation • Update win rate at every playout • Keep numerator and denominator as win rate • Add constant number to both numerator and denominator when win the playout • Add constant number to only denominator when lose the playout
  64. 64. Problems of presented method • Win rates of the nodes that have not been searched are mentioned from the next pages • Many other issues must be hiding, though I have not defined them
  65. 65. Unreached node • On the node that has not been searched and no win rate 0.4 0.6 0.3 unreached
  66. 66. Another win rate • Before this page, knowledge of shogi does not appear and only graph is used • This win rate uses knowledge of shogi • Win rate is calculated by kind of moves • For example, taking piece, promotion, and etc.
  67. 67. Another win rate • Calculate win rate by these factors – Piece position before and after move – Kind of pieces moving and taken – Is position whether controlled or not • Win rate table for all combination of these factors is prepared • These win rates are learned by playout, whose values are not prepared
  68. 68. Another smaller win rate • Another smaller win rate table is prepared – Kind of pieces moving and taken – Is position whether controlled or not • Since it is small, it learns fast • It is used when “another larger win rate” is not learned yet • If all three kinds of win rate have not been learned, let win rate be 1
  69. 69. Conclusion of presented method • Win rates of all searched nodes are remembered and learned by playout • Select node that has highest win rate in playout (“win rate first search”) • Sometimes select node randomly • If win rate has not been learned, other win rates are used
  70. 70. Condition of simulation game • Win rate first search vs. Simple min-max method (evaluation function is composed by only values of pieces) • If the game continues till 80 moves, the game is regarded as even (special rule for this simulation)
  71. 71. Result of simulation 1 Number of playout 10000 30000 100000 Presented method: 22-76 44-52 48-49 black Presented method: 16-81 30-68 61-35 white Win-lose for presented method in 100 games Some even games exist Depth of min-max method is 6 More the playouts are, stronger the method is
  72. 72. Result of simulation 2 Depth of min-max 4 5 6 7 8 9 Present method: 94-6 77-20 48-49 37-61 24-73 14-85 black Present method: 78-21 78-20 61-35 38-57 40-52 20-74 white Win-lose for presented method in 100 games Some even games exist 100000 playouts for presented method Almost same strongness to 6-depth min-max
  73. 73. Impression by human viewer • Frequently presented method take bad moves • Although it is a variation of Monte-Carlo method, it can find mate route • It is good at finding narrow route • Difference of the number of playout shows clearly difference of strongness
  74. 74. Conclusion and future issue • Conclusion – Playout by win rate first – Select moves without preset knowledge – Select moves by result of playout • Future – Someone can apply it to “Go” or other chess-like games – I return to research speech signal processing
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×