Programs that Play better than Us
Melvin Zhang
melvin@melvinzhang.net
@melvinzhangzy

https://en.wikipedia.org/wiki/File:ST Battle Chess.png

https://en.wikipedia.org/wiki/Deep Blue (chess computer)
Deep Blue (IBM, 1996)

http://afflictor.com/2012/09/11/chess-programs-regularly-play-at-good-amateur-level/

Game tree

Optimal play
Terminal
min player
max player

Optimal play
1 01 1 1Terminal
min player
max player

Optimal play
1 01 1 1
0
Terminal
min player
max player

Optimal play
1 01 1 1
0 1
Terminal
min player
max player

Optimal play
1 01 1 1
0 1
1
Terminal
min player
max player

Chess has about 1046
states!

Minimax algorithm
Cut-off
min player
max player

Minimax algorithm
.7 .1 .6 .9Cut-off
min player
max player

Minimax algorithm
.7 .1 .6 .9
.1
Cut-off
min player
max player

Minimax algorithm
.7 .1 .6 .9
.1 .6
Cut-off
min player
max player

Minimax algorithm
.7 .1 .6 .9
.1 .6
.6
Cut-off
min player
max player

https://stockfishchess.org/
Stockfish

https://tests.stockfishchess.org/
Testing AI changes is crucial

Value functions are hard!

http://mathworld.wolfram.com/Go.html

http://www.remi-coulom.fr/CrazyStone/
R´emi Coulom

http://www.wired.com/2014/05/the-world-of-computer-go/

Monte Carlo evaluations
Cut-off
min player
max player

Monte Carlo evaluations
Cut-off
min player
max player

Monte Carlo evaluations
Cut-off
min player
max player

Monte Carlo evaluations
Cut-off
min player
max player

Monte Carlo evaluations
Cut-off
min player
max player
.7

Monte Carlo Tree Search (MCTS)

by Google Deepmind
https://deepmind.com/research/alphago/

https://gogameguru.com/alphago-races-ahead-2-0-lee-sedol/

MCTS + Policy and value networks

http://www.nature.com/nature/journal/v529/n7587/full/nature16961.html
MCTS + Policy and value networks

Some games have hidden information!

http://magic.wizards.com/en/events/coverage/gpsin15/father-son-2015-06-27

https://magarena.github.io

Determinization: choose a random instance of the
hidden information during simulation

Comparison of Minimax and MCTS
At 1s thinking time:
Minimax MCTS
1 0.88

Comparison of Minimax and MCTS
At 1s thinking time:
Minimax MCTS
1 0.88
At 4s thinking time:
Minimax MCTS
1 1.71

Open problems

MCTS is bad at tight tactical play.

MCTS plays badly when it is behind in the game.

Further readings

Further readings