Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AbadIA ING Direct - Madrid 2019

25 views

Published on

Using Reinforcement Learning to play and solve the abbey of crime

Published in: Technology
  • Be the first to comment

  • Be the first to like this

AbadIA ING Direct - Madrid 2019

  1. 1. AbadIA Using Reinforcement Learning to play and solve the abbey of crime Juantomás García - Chief Envisioning Officer - Sngular
  2. 2. 
 Good Morning ING
  3. 3. Who I am Juantomás García ( 0-) •Chief Envisioning Officer @ Sngular •GDEx2 (Google Developer Expert) for cloud and Machine Learning •#AbadIA Cheer Leader Others •Co-Author of the first Spanish free software book “La Pastilla Roja” •Former President of Hispalinux (Spanish Linux User Group) •Organizer of the Machine Learning Spain and GDG Cloud Madrid.
  4. 4. What we will see
  5. 5. How #AbadIA started
  6. 6. ? My buddy Sebas told me: Is it possible to create an AI that can learn to play and solve the game? It’s a really crazy idea with an insane cult 8-bit game, but I like it “ ”
  7. 7. The Game:The Abbey of the Crime
  8. 8. First 8-bit RPG in pseudo 3D (2.5D) It was at 1987 and this game is a kind of legend in the video games world. Based in Umberto Eco book “In the name of the rose” Do you know the game? THE GAME
  9. 9. Size of the program: 87 Kb That include: • Code • Graphics • Music • Maps Do you know the game? THE GAME
  10. 10. The size of this avatar is: 87Kb Do you know the game? THE GAME
  11. 11. It was programmed using Z80 assembler for Amstrad CPC and Sinclair Spectrum ABOUT THE GAME Was programmed by Paco Menéndez Graphics was designed by Juan Delcán
  12. 12. The Plan
  13. 13. The plan: Make an AI that can learn to play and solve the game
  14. 14. What we had
  15. 15. The original executable
  16. 16. Manuel Abadía disassembled the original Amstrad CPC game rom. He documented all the code, marked the graphics, sounds, etc. GitHub page: https://bit.ly/2I9TMZk Original Code
  17. 17. Current Version
  18. 18. Manuel Abadía make a port to C++ of the game using his special video games framework: Vigasoco Sebastian Blanes fork the project to use SDL. VigasocoSDL was born. Now the abbey can be played in chrome, PS3, CC, linux, MacOS, Windows, etc. VigasocoSDL GitHub page: https://github.com/luzbel/VigasocoSDL/wiki GAME EVOLUTION
  19. 19. How difficult is the Challenge
  20. 20. Number of atoms in our universe? HOW DIFFICULT IS THE CHALLENGE?
  21. 21. 10^80 Number of atoms in our universe (Hawking said there are more than one universe!!) HOW DIFFICULT IS THE CHALLENGE?
  22. 22. 10^80 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves? HOW DIFFICULT IS THE CHALLENGE?
  23. 23. 10^12010^80 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves HOW DIFFICULT IS THE CHALLENGE?
  24. 24. 10^12010^80 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves? HOW DIFFICULT IS THE CHALLENGE?
  25. 25. 10^12010^80 10^761 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves HOW DIFFICULT IS THE CHALLENGE?
  26. 26. 10^12010^80 10^761 Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves AbadIA moves needs to solve the game? HOW DIFFICULT IS THE CHALLENGE?
  27. 27. 10^12010^80 10^761 10^10k Number of atoms in our universe (Hawking said there are more than one universe!!) Number of chess legal moves Number of GO legal moves AbadIA legal moves where N is the depth of the game you wish to check. For N= 10000 is 10^10000 HOW DIFFICULT IS THE CHALLENGE?
  28. 28. Universe 100000000000000000000000000000000000000000000000000000000000000000000000000000000 Chess 1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 GO 100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000 The Abbey of Crime 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 Just for the guys that not attended enought math classes.
  29. 29. The AI & Human Players only know if the game is solved after 10.000 actions.
  30. 30. HOW DIFFICULT IS THE CHALLENGE?
  31. 31. HOW DIFFICULT IS THE CHALLENGE?
  32. 32. HOW DIFFICULT IS THE CHALLENGE?
  33. 33. HOW DIFFICULT IS THE CHALLENGE?
  34. 34. First Steps
  35. 35. Interacting with the game
  36. 36. How to interact with the game. You will need lots of tools before starting to create the AI 1 2 3 How you get and save the game information. How to play millions of games. AI TOOLS
  37. 37. You will need send actions, get game info, reset it, etc. How to interact with the game?? HOW TO INTERACT
  38. 38. First version with too BeersHOW TO INTERACT
  39. 39. You may use a fake keyboard driver, a unix pipe, a socket, but do you know what is the really cool way to do this: HOW TO INTERACT
  40. 40. An embedded web server inside the game !!!! HOW TO INTERACT
  41. 41. Now we have a REST based API. Two way communication. We send actions - Moves - Resets - Save/Load states We got information - State dumps - Actions - Checkpoints HOW TO INTERACT WITH THE GAME
  42. 42. curl -X POST http:!//localhost:4477/abadIA/game/current/actions/RIGHT or curl -X GET http:!//localhost:4477/abadIA/game/current —H 'accept: application/json’ or Use python request library AS EASY AS
  43. 43. Demo game engine + web server + curl
  44. 44. Gathering Information
  45. 45. At the beginning a laptop was enough. But very soon you need more CPU/ GPU. Then product like Google Cloud is your best ally. We had created a few Dockers, so now we can execute lots of instances of the game in parallel. note: If we use Google Cloud services like GKE, we can launch hundred of games in parallel. HOW TO SCALE IT
  46. 46. We recollect a lot of information - Game Info (timestamps, rewards, bonus, obsequium) - Games moves (state, action, reward, new_state) - Checkpoints (to restore the game at an interesting time) - ML Models (for recovering good models o just make a benchmark) GATHERING INFORMATION
  47. 47. It takes a lot of time to get all the parts working all together. Building tools, testing every piece, every option. Sometimes I feel like I was Mario Bros. GATHERING INFORMATION
  48. 48. Playground for the game
  49. 49. Github Repo: https://www.github.com/LaAbadIAdelCrimen/ abadia-gym One the most frequently used tool is OpenAI Gym So we design an AbadIA gym The gym is a standard place to train and interact with Reinforcement Learning agents. In our project the gym is framework to wrap the game engine. PLAYGROUND FOR PLAYING
  50. 50. A game server with REST API An openAI Gym Enought hardware resources So what’s the next step SO WE HAVE
  51. 51. An small intro to Reinforcement Learning
  52. 52. A RL agent is a program that interacts with an environment, in our case a OpenAI gym for AbadIA, and learn from observations and rewards. CREATE A RL AGENT
  53. 53. And that looks like:
  54. 54. Demo Time + Video
  55. 55. AbadIA RL Strategies
  56. 56. Random Q*tables Q*Learning DQN CNN A3C MiniGames REINFORCEMENT LEARNING STRATEGIES
  57. 57. Simple Neural Network Model
  58. 58. AbadIA Architecture Parts
  59. 59. Game Engine & Agent Containers
  60. 60. First we create the first 2 containers for: ● AbadIA Game Engine ● AbadIA Agent
  61. 61. K8S small cluster
  62. 62. Cluster Features: ● Preemtible ● n1-standard-1 (7€ month)
  63. 63. Gathering Information
  64. 64. Game info pipeline
  65. 65. AutoGaming
  66. 66. Game Generator Games Checkpoints
  67. 67. Storage of the info: ● Game Information ● Actions - States - Rewards ● Snapshots ● Models
  68. 68. How to Integrate the all the information: ● Serverless ● Easy ● Fast
  69. 69. Advanced and Massive Analytics: ● Game Information ● Actions - States - Rewards ● Snapshots ● Even some ML (LRs)
  70. 70. Notebooks
  71. 71. How to manipulate and visualize with Datalabs or Colab: ● Notebooks ● Managed ● Integrated with Google Drive / Bigquery
  72. 72. AI Frameworks
  73. 73. How to training and serve models at scale: ● Tensorflow / Keras ● ML Engine ● AutoML
  74. 74. Orchestrating
  75. 75. How to execute hundreds of instances: ● Kubernetes ● Google Containers
  76. 76. Show me the code (how to collaborate)
  77. 77. GitHub LaAbadIAdelCrimen https://github.com/LaAbadIAdelCrimen HOW TO COLLABORATE
  78. 78. Join US, we’ll share the glory with you
  79. 79. Slack AbadIA JOIN US!!!!
  80. 80. Questions? This talk have a free questions lifetime warranty: If you have any questions or concerns about this talk, feel free to contact me anytime. Selfie Time: If you like the talk just smile while I take the selfie ;-) We’re Hiring, Sngular People twitter: @juantomas juantomas.garcia@sngular.com
  81. 81. Thank You!!

×