Juantomás García gave a talk on machine learning pipelines for developing AI that can learn to play and solve the 1980s video game "The Abbey of the Crime". He discussed gathering game data, exploring different reinforcement learning strategies, and developing a simple neural network model with policy and value networks to determine moves and rewards. He described his current pipeline that moves raw game data through processing steps using technologies like Kubernetes, PubSub, training jobs, and model storage. The talk encouraged attendees to collaborate on the open source project on GitHub and join the AbadIA Slack channel.

1. AbadIA
Machine Learning Pipelines:
Lo que nadie te cuenta sobre
como hacer AI
Juantomás García - Chief Envisioning Officer - Sngular

2. Buenas y lluviosas
tardes
Madrid!!

3. Who I am
Juantomás García ( 0-)
•Chief Envisioning Officer @ Sngular
•GDEx2 (Google Developer Expert) for cloud and Machine Learning
•#AbadIA Cheer Leader
Others
•Co-Author of the first Spanish free software book “La Pastilla Roja”
•Former President of Hispalinux (Spanish Linux User Group)
•Organizer of the Machine Learning Spain and GDG Cloud Madrid.

4. What we will see

5. Even if you don’t noticed it,
you will do lots of ML
pipelines

6. The Game:The
Abbey of the Crime

7. First 8-bit RPG in pseudo 3D (2.5D)
It was at 1987 and this game is a kind
of legend in the video games world.
Based in Umberto Eco book “In the
name of the rose”
Do you know the game?
THE GAME

8. The Plan

9. The plan:
Make an AI
that can learn
to play and
solve the game

10. What we had

11. We recollect a lot of information
- Game Info (timestamps, rewards, bonus, obsequium)
- Games moves (state, action, reward, new_state)
- Checkpoints (to restore the game at an interesting time)
- ML Models (for recovering good models o just make a benchmark)
GATHERING INFORMATION

12. A game
server with
REST API
An openAI
Gym
Enought
hardware
resources
So what’s the next step
SO WE HAVE
Tons of
games
data

13. An small intro to
Reinforcement
Learning

14. A RL agent is a program that interacts with an environment, in our
case a OpenAI gym for AbadIA, and learn from observations and
rewards.
CREATE A RL AGENT

15. And that looks like:

16. AbadIA RL
Strategies

17. Random Q*tables Q*Learning
DQN CNN A3C MiniGames
REINFORCEMENT LEARNING STRATEGIES

18. Current Version

19. Simple Neural Network Model

20. 2 DNN to learn and
play

21. A Policy network

22. Its function is to determinate the which is
the best next move.
This policy is one of the more important
parts of the project.

23. A Value Network
Give us the max reward for
position/day/hour.

24. How we (will) explore
the actions space

25. Huge combinatorial problem if
you want to explore 10 deep
levels:
10^10 actions to test

26. Monte Carlo Tree Search

27. AbadIA
Architecture Parts

28. Game Engine &
Agent Containers

29. First we create the first 2
containers for:
● AbadIA Game Engine
● AbadIA Agent

30. K8S small cluster

31. Cluster Features:
● Preemtible
● n1-standard-1 (7€
month)

32. Gathering
Information

33. Game info basic pipeline

34. AutoGaming

35. Game Generator
Games
Checkpoints

36. So, where are the
AI pipelines?

37. ML generic pipeline

38. So, where are the
AbadIA pipelines?

39. Short answer:
Github ;-P

40. What we have:

41. Game Actions Data

42. Two versions:
V6 Winter(75 params)
V7 Abad Chinao (500 params)

43. Step 1:
Get the raw data from a
Google Storage Bucket

44. Step 2:
Queue the last raw data or any
other criteria into a pub sub queue

45. Step 3:
Convert raw data into a
normalized vector

46. Step 3:
It not just normalized data, Temporary
Difference (juantomas version kind of) is
applied here, featuring engineering, etc

47. Step 4:
Launch a ML training process
with this bunch of games (transfer
learning like)

48. Step 5:
Export the new model, save same
metadata, ready to be used
(production)

49. ML generic pipeline

50. My current pipeline (ascii art
version)
Last raw data -> PubSub -> K8S
Job Pre-training -> Pubsub -> K8
s job Training -> New models
Bucket

51. Looking for new tools

52. My favorite Features

53. • Simple vs Complex
• Managed
• Serverless
• No vendor locking
• Cheap
• Fast
• Ephimeral

54. Kubeflow

56. Tensorflow TFX

57. ML Flow

58. MLflow

60. Airflow + Spark ML + Dask
+ Scikit Pipelines + Laks +
(fill your favorite options)

61. Show me the code
(how to collaborate)

62. GitHub
LaAbadIAdelCrimen
https://github.com/LaAbadIAdelCrimen
HOW TO COLLABORATE

63. Slack AbadIA
JOIN US!!!!

64. Join US, we’ll share
the glory with you

65. Takeaways
• Don’t Over Engineering
• It’s all about data
• Marios Cap
• Serverless
• RF is simple, powerful and
Easy
• Lots of tools to use.

66. Questions?
This talk have a free questions lifetime warranty: If you have any questions or concerns
about this talk, feel free to contact me anytime.
Selfie Time: If you like the talk just smile while I take
the selfie ;-)
We’re Hiring, Sngular People
twitter: @juantomas
juantomas.garcia@sngular.com

67. Thank You!!