The document outlines a project to develop an AI that plays a game focused on finding the shortest path to a princess using a policy network and evolutionary strategies. It describes the game's mechanics, the proposed computational infrastructure, and the benefits of using evolutionary strategies over traditional reinforcement learning methods. The author emphasizes the importance of efficient coding, cloud-scale computation, and structured data processing for enhancing the AI's learning experience.

1. Save the princess!
Simon Belak
@sbelak
simon@metabase.com

2. We will build an AI to play a silly little game
by training a policy network defined using
Cortex, using a hot new training algorithm we
will implement from the paper first using
Neanderthal and then make massively
parallel using Onyx.

3. The game
• Find the shortest path to the princess
• Moves: up, down, left, right
• Don’t fall off the edge of the world

4. The game
• Find the shortest path to the princess
• Moves: up, down, left, right
• Don’t fall off the edge of the world

5. Computers playing
computer games

6. Reinforcement learning
• Interact with the environment [embodied cognition]
• Not a single solution but an action to take given environment
[model of the world + model of self, consciousness?]
• Learns via positive/negative feedback

7. Reinforcement learning:
how it’s usually done
Train a deep neural network using raw sensor
data, usually pixels (ie. no feature engineering)

8. … but there is another way

10. population
mutate crossover
next generation
solution
jitter jitter … jitter
update
populate
sample weighted
Classic evolutionary algorithm Evolution strategies
combine weighted

11. Using ES to train a neural
network
Benefits
• highly parallelizable
• more robust (less hyperparameters, more
stabile, doesn’t care about the properties of
reward function)
• can exploit structure
• less computationally expensive
Downsides
• takes longer to converge
• noise must lead to different outcomes
Instead of backpropagation use ES on weights

12. Let’s build it!

13. 1. ES

15. Neanderthal
• Blazing fast matrix and linear algebra library
• Based on ATLAS and LAPACK
• Runs on CPUs and GPUs
• A study in writing efficient code
• Somewhat terse API (fluokitten helps)

20. x+y
ax+y
ax+by

21. x+y
ax+y
ax+by

22. x+y
ax+y
ax+by

23. x+y
ax+y
ax+by

26. 1.1 ES parallelized

27. Onyx
a masterless, cloud scale, fault tolerant,
high performance distributed computation
system

28. Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]

29. [{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]
Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue
Describing computation
with data

31. in
jitter jitter … jitter
update
out
monitor
populate

32. same channel
in
jitter jitter … jitter
update
out
monitor
populate

33. accumulates state :(
in
jitter jitter … jitter
update
out
monitor
populate

34. Resilience and handling
state
• Activity log
• Window and trigger states checkpointed
• Resume points (transfer state from job to job)
• Configurable flux policies (continue/kill/recover)

35. Computation graphs are
a great way to structure
data processing code

36. 2. Policy network

37. Cortex
• Neural networks, regression and feature learning
• Clean idiomatic Clojure API
• Computation encoded as data (and makes good use of it)
• Uses core.matrix for heavy lifting

38. Encode princess = 1, hero = -1

39. 3. Game

40. Simulation
• Find the shortest path to the
princess
• Don’t fall off the edge of the world

41. Reward function
• Play the entire game (planning)
• Collect multiple playthoughts to lessen effects of
randomness

42. Takeouts

43. Explore
Have fun
Go on an adventure!

44. Questions
Simon Belak
@sbelak
simon@metabase.com