OpenAI recently published a fun paper where they showed using evolution algorithms to train policy networks to perform on par with state of the art reinforcement deep learning. In this talk we’ll try to reimplement the main ideas in that paper using Neanderthal (blazing fast matrix and linear algebra computations) and Cortex (neural networks); make it massively distributed using Onyx; build a simulation environment using re-frame; and of course save our princess from no particular harm in our toy game example

1. Save the princess!
Simon Belak
@sbelak
simon@metabase.com

2. We will build an AI to play a silly little game
by training a policy network defined using
Cortex, using a hot new training algorithm we
will implement from the paper first using
Neanderthal and then make massively
parallel using Onyx.

3. The game
• Find the shortest path to the princess
• Moves: up, down, left, right
• Don’t fall off the edge of the world

4. The game
• Find the shortest path to the princess
• Moves: up, down, left, right
• Don’t fall off the edge of the world

5. Computers playing
computer games

6. Reinforcement learning
• Interact with the environment [embodied cognition]
• Not a single solution but an action to take given environment
[model of the world + model of self, consciousness?]
• Learns via positive/negative feedback

7. Reinforcement learning:
how it’s usually done
Train a deep neural network using raw sensor
data, usually pixels (ie. no feature engineering)

8. … but there is another way

10. population
mutate crossover
next generation
solution
jitter jitter … jitter
update
populate
sample weighted
Classic evolutionary algorithm Evolution strategies
combine weighted

11. Using ES to train a neural
network
Benefits
• highly parallelizable
• more robust (less hyperparameters, more
stabile, doesn’t care about the properties of
reward function)
• can exploit structure
• less computationally expensive
Downsides
• takes longer to converge
• noise must lead to different outcomes
Instead of backpropagation use ES on weights

12. Let’s build it!

13. 1. ES

15. Neanderthal
• Blazing fast matrix and linear algebra library
• Based on ATLAS and LAPACK
• Runs on CPUs and GPUs
• A study in writing efficient code
• Somewhat terse API (fluokitten helps)

20. x+y
ax+y
ax+by

21. x+y
ax+y
ax+by

22. x+y
ax+y
ax+by

23. x+y
ax+y
ax+by

26. 1.1 ES parallelized

27. Onyx
a masterless, cloud scale, fault tolerant,
high performance distributed computation
system

28. Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue[{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]

29. [{:onyx/name :add-5
:onyx/fn :my/adder
:onyx/type :function
:my/n 5
:onyx/params [:my/n]}
{:onyx/name :in
:onyx/plugin :onyx.plugin.core-async/input
:onyx/type :input
:onyx/medium :core.async
:onyx/batch-size batch-size
:onyx/max-peers 1
:onyx/doc "Reads segments from a core.async channel"}
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/type :output
:onyx/medium :core.async
:onyx/doc "Writes segments to a core.async channel"}]
Job =
[[:input :processing-1]
[:input :processing-2]
[:processing-1 :output-1]
[:processing-2 :output-2]]
[{:flow/from :input-stream
:flow/to [:process-adults]
:flow/predicate :my.ns/adult?
:flow/doc "Emits segment if an adult.”}]
workflow
+ flow conditions
+ catalogue
Describing computation
with data

31. in
jitter jitter … jitter
update
out
monitor
populate

32. same channel
in
jitter jitter … jitter
update
out
monitor
populate

33. accumulates state :(
in
jitter jitter … jitter
update
out
monitor
populate

34. Resilience and handling
state
• Activity log
• Window and trigger states checkpointed
• Resume points (transfer state from job to job)
• Configurable flux policies (continue/kill/recover)

35. Computation graphs are
a great way to structure
data processing code

36. 2. Policy network

37. Cortex
• Neural networks, regression and feature learning
• Clean idiomatic Clojure API
• Computation encoded as data (and makes good use of it)
• Uses core.matrix for heavy lifting

38. Encode princess = 1, hero = -1

39. 3. Game

40. Simulation
• Find the shortest path to the
princess
• Don’t fall off the edge of the world

41. Reward function
• Play the entire game (planning)
• Collect multiple playthoughts to lessen effects of
randomness

42. Takeouts

43. Explore
Have fun
Go on an adventure!

44. Questions
Simon Belak
@sbelak
simon@metabase.com