NYAI #25: Evolution Strategies: An Alternative Approach to AI w/ Maxwell Rebo at Capital One Labs on Tues, 10/23/18 Join us for what's sure to be an awesome night in AI! This month's event is focused Evolution Strategies, and will touch on many themes discussed here (https://blog.openai.com/evolution-strategies/). Maxwell Rebo is a machine learning founder working on a stealth project in ML-powered simulation engine. A class of heuristic search algorithms have been shown to be viable alternatives to reinforcement learning as well as other ML tasks. These methods can be parallelized on arbitrary numbers of CPUs and do not require GPUs to be effective. To increase explicability, it is possible to create attribution mechanisms within these methods. Maxwell is the former founder of Machine Colony, and enterprise AI platform company, and a founding member of NYAI. A machine learning developer and three-time founder, he has been doing ML at massive scale since 2010. He has previously spoken at venues such as the Ethereal conference in NYC and the joint Asian Leadership/HelloTomorrow conference in Seoul.

1. Evolution Strategies
An Alternative Approach to AI

2. What is ES (and who uses it)?
● Class of heuristic search algorithms for black-box optimization
● Closely related to genetic algorithms (GA)
● Several groups have made serious commitments to developing additional
variants:

3. Why should I care?
Organizations have yet to adopt reinforcement learning, and as such are leaving
billions of dollars of value on the table
ES presents a perfect excuse to tackle serious problems with RL
It’s no panacea, but hard optimization problems become much easier
DRL is heavy and esoteric; ES is lightweight and intuitive
Once ES becomes explicable, then it’s game over

4. What do I use ES for?
● Algorithmic trading (c’mon, you knew that was going to come up)
● Token incentives in blockchain applications
● Opportunity cost problems in investment and staking
● Training one-shot computer vision models
● Modeling causal trends in property markets (2016-2017)

5. Pros
● Gradient-free
● Can run on arbitrary numbers
of CPUs, no GPUs needed
● Can optimize mixed-type
models
● Can handle non-convex
problems
● Conveniently sidesteps
exploding gradients
● Easy to extend/modify
Cons
● Less data-efficient
● No guarantee of convergence
● Still a black box*
● Without modifications, does
poorly with hard constraints as
reward cutoffs
● Can break down with highly
complex/multimodal solutions
● Can be much slower in typical
supervised learning settings

6. Core Procedure
Credit: OpenAI, https://arxiv.org/abs/1703.03864

7. 2D visualization
Image credit: OpenAI

8. DRL ES
● Optimizing arbitrary parameters
for a given task
● Largely agnostic to underlying
model structure
● Can implement virtual batch
schemes, but not required for
good performance
● Value function is implicit
● Parameters are smooshed into
1D array
● Optimizing network(s) for a
given task, typically at
episode-level
● Highly sensitive to batch size
and training scheme
● Requires estimation of value
function
● Parameters are grouped by
component/layer

9. How to gray the black box?
● Feature importance / parameter saliency still a challenge
● Tried Bayesian attribution: too sparse, and still get artifacts from discretization
● One way that works:
○ Calculate MI(x_i’, y’)
○ x_i’ is Δε
○ y’ is change in reward for candidate solution
● For pairwise and partition-wise attribution, do partial information
decomposition in hierarchical fashion
● No free lunch: Attribution computation is expensive

10. PRESS
● PRincipled Evolution StrategieS (in keeping with weird acronym structures)
● Saliency attribution at the optimizer level
● Short version: find which parameters are important and which ones aren’t
● Long version: Use information-theoretic measures to attribute reward
saliency to parameter perturbations
● Step 1) estimate parameter saliency after N iterations
Step 2) apply saliency estimates as normalized prior in perturbation-sampling
step
● Bears similarity to integrated gradients, guided ES
● Essentially an approach to estimating the partial derivative ∂y/∂x_i
○ Where y is reward, x_i is parameter of a model such as a weight in a neural net, mean of a
probability distribution, etc.

11. Summary
● For RL problems, using ES solvers confers a number of benefits
● Where problems are mixed-mode and/or gradients are difficult to to evaluate,
ES shines
● Getting away from GPU dependency is especially nice - less $$, for one
● Impact on industry: Compared to other RL methods, ES is easier to apply,
faster, and often yields comparable or even better results. This makes it
easier and cheaper to deploy practitioners to a problem and get results.
● Areas of especially exciting application are synthetic biology, economics
(including economic policy), multi-agent modeling and simulation, and
diagnostic medicine

12. Resources & Reading
● OpenAI’s intro post: https://blog.openai.com/evolution-strategies/
● Great Python implementation: https://github.com/hardmaru/estool
● Google Brain’s variant: https://github.com/brain-research/guided-evolutionary-strategies
● Uber’s variants: https://arxiv.org/abs/1712.06560
● Uber’s implementations: https://github.com/uber-research/deep-neuroevolution
Blog post: https://eng.uber.com/deep-neuroevolution/
● OpenAI’s starter: https://github.com/openai/evolution-strategies-starter
● Natural ES paper: http://www.jmlr.org/papers/volume15/wierstra14a/wierstra14a.pdf