December 2016

1. Deep RL with Policy
Optimization
Evan Casey

2. Reinforcement Learning
Figure source: Sutton & Barto, 1998

3. Policy Optimization
Figure source: Sutton & Barto, 1998

4. Defining the Objective
Cumulative reward
for the trajectory
Joint probability of individual
state-action pair probs:

5. Why Policy Gradients?
Learning policy directly instead of Q or V
Don’t need dynamics model:
Vanilla Q-learning is intractable for large action spaces

6. How does it work?

7. The Policy Gradient Theorem

8. Using Empirical Estimates

9. REINFORCE
Williams, 1992

10. Intuition
Gradient tries to:
● Increase probability of paths with
positive
● Decrease probability of paths with
negative
Figure source: Schulman & Abbeel

11. Adding a baseline
Recall:

12. Generalized advantage actor-critic

13. Implementation

14. Implementation

15. Implementation

16. Implementation (GAE)

17. Results

18. A quick note on the cross-entropy method (CEM)

19. A3C
Entropy
regularization term
Minh, et al. 2016

20. A3C Results
Minh, et al. 2016

21. Active areas of research
New environments!
Better sample efficiency
Transfer learning
Perception
Exploration/Auxiliary Tasks

22. Tensorflow tips and tricks
Use tf summaries for bookkeeping
Variable scoping for re-use
● tf.get_collection(tf.GraphKeys.VARIABLES)
Global counters
Coordinator and server APIs for multi-threaded/distributed
training
Gradient clipping

23. Tensorboard

24. RL tips and tricks
Standardize your rollouts
Batch size makes a big difference
Neural net architecture doesn’t matter that much:
● batch norm, dropout, etc
Policy gradients don’t benefit as much from off-policy exploration (eg.
e-greedy)

25. Sources
NIPS Tutorial (Schulman and Abbeel):
http://people.eecs.berkeley.edu/~pabbeel/nips-tutorial-policy-optimization-Schulm
an-Abbeel.pdf
Source code:
http://github.com/evancasey/demeter