This document summarizes and compares recent research on hierarchical reinforcement learning and model-based reinforcement learning using neural networks. It discusses work on reward augmentation, hierarchical actions, model-based video prediction, Value Iteration Networks, and the Predictron. Value Iteration Networks allow planning using value iteration in a neural network by treating the Bellman operator as a convolutional layer. The Predictron enables end-to-end learning and simulation of Markov reward processes without requiring interpretability of the state space. Future research directions include theoretical bounds for optimal policies on abstracted MDPs and learning smaller MDPs with hierarchical actions.

1. The Predictron: End-to-end Learning and
Planning
Yoonho Lee
Department of Computer Science and Engineering
Pohang University of Science and Technology
December 27, 2016

2. Hierarchical RL
motivation

3. Hierarchical RL

4. Hierarchical RL

5. Reward augmentation1
1
Bellemare et al. Unifying Count-Based Exploration and Intrinsic
Motivation, NIPS 2016

6. Hierarchical RL

7. Hierarchical actions2
2
Florensa et al. Stochastic Neural Networks for Hierarchical Reinforcement
Learning, under review for ICLR 2017

8. Hierarchical RL

9. Model-based video prediction3
3
Oh et al. Action-Conditional Video Prediction using Deep Networks in
Atari Games, NIPS 2015

10. Value Iteration Networks
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, Pieter Abbeel
Best paper at NIPS 2016

11. Value Iteration Networks
Motivation

12. Value Iteration Networks
Network diagram

13. Value Iteration Networks
Bellman Operator as a CNN forward pass
Bellman Operator
Every V converges to V ∗ under the Bellman Operator T defined
as:
(TV )(s) = max
a∈A
{R(s, a) + γ
s ∈S
P(s |s, a)V (s )} (1)
V ∗
= lim
n→∞
Tn
V ∀V (2)
The Bellman Operator can be viewed as a CNN forward pass:
(T∗
V )(s) = max
a∈A
{R(s, a) + γconvP(a)(V (s))}

14. Value Iteration Networks
VI Module

15. Value Iteration Networks
Results

16. Value Iteration Networks
Summary
Neural network architecture that plans using value iteration
Assumes that the state is a sufficient statistic for the reward
function
The small MDP must have a finite state space
Uses prior knowledge about the environment’s structure

17. The Predictron: End-to-End Learning and Planning
David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul,
Arthur Guez, Tim Harley, Gabriel Dulac-Arnold, David Reichert,
Neil Rabinowitz, Andre Barreto, Thomas Degris
Under review for ICLR 2017

18. Predictron

19. Predictron

20. Predictron

21. Predictron

22. Predictron

23. Predictron

24. Predictron

25. Predictron
Summary
End-to-end simulation of an MRP
Works with arbitrary state space for small MRP
Small MRP has a non-interpretable state space
Designed for RL, but does not take actions into account

26. Summary
Hierarchical Reinforcement Learning attempts to identify
crucial decisions
Agents can now use NN-based planning for better decision
making
Research directions
Theoretical bounds for optimal policy of a smaller MDP
Learning a smaller MDP with abstract actions
End-to-end planning based Q network or policy network

27. Thank You
Value Iteration Networks
The Predictron: End-to-End Learning and Planning