Deep Reinforcement Learning by Pieter

How to Make Deep Reinforcement Learning less Data Hungry
Pieter Abbeel
Embodied Intelligence
UC Berkeley
Gradescope
OpenAI
AI for robo<c automa<on
AI research
AI for grading homework and exams
AI research / advisory role

Reinforcement Learning (RL)
Robot +
Environment
max
✓
E[
HX
t=0
R(st)|⇡✓]
[Image credit: Sutton and Barto 1998] Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope

Reinforcement Learning (RL)
Robot +
Environment
Compared ton
supervised learning,
addi3onal challenges:
Credit assignmentn
Stabilityn
Explora3onn
[Image credit: Sutton and Barto 1998]
max
✓
E[
HX
t=0
R(st)|⇡✓]
Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope

Deep RL Success Stories
DQN Mnih et al, NIPS 2013 / Nature 2015
MCTS Guo et al, NIPS 2014; TRPO Schulman, Levine, Moritz, Jordan, Abbeel, ICML 2015; A3C Mnih et al,
ICML 2016; Dueling DQN Wang et al ICML 2016; Double DQN van Hasselt et al, AAAI 2016; Prioritized
Experience Replay Schaul et al, ICLR 2016; Bootstrapped DQN Osband et al, 2016; Q-Ensembles Chen et al,
2017; Rainbow Hessel et al, 2017; …

AlphaGo Silver et al, Nature 2015
AlphaGoZero Silver et al, Nature 2017
Tian et al, 2016; Maddison et al, 2014; Clark et al, 2015
AlphaZero Silver et al, 2017

n Super-human agent on Dota2 1v1, enabled by
n Reinforcement learning
n Self-play
n Enough computation
[OpenAI, 2017] Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope

^ TRPO Schulman et al, 2015 + GAE Schulman et al, 2016
See also: DDPG Lillicrap et al 2015; SVG Heess et al, 2015; Q-Prop Gu et al, 2016; Scaling up ES Salimans et
al, 2017; PPO Schulman et al, 2017; Parkour Heess et al, 2017;

Guided Policy Search Levine*, Finn*, Darrell, Abbeel, JMLR 2016

[Geng*, Zhang*, Bruce*, Caluwaerts, Vespignani, Sunspiral, Abbeel, Levine, ICRA 2017] Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope

Mastery? Yes
Deep RL (DQN) vs. Human
Score: 18.9 Score: 9.3

But how good is the learning?
Deep RL (DQN) vs. Human
Score: 18.9 Score: 9.3
#Experience
measured in real
time: 40 days
#Experience
measured in real
time: 2 hours
“Slow” “Fast”

Humans vs. DDQN
Black dots: human play
Blue curve: mean of human play
Blue dashed line: ‘expert’ human play
Red dashed lines:
DDQN a@er 10, 25, 200M frames
(~ 46, 115, 920 hours)
[Tsividis, Pouncy, Xu, Tenenbaum, Gershman, 2017] Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope
Humans after 15 minutes tend to outperform DDQN after 115 hours

How to bridge this gap?

Outline
Hindsight Experience Replayn
Metan -learning – learn-to-learn faster

n On-Policy Algorithms:
n Can only perform update based on data collected under current policy
n Policy gradients, A3C, TRPO
n Atari: Most wall-clock efficient
n Off-Policy Algorithms:
n Can re-use all past data for updates (“experience replay”)
n Q learning, DQN, DDPG
n Atari: Most sample efficient
On-Policy vs. Off-Policy
X
k from replay bu↵er
(Q✓(sk, ak) (rk + max
a
Q✓OLD
(sk+1, a))2

But Signal?

Main idea: Get reward signal from any experience by simply assuming then
goal equals whatever happened
Representa=on learned: Universal Qn -func=on Q(s, g, a)
Replay buﬀer:n
Actual experience:n
Experience replay:n
Hindsight replay:n
Hindsight Experience Replay
[Andrychowicz et al, 2017] [Related work: Cabi et al, 2017]
(st, gt, at, rt, st+1)
X
(Q✓(sk, gk, ak) (rk + max
a
Q✓OLD
(sk+1, gk, a))2
X
(Q✓(sk, g0
k,ak) (r0
k + max
a
Q✓OLD
(sk+1, g0
k, a))2

Hindsight Experience Replay: Pushing, Sliding, Pick-and-Place
Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope[Andrychowicz, Wolski, Ray, Schneider, Fong, Welinder, McGrew, Tobin, Abbeel, Zaremba, NIPS2017]

Quan%ta%ve Evalua%on
~3s/episode à 1.5 days for 50 epochs, 6 days for 200 epochs

Outline
n Hindsight Experience Replay
n Meta-learning – learn-to-learn faster

Starting Observations
n TRPO, DQN, A3C, DDPG, PPO, Rainbow, … are fully general RL
algorithms
n i.e., for any environment that can be mathematically defined, these
algorithms are equally applicable
n Environments encountered in real world
= tiny, tiny subset of all environments that could be defined
(e.g. they all satisfy our universe’s physics)
Can we develop “fast” RL algorithms that take advantage of this?

Reinforcement Learning
[Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel, 2016] also: [Wang et al, 2016]
Agent
Environment
asr

RL Algorithm
Policy
Environment
aor
RL Agent

RL Algorithm
Policy
Environment A
asr
…
RL Agent
RL Algorithm
Policy
Environment B
asr
RL Agent

…
RL Algorithm
Policy
Environment A
aor
RL Algorithm
Policy
Environment B
aor
RL Agent RL Agent
Traditional RL research:
• Human experts develop
the RL algorithm
• After many years, still
no RL algorithms nearly
as good as humans…
Alternative:
• Could we learn a better
RL algorithm?
• Or even learn a better
entire agent?

Meta-Reinforcement Learning
"Fast” RL
Agent
Environment A
Meta RL
Algorithm
Environment B
…
Meta-training environments

"Fast” RL
Agent
Environment A
ar,o
Meta RL
Algorithm
Environment B
…
Environment F
Testing environments

"Fast” RL
Agent
Environment A
ar,o
Meta RL
Algorithm
Environment B
…
Environment G
Testing environments

[Duan, Schulman, Chen, Bartle:, Sutskever, Abbeel, 2016] also: [Wang et al, 2016]
"Fast” RL
Agent
Environment A
ar,o
Meta RL
Algorithm
Environment B
…
Environment H
TesNng environments

Formalizing Learning to Reinforcement Learn
[Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel, 2016] also: [Wang et al, 2016] Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope
max
✓
EM E⌧
(k)
M
" KX
k=1
R(⌧
(k)
M ) | RLagent✓
#

Formalizing Learning to Reinforcement Learn
[Duan, Schulman, Chen, Bartle:, Sutskever, Abbeel, 2016] also: [Wang et al, 2016]
M : sample MDP
⌧
(k)
M : k’th trajectory in MDP M
max
✓
EM E⌧
(k)
M
" KX
k=1
R(⌧
(k)
M ) | RLagent✓
#
Meta-train:
max
✓
X
M2Mtrain
E⌧
(k)
M
" KX
k=1
R(⌧
(k)
M ) | RLagent✓
#

n RLagent = RNN = generic computation architecture
n different weights in the RNN means different RL algorithm and prior
n different activations in the RNN means different current policy
n meta-train objective can be optimized with an existing (slow) RL algorithm
Representing RLagent✓
max
✓
X
M2Mtrain
E⌧
(k)
M
" KX
k=1
R(⌧
(k)
M ) | RLagent✓
#
Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope[Duan, Schulman, Chen, BartleN, Sutskever, Abbeel, 2016] also: [Wang et al, 2016]

Evaluation: Multi-Armed Bandits
Mul5n -Armed Bandits se6ng
Each bandit has its own distribu5onn
over pay-outs
Each episode = choose 1 banditn
Good RL agent should exploren
bandits suﬃciently, yet also exploit
the good/best ones
Provably (asympto5cally) op5maln
RL algorithms have been invented
by humans: Gi6ns index, UCB1,
Thompson sampling, …
[Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel, 2016] Pieter Abbeel -- embody.ai / UC Berkeley / Gradescope

[Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel, 2016]
Evaluation: Multi-Armed Bandits
We consider Bayesian evaluaEon seFng. Some of these prior works also have adversarial
guarantees, which we don’t consider here.

Evaluation: Visual Navigation
Agent’s view Maze
Agent input: current image
Agent action: straight / 2 degrees left / 2 degrees right
Map just shown for our purposes, but not available to agent
Related work: Mirowski, et al, 2016; Jaderberg et al, 2016; Mnih et al, 2016; Wang et al, 2016

Evaluation: Visual Navigation
After learning-to-learnBefore learning-to-learn
[Duan, Schulman, Chen, Bartlett, Sutskever, Abbeel, 2016]
Agent input: current image
Agent ac,on: straight / 2 degrees leE / 2 degrees right
Map just shown for our purposes, but not available to agent

n Simple Neural Attentive Meta-Learner (SNAIL)
[Mishra, Rohaninejad, Chen, Abbeel, 2017]
n Model-Agnostic Meta-Learning (MAML)
[Finn, Abbeel, Levine, 2017; Finn*, Yu*, Zhang, Abbeel, Levine 2017]
Other Meta RL Architectures

Hindsight Experience Replayn
Metan -learning – learn-to-learn faster
Summary

OpenAIn Gym: h-ps://gym.openai.com
OpenAIn baselines: h-ps://github.com/openai/baselines
rllabn : h-ps://github.com/rll/rllab
Want to Try Things Out?

n Deep RL Bootcamp [intense long weekend]
n 12 lectures by: Pieter Abbeel, Rocky Duan, Peter Chen, John Schulman,
Vlad Mnih, Chelsea Finn, Sergey Levine
n 4 hands-on labs
n https://sites.google.com/view/deep-rl-bootcamp/
n Deep RL Course [full semester]
n Originated by John Schulman, Sergey Levine, Chelsea Finn
n Latest offering (by Sergey Levine):
http://rll.berkeley.edu/deeprlcourse/
Want to learn more?

Fast Reinforcement Learningn
Long Horizon Reasoningn
Taskabilityn
Lifelong Learningn
Leverage Simula;onn
Maximize Signal Extractedn
Many Exciting Directions
n Safe Learning
n Value Alignment
n Planning + Learning
n …

Thank you
@pabbeel
pabbeel@cs.berkeley.edu

Deep Reinforcement Learning by Pieter

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deep Reinforcement Learning by Pieter

Similar to Deep Reinforcement Learning by Pieter (20)

More from Bill Liu

More from Bill Liu (20)

Recently uploaded

Recently uploaded (20)

Deep Reinforcement Learning by Pieter