This document summarizes a presentation on deep reinforcement learning. It introduces Markov decision processes and describes how reinforcement learning can be used for prediction and control tasks. It then discusses different deep reinforcement learning algorithms like DQN for value-based methods and policy gradient methods like REINFORCE, TRPO, and PPO for policy-based optimization. Actor-critic algorithms like A3C and DDPG that combine value and policy learning are also covered. The presentation provides an overview of core concepts and algorithms in deep reinforcement learning.

1. DEEP
REINFORCEMENT
LEARNING
ON THE ROAD TO SKYNET!
UW CSE Deep Learning – Felix Leeb

2. OVERVIEW
TODAY
→ MDPs – formalizing decisions
→ Function Approximation
→ Value Function – DQN
→ Policy Gradients – REINFORCE,
NPG
→ Actor Critic – A3C, DDPG
NEXT TIME
→ Model Based RL –
forward/inverse
→ Planning – MCTS, MPPI
→ Imitation Learning – DAgger,
GAIL
→ Advanced Topics – Exploration,
MARL, Meta-learning, LMDPs…
UW CSE DEEP LEARNING - FELIX LEEB 2

3. Supervised
Learning
→ Classificati
on
→ Regression
Unsupervised
Learning
→ Inference
→ Generati
on
Reinforcemen
t Learning
→ Predictio
n
→ Control
Objective
Applications
Paradigm
UW CSE DEEP LEARNING - FELIX LEEB 3

4. Prediction
Control
𝑥 𝑦
𝑦
𝑥
𝑢
UW CSE DEEP LEARNING - FELIX LEEB 4

5. SETTING
UW CSE DEEP LEARNING - FELIX LEEB 5
Agent
Environm
ent
Action
State/Observatio
n
Reward
using policy

6. MARKOV PROCESSES
UW CSE DEEP LEARNING - FELIX LEEB 6
State
space
Action
space
Transition
function
Reward
function

7. DISCOUNT FACTOR
→ We want to be greedy but not impulsive
→ Implicitly takes uncertainty in dynamics into
account
→ Mathematically: γ<1 allows infinite horizon
returns
UW CSE DEEP LEARNING - FELIX LEEB 7
Return:

8. SOLVING AN MDP
Goal:
UW CSE DEEP LEARNING - FELIX LEEB 8
Objectiv
e:

9. VALUE FUNCTIONS
→ Value = expected gain of a state
→ Q function – action specific value function
→ Advantage function – how much more valuable is
an action
→ Value depends on future rewards  depends on
policy
UW CSE DEEP LEARNING - FELIX LEEB 9

10. TABULAR SOLUTION: POLICY ITERATION
Policy
Evaluatio
n
Policy
Update
UW CSE DEEP LEARNING - FELIX LEEB 10

11. Q LEARNING
UW CSE DEEP LEARNING - FELIX LEEB 11

12. FUNCTION APPROXIMATION
UW CSE DEEP LEARNING - FELIX LEEB 12
Model:
Training
data:
Loss
function: wher
e

13. IMPLEMENTATION
UW CSE DEEP LEARNING - FELIX LEEB 13
Action-in Action-out Off-Policy
Learning
→ The target depends in
part on our model 
old observations are
still useful
→ Use a Replay Buffer of
most recent
transitions as dataset

14. DEEP Q NETWORKS (DQN)
UW CSE DEEP LEARNING - FELIX LEEB 14
Mnih et al. (2015)

15. DQN ISSUES
UW CSE DEEP LEARNING - FELIX LEEB 15
→ Convergence is not guaranteed – hope for deep magic!
Replay Buffer Error Clipping Reward scaling Using replicas
→ Double Q Learning – decouple action selection and value
estimation

16. POLICY GRADIENTS
UW CSE DEEP LEARNING - FELIX LEEB 16
→ Parameterize policy and update those parameters directly
→ Enables new kinds of policies: stochastic, continuous
action spaces
→ On policy learning  learn directly from your actions

17. POLICY GRADIENTS
UW CSE DEEP LEARNING - FELIX LEEB 17
→ Approximate expectation value from
samples

18. REINFORCE
UW CSE DEEP LEARNING - FELIX LEEB 18
Sutton et al. (2000)

19. VARIANCE REDUCTION
UW CSE DEEP LEARNING - FELIX LEEB 19
→ Constant offsets make it harder to
differentiate the right direction
→ Remove offset  a priori value of each
state

20. ADVANCED POLICY GRADIENT METHODS
UW CSE DEEP LEARNING - FELIX LEEB 20
→ For stochastic functions, the gradient is not the best
direction
→ Consider the KL divergence
Approximating the Fisher information
matrix
Computing gradients with KL constraint
Gradients with KL penalty
NPG 
TRPO 
PPO 

21. ADVANCED POLICY GRADIENT METHODS
UW CSE DEEP LEARNING - FELIX LEEB 21
Rajeswaran et al. (2017) Heess et al. (2017)

22. ACTOR CRITIC
UW CSE DEEP LEARNING - FELIX LEEB 22
Critic
Actor
Estimate
Advantag
e
Propose
Actions
using Q learning update
using policy gradient
update

23. ASYNC ADVANTAGE ACTOR-CRITIC (A3C)
UW CSE DEEP LEARNING - FELIX LEEB 23
Mnih et al. (2016)

24. DDPG
UW CSE DEEP LEARNING - FELIX LEEB 24
Max Ferguson (2017)
→ Off-policy learning – using deterministic policy gradients