This document provides an introduction to deep reinforcement learning. It begins with background sections on deep learning and reinforcement learning. Deep learning involves using neural networks to learn representations of data, while reinforcement learning describes how agents can learn behaviors through trial-and-error interactions with an environment. The document then explains how deep reinforcement learning combines the two by using deep neural networks to approximate value and policy functions for reinforcement learning agents. Examples of deep reinforcement learning successes in playing Atari games are provided.

1. Introduction to Deep
Reinforcement Learning
Shenglin Zhao
Department of Computer Science & Engineering
The Chinese University of Hong Kong

2. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

3. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

4. Milestone Issues
• NIPS 2013, DeepMind, Playing Atari with Deep Reinforcement Learning,
https://arxiv.org/abs/1312.5602
• Nature cover paper 2015, DeepMind, Human-level control through deep
reinforcement learning, www.nature.com/articles/nature14236
• Nature cover paper 2016, DeepMind, Mastering the game of Go with deep
neural networks and tree search, www.nature.com/articles/nature16961

5. Reinforcement Learning in a nutshell
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

6. Deep Learning in a nutshell
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

7. Deep Reinforcement Learning:AI = RL + DL
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

8. Examples of Deep RL@DeepMind
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

9. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

10. Deep Learning = Learning Representations/Features
• Traditional Model
• Deep Learning
http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf

11. Deep Representations
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

12. Deep Neural Network
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

13. Example-CNN
http://xrds.acm.org/blog/2016/06/convolutional-neural-networks-cnns-illustrated-explanation/
Convolutional Operator
Discrete Form
Matrix Element-wise Multiplication
1 2 0
0 1 0
0 0 1
2 2 7
5 0 7
1 2 1
= 7

14. Example-CNN
https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
Pixel representation of filter Visualization

15. Example-CNN
https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
Original image What we find

16. Example-CNN
http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf
Low-Level
Feature
Mid-Level
Feature
High-Level
Feature
Trainable
Classifier

17. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

18. Reinforcement Learning
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

19. Agent and Environment
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf
Agent
Environment

20. State
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

21. Major Components
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

22. Policy
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

23. Value Function
Bellman equation
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

24. Optimal Case
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

25. Approaches to Reinforcement Learning
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

26. RL Example
• Assumption
• Suppose we have 5 rooms in a building connected by doors
• The outside of the building can be thought of as one big room (5)
• Target
• Put an agent in any room, and from that room, go outside the building
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

27. RL Example

28. Q-learning for the RL Problem
• Assuming rewards for each step, the goal is to reach the state with
the highest reward.
• Terms— state: room, action: move decision, reward: 0 or 100
RL Problem Markov Decision Process
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

29. Q-learning for the RL Problem
Reward Table Q-value Table

30. Q-learning for the RL Problem
• Q-table is the brain of our agent, representing the memory of what
the agent has learned through experience.
• The agent starts out knowing nothing, the matrix Q is initialized to
zero.
• Simple transition rule of Q learning,
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

31. Q-learning for the RL Problem
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

32. Q-learning for the RL Problem
• Example
• Initial state: room 1, action: move to 5
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

33. Q-learning for the RL Problem
• Example
• Initial state: room 3, action: move to 1
Q(3, 1) = 0 + 0.8 * 100 = 80
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

34. Q-learning for the RL Problem
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

35. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusions

36. Deep Reinforcement Learning
http://videolectures.net/rldm2015_silver_reinforcement_learning/
Deep
Learning
Reinforcement
Learning
Deep
Reinforcement
Learning

37. Deep Q-learning
Deep Q-Network (DQN)

38. DQN Architecture
Naive formulation of deep Q-network DQN in DeepMind paper
https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

39. DQN Architecture
https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

40. DQN
Loss function
Q-table update algorithm
https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

41. Exploration-Exploitation

42. Value Iteration

43. Policy Iteration

44. Stability Issues with Deep RL
http://videolectures.net/rldm2015_silver_reinforcement_learning/

45. DQN
http://videolectures.net/rldm2015_silver_reinforcement_learning/

46. Experience Replay
http://videolectures.net/rldm2015_silver_reinforcement_learning/

47. Fixed Target Q-Network
http://videolectures.net/rldm2015_silver_reinforcement_learning/

48. Reward/Value Range
http://videolectures.net/rldm2015_silver_reinforcement_learning/

49. DQN Results in Atari

50. DQN Atari Demo

51. Conclusion

52. Demo

53. References
• http://karpathy.github.io/2016/05/31/rl/
• https://gym.openai.com/docs/rl
• https://www.nervanasys.com/demystifying-deep-reinforcement-
learning/
• http://mnemstudio.org/path-finding-q-learning-tutorial.htm
• http://videolectures.net/rldm2015_silver_reinforcement_learning/
• http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf