introduction to deep reinforcement learning

1. Introduction to Deep
Reinforcement Learning
Shenglin Zhao
Department of Computer Science & Engineering
The Chinese University of Hong Kong

2. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

3. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

4. Milestone Issues
• NIPS 2013, DeepMind, Playing Atari with Deep Reinforcement Learning,
https://arxiv.org/abs/1312.5602
• Nature cover paper 2015, DeepMind, Human-level control through deep
reinforcement learning, www.nature.com/articles/nature14236
• Nature cover paper 2016, DeepMind, Mastering the game of Go with deep
neural networks and tree search, www.nature.com/articles/nature16961

5. Reinforcement Learning in a nutshell
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

6. Deep Learning in a nutshell
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

7. Deep Reinforcement Learning:AI = RL + DL
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

8. Examples of Deep RL@DeepMind
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

9. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

10. Deep Learning = Learning Representations/Features
• Traditional Model
• Deep Learning
http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf

11. Deep Representations
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

12. Deep Neural Network
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

13. Example-CNN
http://xrds.acm.org/blog/2016/06/convolutional-neural-networks-cnns-illustrated-explanation/
Convolutional Operator
Discrete Form
Matrix Element-wise Multiplication
1 2 0
0 1 0
0 0 1
2 2 7
5 0 7
1 2 1
= 7

14. Example-CNN
https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
Pixel representation of filter Visualization

15. Example-CNN
https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/
Original image What we find

16. Example-CNN
http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf
Low-Level
Feature
Mid-Level
Feature
High-Level
Feature
Trainable
Classifier

17. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusion

18. Reinforcement Learning
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

19. Agent and Environment
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf
Agent
Environment

20. State
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

21. Major Components
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

22. Policy
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

23. Value Function
Bellman equation
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

24. Optimal Case
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

25. Approaches to Reinforcement Learning
http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf

26. RL Example
• Assumption
• Suppose we have 5 rooms in a building connected by doors
• The outside of the building can be thought of as one big room (5)
• Target
• Put an agent in any room, and from that room, go outside the building
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

27. RL Example

28. Q-learning for the RL Problem
• Assuming rewards for each step, the goal is to reach the state with
the highest reward.
• Terms— state: room, action: move decision, reward: 0 or 100
RL Problem Markov Decision Process
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

29. Q-learning for the RL Problem
Reward Table Q-value Table

30. Q-learning for the RL Problem
• Q-table is the brain of our agent, representing the memory of what
the agent has learned through experience.
• The agent starts out knowing nothing, the matrix Q is initialized to
zero.
• Simple transition rule of Q learning,
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

31. Q-learning for the RL Problem
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

32. Q-learning for the RL Problem
• Example
• Initial state: room 1, action: move to 5
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

33. Q-learning for the RL Problem
• Example
• Initial state: room 3, action: move to 1
Q(3, 1) = 0 + 0.8 * 100 = 80
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

34. Q-learning for the RL Problem
http://mnemstudio.org/path-finding-q-learning-tutorial.htm

35. Outline
• Background
• Deep Learning
• Reinforcement Learning
• Deep Reinforcement Learning
• Conclusions

36. Deep Reinforcement Learning
http://videolectures.net/rldm2015_silver_reinforcement_learning/
Deep
Learning
Reinforcement
Learning
Deep
Reinforcement
Learning

37. Deep Q-learning
Deep Q-Network (DQN)

38. DQN Architecture
Naive formulation of deep Q-network DQN in DeepMind paper
https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

39. DQN Architecture
https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

40. DQN
Loss function
Q-table update algorithm
https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

41. Exploration-Exploitation

42. Value Iteration

43. Policy Iteration

44. Stability Issues with Deep RL
http://videolectures.net/rldm2015_silver_reinforcement_learning/

45. DQN
http://videolectures.net/rldm2015_silver_reinforcement_learning/

46. Experience Replay
http://videolectures.net/rldm2015_silver_reinforcement_learning/

47. Fixed Target Q-Network
http://videolectures.net/rldm2015_silver_reinforcement_learning/

48. Reward/Value Range
http://videolectures.net/rldm2015_silver_reinforcement_learning/

49. DQN Results in Atari

50. DQN Atari Demo

51. Conclusion

52. Demo

53. References
• http://karpathy.github.io/2016/05/31/rl/
• https://gym.openai.com/docs/rl
• https://www.nervanasys.com/demystifying-deep-reinforcement-
learning/
• http://mnemstudio.org/path-finding-q-learning-tutorial.htm
• http://videolectures.net/rldm2015_silver_reinforcement_learning/
• http://icml.cc/2016/tutorials/deep_rl_tutorial.pdf