고려대학교 지능 미디어 연구실 스터디에 사용한 자료

1. DeepMind Technologies
Playing Atari with
Deep
Reinforcement
Learning

2. Contents
 Abstract
 Introduction
 Background
 Deep Reinforcement Learning
 Experiments
 Conclusion

3. Abstract
 First deep learning model using reinforcement
learning
• Successfully learn control policies directly
• From high-dimensional sensory input (pixels)
 CNN model trained on a variant of Q-learning
• Input: raw pixel
• Output: a value function estimating future reward
 Applied seven Atari 2600 games (no adjustment)
• Outperforms ALL previous approaches on six games
• Surpasses a human expert on three games

4. Introduction
 Learning directly from high-dimensional sensory input
is Long-standing challenges of RL
 Most successful RL relies on hand-crafted features
 Recent advances in deep learning extract high-level
features from raw sensory data
• Breakthroughs in computer vision/Speech recognition
 Neural architectures on supervised/unsupervised
learning
• Convolutional networks
• Multilayer perceptrons
• Restricted Boltzmann machines
• Recurrent Neural nets
Problems to solve : Motivation

5. Introduction
 Most DL requires hand labeled training data
• RL must learn from a scalar reward signal
• Reward signal is often sparse, noisy, and delayed
• Delay between actions and resulting rewards can be
thousand time steps
CNN with a variant Q-learning
• Most DL assumes data samples are independent
• RL encounters sequences of highly correlated states
Experience replay
Remaining challenges

6. Introduction

7. Background
Agent and Environment

8. Background
State

9. Background
Major Components of an RL Agent

10. Background
Policy

11. Background
On-policy learning vs Off-policy learning

12. Background
On-policy learning vs Off-policy learning
http://www.modulabs.co.kr/RL_library/2621

13. Background
Value Function

14. Background
Optimal Value Functions

15. Background
Q-Networks

16. Background
Q-Learning

17. Background
Model

18. Deep Reinforcement Learning
Deep Q-Networks : Experience Replay

19. Deep Reinforcement Learning
Deep Q-Networks : Experience Replay

20. Deep Reinforcement Learning
Model Architecture

21. Experiments

22. Experiments

23. Experiments

24. Conclusion

25. http://leehyekang.com
THANK YOU