SageMaker로 강화학습(RL) 마스터링 남궁선

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SageMaker (RL)
SageMaker and DeepRacer
Seon Namkung
Student
Hanyang University ERICA

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
(Reinforcement Learning)
SageMaker RL
Demo - SageMaker RL &Auto Scaling
DeepRacer

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
(Reinforcement Learning)

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
…(1)

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
…(2)

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
(Reinforcement) .
- Wikipedia -

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“ ”
- Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction -

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
inAI
Artificial
Intelligence
Machine
Learning
Reinforcement
Learning
Supervised
Learning
Unsupervised
Learning

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
(Environment) ?

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rewards
RL Algorithm

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AUTONOMOUS CARS FINANCIAL TRADING DATACENTER COOLINGFLEET LOGISTICS

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
, RL …
RL agent Environment

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker RL

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker
Build
-
-
-
- ,
Train
-
-
-
-
,
Deploy
-
- API
-
-

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RL
RL toolkits that provide RL agent algorithm implementations
RL-Coach RL-Ray RLLib Open AI Baselines …
DQN PPO A3C Rainbow … DQN PPO A3CIMPALA … DQN PPO …
Amazon SageMaker deep learning frameworks
Tensorflow MxNet PyTorch Chainer
SageMaker supported Customer BYO

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common gym interfaces
env.reset()
env.render()
env.step()
env.action_space
env.observation_space
Simulation environments to model real-world problems
AWS simulation
environments
Open source
environments
Custom developed
environments
Commercial
simulators
AWS
RoboMaker
EnergyPlus RoboSchool PyBullet … Bring Your Own SimuLink
Amazon
Sumerian

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simulation
environment
RL
agent
SageMaker
Training
R O L L O U T
Initial
model

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Actions and Observations
Simulation
environment
Actions
Observation
RL
agent
Initial
model
SageMaker
Training
R O L L O U T

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Training data
Simulation
environment
Actions
Observation
RL
agent
Training Data
Observation, action, reward
SageMaker
Training

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simulation
environment
Actions
Observation
RL
agent
Model updates
Training data
Observation, action, reward
SageMaker
Training

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simulation
environment
Actions
Observation
Trained Model
RL
agent
Model updates
Training data
Training
visualization
Observation, action, reward
Deploy Model
SageMaker
Training

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DEMO - SageMaker RL & Auto Scaling

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example :Auto Scaling
:
• Environment : &
• Action : /
• State : ( , )
• Reward : R( , )
< >

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configures

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
(Reward)

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observation, Reward

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Training

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DeepRacer

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer
1/18
( )

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS DeepRacer

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DeepRacer RL
ACTION GOALAGENT ENVIRONMENT

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AGENT
STATE
REWARD
ENVIRONMENT
ACTION
VALUE FUNCTION
POLICY
FUNCTION

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker RL +AWS RoboMaker accelerate learning forAWS DeepRacer

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Samples

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker
https://aws.amazon.com/ko/sagemaker/
Amazon SageMaker
https://aws.amazon.com/ko/sagemaker/pricing/
Amazon SageMaker RL
https://aws.amazon.com/ko/blogs/korea/amazon-sagemaker-rl-
managed-reinforcement-learning-with-amazon-sagemaker/
Amazon SageMaker RL
https://docs.aws.amazon.com/ko_kr/sagemaker/latest/dg/
sagemaker-rl-workflow.html

40. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Seon Namkung
whitesoil.kr@gmail.com