Downloaded 29 times
Uploaded byThom Lane
Reinforcement Learning with Amazon SageMaker RL
This document provides an overview of reinforcement learning and Amazon SageMaker RL. It discusses key RL concepts like the reinforcement learning loop, exploration vs exploitation, and common algorithms like DQN and PPO. It then introduces SageMaker RL as a way to simplify RL training through prebuilt environments, agents, and tools to train models on complex tasks without having to implement algorithms from scratch. SageMaker RL handles challenges like unstable training, sparse rewards, and hyperparameters through its presets, toolkits, and managed infrastructure for computationally expensive RL training.
Reinforcement Learning with Amazon SageMaker RL
- 1. Thom Lane 14th December2018 Reinforcement Learning DQN and PPO on SageMaker RL
- 2. Agenda Reinforcement Learning Overview SageMakerRL Overview CartPole with DQN and Coach Q-learning and Deep Q Network Roboschool Hopper with PPO and Ray Policy Gradients and Proximal Policy Optimization Mountain Car with PPO and Coach
- 3. What is ReinforcementLearning?
- 4. Agent interacts withthe environment EnvironmentAgent Actions Observations Rewards
- 5. Actions Observations Rewards Agent Environment AWS DeepRacerinteracts with the racetrack
- 6. Challenges Unstable training Training Dataisn’t static Current training data is from current policy! Sparse Rewards High Sensitivity to HPs broken RL code almost always fails silently Exploration vs exploitation
- 7.
- 8. Closer look atthe environment Many kinds of observation spaces Continuous or discrete Could be vector, audio, image, video, etc. Observation != State Observation: e.g. image from camera State: e.g. momentum (physical dynamics) Many kinds of action spaces Continuous or discrete !" !# !$ %" %# %$ &" &# &$
- 9. Markov Decision Process(MDP) Stay Go Stay GoStay Go StayGo StayGo +10 +5 -10 -20 +1 80% 20% -100-100
- 10. Modeling the MDP Usuallywe don’t have the MDP in reality! “Model-based” RL is when the following is given or learnt: 1) Transition function: e.g. ℙ "#$% "#, '#) 2) Reward function: e.g. ℙ )#$% "#, '#) And then use planning methods to decide on actions. “Model free” RL doesn’t have these requirements.
- 11. Collecting trajectories !"#" $"!%#% $% !&#& $& ' !"#" $" !(#( $( ' !"#" $" !%#% $% ' Episode Transition or Step Done
- 12.
- 13.
- 14. Rewards Get a reward!" from environment after action #". Can be positive, negative or equal to 0. Return $" is cumulative discounted future reward. $" = 1 −1 + 0.9 −1 + 0.81 5 = 2.15 $" = !" + /!"01 + /2 !"02 + ⋯ = ∑567 8 /5 !"05 Objective of RL Agent is to maximize expected return.
- 15.
- 16. What are wetrying to learn? With Q-Learning… We learn the value of taking an action from a given state. ‘Q-Value’ is the expected return after taking the action. We’ll use Deep Q Network as an example. With Policy Optimization… We learn the action to take from a given state (i.e. observations). We call the model for this the policy, denoted !"($|&). We’ll use Proximal Policy Optimization as an example. We often learn the value of being in a given state too (i.e. Actor Critic)
- 17. What are wetrying to learn? Q-Learning Policy Optimization Actor Critic Observation Observation Observation Q-values Action probabilities Action probabilities State value
- 18. Exploration vs Exploitation Greedy alwaystake greedy action !-greedy with probability ! take random action otherwise take greedy action
- 19. Exploration Policies Action Space Noise Parameter SpaceNoise Ornstein Uhlenbeck Process Entropy Loss
- 20. Exploration Policies: Curiosity IntrinsicCuriosity Module Without Curiosity (i.e. random actions) With Curiosity Source: https://blogs.unity3d.com/2018/06/26/solving-sparse-reward-tasks-with-curiosity/
- 21. Agent interacts withthe environment EnvironmentAgent Actions Observations Rewards Neural Networks Exploration Policies Algorithm Reward Aggregation
- 22.
- 23. Applicable in manydomains (not just gaming) Robotics Industrial control HVAC Autonomous vehicles NLP Operations Finance Resource allocation Advertising Online content delivery
- 24. But RL modelscan be tricky to train Difficult to get started RL agent algorithms are complex to implement Hard to integrate environments for training Training is computationally expensive and time consuming Requires trial and error and frequent tuning of hyperparameters
- 25. SageMaker RL: Environments OpenAI Gym (Cartpole, Mountain Car, Atari) Open AI Roboschool EnergyPlus AWS RoboMaker Simulink Amazon Sumerian Custom Environments Travelling Salesman Portfolio Management Auto-scaling Model Compression
- 26. SageMaker RL: Toolkits& Agents Intel Coach Open AI Baselines Stable BaselinesRay RLlib
- 27.
- 28.
- 29.
- 30. Customers are usingAmazon SageMaker RL
- 31.